Platform high availability (addendum 20.4)

Model

Every cluster and service carries ownership_mode (PLATFORM_MANAGED, CLIENT_MANAGED) and deployment_mode (SINGLE_HOST, NODE_GROUP, K8S_EXISTING, K8S_MANAGED, MULTI_CLUSTER, EXTERNAL_MANAGED), migration server/sql/V050__availability.sql. An HA group (nm.ha_groups) declares desired_replicas, fault_domains, ha_mode, failover_policy, a profile standalone, ha (2 replicas, 2 domains) or critical-ha (3 replicas, 3 domains) and RTO and RPO targets given by the client (empty by default).

ha_validation_status stays HA_CONFIGURED until a recorded failover test (nm.ha_test_runs, status passed) newer than the last change of the group exists; only then HA_VALIDATED. A member without a recent heartbeat (90 s) is unknown, never healthy: the group becomes DEGRADED. MULTI_CLUSTER is marked unsupported (no second site here).

Routes

GET /api/v1/ha-modes, POST /api/v1/ha-groups, GET /api/v1/ha-groups, GET /api/v1/ha-groups/{id}, POST /api/v1/ha-groups/{id}/test-failover, GET /api/v1/ha-groups/{id}/operations. Authenticated, authorized (cluster:manage, services:operate, cluster:read), audited. A destructive action on CLIENT_MANAGED or EXTERNAL_MANAGED is refused (409 ha_client_managed_refused, audit denied). Database or audit trail down: refused (503 ha_fail_closed).

Exclusive operations

nm.exclusive_leases: one lease per scope (install, upgrade, failover, restore), taken by an atomic INSERT ... ON CONFLICT, growing fencing token (nm.fencing_token_seq). A test result is refused when its token is no longer valid. Tested with two racing processes.

Measured local proofs (official binaries, no Docker)

Component	Scenario	Measured RTO	Measured loss
PostgreSQL 16.2	primary killed (-m immediate), replica promoted by pg_ctl promote, old primary fenced (pg_rewind, standby, write refused), rejoin	0.215 s	0 of 2000 rows
Patroni 4.0.6 + etcd 3.5.21	leader killed (Patroni and PostgreSQL), election, old leader rejoined read only	19.878 s (ttl 10 s)	0 of 500 rows
Redis 7.0.15 Sentinel	1 primary, 2 replicas, 3 sentinels, primary killed, client resolved through SENTINEL get-master-addr-by-name	2.223 s	0 of 2000 keys
NATS 2.15.2 JetStream	R3 stream, leader killed, publish and consume resumed, duplicate refused (Nats-Msg-Id), 10 redeliveries deduplicated	5.569 s	0 of 100 messages

PostgreSQL and Redis replication is asynchronous: zero loss is not guaranteed, only measured zero in these runs. The figures hold for this local host and do not replace the client targets.

Kubernetes (configured only)

deploy/helm/enterprise-ai/values-ha.yaml: at least 3 replicas, PDB minAvailable: 2, spread over hostname and zone, required anti affinity per node, probes, 60 s graceful shutdown. The CloudNativePG manifest is validated offline by kubeconform against the pinned CRD schema (tools/ha/fetch.sh, sha256). No failover was run on a cluster: BLOCKED, no test cluster (the available context is production). Temporal is modelled HA_CONFIGURED only. Loss of a LiveKit room: session interruption then client reconnection, no transparent transfer.

Catalog services

What each catalog service can deliver in high availability (cluster, failover, quorum, prerequisites, official documentation, topology per mode) is declared in its high_availability block, served by GET /api/v1/catalog/high-availability and summed up in the generated matrix en/high-availability.md.

Install checks and supervision

Before an install (POST /clusters/{cid}/services, POST /clusters/{cid}/blueprint/apply) and before a topology change of an installed service (POST .../services/{s}/components adds instances, DELETE .../services/{s}/components/{c}/hosts/{h} removes one), server/src/navin_manager/services/ha_checks.py checks the chosen mode against the chosen hosts: ha.distinct_hosts, ha.fault_domains (host label zone, else its rack, else unknown), ha.quorum, ha.prerequisites (an ha: true prerequisite bound to a catalog service running in a standalone mode fails, bound to an external connection asks to confirm its HA), ha.entrypoint (a load-balancer or VIP entry point needs the HAProxy pair with Keepalived of the cluster, or a declared external load balancer), ha.peer_tls, ha.readiness, and ha.not_ha (info) for a standalone mode. POST .../services/validate and the install responses return these views under ha.

• A fail refuses the request: 409 ha_requirements_not_met, details.checks.
• A warn needs ha_acknowledged: true in the body (query parameter on the DELETE), else 409 ha_acknowledgement_required; the acknowledgement is recorded in the audit event. Bodies without warnings stay valid unchanged. A topology change only counts the checks that get worse than with the current placement.
• Operating modes (POST .../services/{s}/mode: drain, emergency stop, security suspension...) are never refused on HA grounds; their response carries an informative ha impact.
• The service dashboard returns ha_status (level, healthy instances, quorum state ok, at_risk, lost, state healthy, degraded, quorum_lost or standalone, docs, cluster status command). The console shows it on the service sheet with a button running the declared healthcheck.cluster_status command. The Services page gathers the HA groups, quorum, failover tests and catalog capability of each service in its "High availability (HA)" row action; the catalog matrix and the target architecture are in en/high-availability.md (the /availability and /high-availability pages redirect to Services).

Tests

server/tests/test_availability_api.py, test_availability_local_ha.py, test_availability_helm.py, ui/src/test/services-consoles-ha.test.tsx. Measurements written to /tmp/ha-evidence/*.json.