Skip to main content

ADR-0126: Configured high availability is not validated high availability

Context​

Kit addendum 20.4 and 20.10. Three replicas do not prove HA: failures must be tested, recovery times measured, and no critical operation may be lost or run twice.

Decision​

  • Three profiles: standalone (no fault tolerance guarantee), ha (critical services replicated over several nodes, failover) and critical-ha (several fault domains, independent DR and backups). The banking production profile starts from ha.
  • Each HA group exposes its desired and healthy replicas, fault domains, quorum, failover policy, target and observed RTO and RPO. It stays HA_CONFIGURED until a recorded failure test moves it to HA_VALIDATED.
  • Exclusive operations (install, upgrade, failover, restore) go through a lease with a fencing token and a persistent state machine: two Manager instances never run the same operation.
  • PostgreSQL: CloudNativePG inside Kubernetes or Patroni outside Kubernetes, never both on one cluster, or HA managed by the client. The old primary is fenced before any promotion.
  • Redis: Sentinel (at least three sentinels in independent fault domains) or Redis Cluster. Replication is asynchronous and can lose writes, so Redis is never the source of truth.
  • NATS JetStream: streams replicated R3 or R5 with quorum. LiveKit: a room lives on one node, losing it interrupts the session and forces a reconnection; draining limits the effect of planned stops.
  • Security decisions and orders stay available when the console is down. Critical operations are refused when the Manager database or the audit write path is unavailable.

Consequences​

No service is announced HA without execution evidence. RTO and RPO targets come from the client; only values measured in a test are shown as observed.

Evidence​

  • KIT_AGENT_PLATEFORME_IA_ENTERPRISE/ADDENDUM_INFRA_HA_BACKUP_PRA.md
  • navin-manager/server/src/navin_manager/availability
  • navin-manager/server/tests/test_availability_local_ha.py
  • navin-manager/server/tests/test_availability_helm.py
  • deploy/helm/enterprise-ai/values-ha.yaml