High availability and disaster recovery

Targets

Scope	RPO	RTO
PostgreSQL (CNPG, continuous WAL)	5 min	30 min
Audit WORM (S3 object lock, replication)	0 (synchronous write)	1 h
Kubernetes config (hourly Velero + GitOps)	1 h	1 h
Qdrant vector indexes (daily snapshots, reindex possible)	24 h	4 h
OpenBao (daily raft snapshot + 3 node raft HA)	24 h	1 h
vLLM inference (stateless)	n/a	15 min (model from cache)

Multi-AZ

• Control plane, portal, LiteLLM: >= 3 replicas in prod, PDB minAvailable: 2, zone anti-affinity (values-prod.yaml), topology spread.
• CNPG: 3 instances, required zone anti-affinity, automatic failover.
• NATS JetStream and Qdrant: 3 replicas, replication factor 3.
• Redis: sentinel or cluster; APISIX rate limits tolerate a transient loss.

Backups

• CNPG: daily base backup (ScheduledBackup) + encrypted WAL archive to S3, 30 day retention. PITR: new Cluster with bootstrap.recovery.recoveryTarget.targetTime.
• Verified logical backup (custom format): PGDATABASE=eai PGPASSFILE=/run/secrets/pgpass scripts/pg_backup.sh /backup/eai.dump, restore into an empty database: createdb eai_drill && PGDATABASE=eai_drill EAI_PYTHON=.venv/bin/python EAI_DATABASE_URL=... scripts/pg_restore.sh /backup/eai.dump (fails when a migration is pending). Tested by tests/test_core_pg_backup.py (data, migrations, FORCE RLS). Details: postgresql.md.
• Redis: no backup needed (cache, rebuilt from PostgreSQL). See redis.md.
• Velero (deploy/velero/schedules.yaml): daily (volumes + pg_dumpall, Qdrant snapshot, OpenBao raft snapshot hooks) and hourly (config). Secrets are restored from OpenBao.

Restore drill (quarterly)

1. velero restore create --from-backup eai-platform-daily-<date> --namespace-mappings enterprise-ai:enterprise-ai-drill.
2. CNPG PITR into an isolated namespace, run GET /v1/audit/verify.
3. Record measured RPO/RTO.

Velero restore test and isolated site (release bundle)

• Manifests deploy/velero/restore-test.yaml: on-demand Backup eai-restore-test of the enterprise-ai and eai-security namespaces, then a Restore into the check namespaces eai-restore-check and eai-restore-check-security (live namespaces untouched).
• Validated offline with kubeconform and the pinned Velero schemas (tools/release/fetch-tools.sh), test services/control-plane/tests/test_release_ci.py::test_rendered_platform_manifests_validate.
• Real run: acceptance cluster only (kubectl apply -f deploy/velero/restore-test.yaml, then velero restore describe eai-restore-test). Limit: not run here, no acceptance cluster available.
• Recovery on an isolated site: reinstall from the signed bundle verified offline (bash tools/release/verify.sh <bundle>), see installation-handbook.md; the no-network proof is tools/release/airgap-check.sh (result in release/airgap/<date>.json).

High availability managed by the Platform Manager (addendum 20.4)

Helm profile deploy/helm/enterprise-ai/values-ha.yaml (3 replicas, PDB minAvailable: 2, hostname and zone spread, anti affinity, 60 s graceful shutdown) and manager HA groups: see navin-manager/docs/availability.en.md (model, measured local proofs for PostgreSQL, Patroni, Redis Sentinel, NATS JetStream, limits). On Kubernetes the configuration is rendered and validated offline only (HA_CONFIGURED).