Backup, restore and disaster recovery

Kit addendum sections 20.5, 20.6, 20.7 and 20.10. HA, backup and DR are distinct: HA (availability package) keeps a service up inside a site; a backup is an encrypted, checksummed artefact on client storage; DR moves the platform to another site following an approved plan. Every route is listed in navin-manager/CONTRACT.md section 20.5.

Backup

• POST /api/v1/backup-targets: provider (s3, azure_blob, gcs, ibm_cos, seaweedfs, filesystem), bucket, secret_ref (a reference only, never a secret), capabilities recorded by a qualification ({"supported": true, "verified_at"}).
• POST /api/v1/backup-plans: scope, interval, window, encryption_key_ref, retention in days, immutability_required. An immutable plan on a target without verified Object Lock and retention is refused (409 backup_worm_unsupported): fail closed.
• Artefacts are AES-256-GCM encrypted; the data key is derived from a custody key held outside the backup storage (NM_BACKUP_KEYS), only its reference is stored.
• POST /api/v1/backups/{id}/verify checks size, SHA-256 and the GCM tag. GET /api/v1/backup-plans returns late or failed plans; each alert is audited (backup.alert).
• Retention: DELETE /api/v1/backups/{id} is refused before the expiry (409 backup_retention_active); POST /api/v1/backups/retention/enforce removes the expired artefacts.
• Periodic restore tests: POST /api/v1/backups/{id}/restore-tests restores into an isolated directory, optionally injects a corruption, signs the report with Ed25519 and raises an audited alarm on failure.
• Cross consistency: POST /api/v1/backups/consistency compares database rows, stored objects and the artefact registry, lists orphans and, with reconcile, marks missing rows and quarantines orphan objects.

Engines per component

Component	Method (module navin_manager.backup.engines)
PostgreSQL	pgBackRest full backup, WAL archive and PITR; in Kubernetes the CloudNativePG method: plugin (deploy/data/postgres/backups.yaml). A volume snapshot is never a PostgreSQL backup.
Agent checkpoints	Tables eai.runs, eai.run_events, eai.checkpoints of the control plane, inside the platform PostgreSQL backup scope (AGENT_STATE_TABLES).
etcd	etcdctl snapshot save, etcdutl snapshot restore into a new data directory, new member.
OpenSearch	Snapshot to a filesystem repository (path.repo), restore into a new index.
Opik	Coordinated plan (navin_manager.backup.opik.plan): quiesce, MySQL logical dump, ClickHouse BACKUP DATABASE, Redis RDB, object storage bucket, resume. Traces: engines.clickhouse_backup and engines.clickhouse_restore (native BACKUP and RESTORE DATABASE to an allowed backup disk, refused when the database already exists). State: engines.mysql_dump (--single-transaction, routines, triggers) and engines.mysql_restore.
Temporal	Persistence copy (SQLite backup API in the development server; the production Temporal databases are in PostgreSQL).
Artefacts	OCI layouts, prompts, skills, plugins, policies and models in a manifest of digests signed with Ed25519 (backup.artifacts). An OCI registry repository is exported to an OCI layout with oras, cosign signature included (artifacts.oci_export), then reloaded after the manifest is verified (artifacts.oci_load): digest unchanged, cosign verify still accepted.
Secrets, certificates, CA	Encrypted archive whose key is handed to the custodian once and never stored with the archive (backup.secrets_archive).
Audit evidence	SeaweedFS or S3 Object Lock bucket distinct from the backup bucket, qualified by backup.worm.qualify.

The toolchains of the tests come from tools/backup/fetch.sh (etcd 3.6.5, pgBackRest 2.56.0 built without root, Backup CRD schema, ClickHouse 25.3.3.42 and MySQL 8.4.5 at the versions pinned in the manager catalogue) and from tools/oci/fetch.sh (zot, oras, cosign), all pinned with sha256.

Restore

1. POST /api/v1/restores/plan: steps and compatibility check (same major version per component).
2. POST /api/v1/restores/{id}/dry-run: digest, decryption and compatibility checked, approvers already recorded.
3. POST /api/v1/restores/{id}/approve by two people other than the requester.
4. POST /api/v1/restores/{id}/execute (permission restore:execute, separate from backup:manage): artefact checked again; on failure the configuration snapshot is restored (rolled_back). No execution without an approved plan, production included.

Disaster recovery

• POST /api/v1/dr-plans: mode active_passive, warm, cold or active_active_stateless. Active-active is limited to the stateless layers (portal, API gateways, workers); stateful data keeps a single writer per site pair.
• RPO and RTO targets are set by the client with PUT /api/v1/dr-plans/{id}/targets, empty by default.
• Exercises and failovers: request, approval by another person, completion with the RPO and RTO measured during the exercise. The PostgreSQL failover exercise of tests/test_recovery_postgres_real.py measures them on a real run.
• Recovery DAG (recovery.dag): steps run as local processes in dependency order with health checks.
• DNS and load balancer: declared models with health checks (/dr-plans/{id}/traffic, /switch-plan); the manager never applies them, the client's provider does.
• Rebuild of the manager (recovery.rebuild): offline bundle verified, database dump restored into a new database, migrations, audit chain check, manifests copied; no call to the publisher.
• Licence (recovery.licence): the control plane verifier runs offline; the licence of the installation group stays valid in the restored installation of the same group.
• Reconciliation (recovery.reconcile): idempotency keys and external receipts, no double payment after failover.
• Fencing: navin_manager.recovery.fencing.fence_postgres makes the old primary read only, writes its standby.signal and stops it before promotion: a client that turns default_transaction_read_only off can no longer write to it. No API route calls this module yet: only its tests do.

Limits

• MySQL and ClickHouse runs of the Opik plan need mysqldump and clickhouse on the host (or the paths given in tools); without them the plan marks these steps blocked.
• Backup and reload of a real Harbor (projects, robot accounts, replication) need Docker or a cluster: Harbor ships only as container images.
• The CloudNativePG Barman Cloud plugin requires operator 1.26 or later; with the pinned 1.25.0 the in-tree barmanObjectStore of cluster.yaml stays active.
• DNS and load balancer changes need the client's DNS and load balancer; Velero runtime needs a cluster.