Skip to main content

H-14 Backup and disaster recovery

Brick H of the Enterprise AI Platform. Technology: Velero (cluster resources, volumes) + database tools (pgBackRest or CloudNativePG barman, clickhouse-backup, Redis RDB), object storage replication. French version: ../../fr/bricks/H-14-backup-dr.md.

Role and responsibilities​

  • Meet RPO/RTO targets: PostgreSQL PITR (RPO minutes), daily cluster backup, cross-region copies, restore drills.

Position and interactions​

  • Execution path: batch (scheduled) and recovery workflow.
  • After restore, GET /v1/audit/verify proves the audit chain integrity.

Open source and developed parts​

  • Open source reused: Velero, pgBackRest/barman, clickhouse-backup.
  • Developed in the platform: Runbooks, drill automation, restore verification tests.

Data model​

Control Plane tables (schema eai): eai.audit_exports.

Every tenant table has tenant_id uuid NOT NULL and a row level security policy on current_setting('app.tenant_id', true)::uuid. ClickHouse usage_events is the analytics replica of eai.usage_events (ReplacingMergeTree keyed by event_id).

Contract: contracts/v1/audit-event.schema.json (schema_version 1.0). Usage is always reported with contracts/v1/usage-event.schema.json, audit with contracts/v1/audit-event.schema.json.

API endpoints​

MethodPath
GET/readyz
GET/v1/audit/verify

GET /readyz​

Response:

{
  "status": "ready",
  "checks": {
    "postgres": "ok",
    "redis": "ok",
    "clickhouse": "ok"
  }
}

GET /v1/audit/verify​

Response:

{
  "valid": true,
  "checked": 90311,
  "first_sequence": 1,
  "last_sequence": 90311,
  "broken_at": null
}

SDK examples​

Python​

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_live_xxx", tenant="acme", locale="en")
client.system.readyz()
client.audit.verify()

TypeScript​

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY, tenant: "acme", locale: "en" });
await client.meta.ready();
await client.audit.verify();

Java​

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .tenant("acme")
    .locale(Locale.ENGLISH)
    .build();
client.meta().ready();
client.audit().verify();

Go​

client, err := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")), eai.WithTenant("acme"), eai.WithLocale("en"))
if err != nil { log.Fatal(err) }
ctx := context.Background()
_, err = client.Meta.Ready(ctx, nil)
_, err = client.Audit.Verify(ctx, nil)

Configuration​

No EAI_* variable: configured through Helm values and the tool's own configuration.

Security​

  • Backups encrypted with tenant aware keys; restore requires two people.
  • Tenant resolved from the credential, never from the body; PostgreSQL row level security on every tenant table.
  • Every mutation and policy decision writes a hash chained audit event (G-11).
  • Errors use the standard envelope with message_i18n en and fr.

Observability​

  • Last successful backup age alerts.
  • W3C traceparent propagated, X-Request-Id returned, spans carry eai.tenant_id, eai.project, eai.agent_id, eai.run_id.

FinOps units​

  • Backup storage allocated per tenant.

Failure modes and resilience​

  • Region loss -> restore in secondary region following the PRA runbook.

Standalone adoption​

Integrates with the client's backup tooling via Velero plugins.

Acceptance tests​

  1. Quarterly drill restores prod snapshot in an isolated namespace within RTO and audit verify returns valid=true.

Implementation roadmap