Skip to main content

C-02 Durable Execution

Brick C: Agents, autonomy, execution. French version: C-02-durable-execution

1. Role and responsibilities​

Guarantees that a run survives crashes, restarts and long waits (approvals, timers). By default durability relies on PostgreSQL: run state, steps and checkpoints are persisted, and AgentLoop.resume(run_id) resumes from the last checkpoint. The Temporal adapter (enterprise_ai.runtime.temporal.build_workflow) is optional for deployments that require Temporal durable workflows.

2. Position in the architecture and interactions​

  • Callers: AgentLoop (Python SDK), TypeScript, Java and Go runtimes through /v1/runs.
  • Persistence: eai.runs, eai.run_steps, eai.checkpoints, eai.run_events; the server worker python -m eai.worker detects stale heartbeats.
  • Resume: POST /v1/runs/{id}/resume on the API side, AgentLoop.resume(run_id) on the engine side.

3. Open source component and developed part​

  • Engine: Navin architecture of the Python SDK, with no declared graph. enterprise_ai.loop.AgentLoop takes in messages, prepares the context, manages sessions, resumes a run after a crash, handles stop, cron and heartbeat, and exposes serve(); enterprise_ai.runtime.AgentRunner runs one turn; enterprise_ai.subagents.SubagentManager runs sub-agents in parallel; enterprise_ai.loop.plan.RunPlan carries the plan; enterprise_ai.runtime.bus.MessageBus links channels and engine.
  • Optional adapters: LangGraph (enterprise_ai.runtime.langgraph.build_graph) and Temporal (enterprise_ai.runtime.temporal.build_workflow) reuse the same steps; neither is required.
  • Server side, python -m eai.worker runs the periodic jobs: scheduler tick, event outbox drain, quota reservation expiry, stale heartbeat detection, retention.
  • Optional open source: Temporal (adapter eai/integrations/temporal.py server side, enterprise_ai.runtime.temporal SDK side).

4. Data model​

Tables (schema eai, tenant_id RLS):

  • eai.runs
  • eai.run_steps
  • eai.checkpoints

Common columns: id uuid, tenant_id uuid NOT NULL, created_at, updated_at; RLS policy current_setting('app.tenant_id', true)::uuid.

5. Contract​

  • Applied contract: execution-contract
  • JSON Schema: contracts/v1/execution-contract.schema.json
  • Key fields: state, retry_policy, resume, errors, steps[]
  • Validation: POST /v1/contracts/{name}/validate; manifest apply: POST /v1/manifests/apply.

6. API endpoints​

  • POST /v1/runs/{id}/resume
  • POST /v1/runs/{id}/cancel
  • GET /v1/runs/{id}

Request POST /v1/runs/{id}/resume:

{
  "from_checkpoint": "chk_0042",
  "reason": "worker restart"
}

Response:

{
  "run_id": "run_01J9",
  "state": "running",
  "resume": {
    "checkpoint_id": "chk_0042",
    "attempt": 2
  }
}

7. SDK examples​

Python:

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_xxx_secret")
client.runs.resume("run_01J9", checkpoint_id="chk_0042")

TypeScript:

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY });
await client.runs.resume("run_01J9", { checkpoint_id: "chk_0042" });

Java:

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .build();
client.runs().resume("run_01J9");

Go:

c, _ := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")))
c.Runs.Resume(ctx, "run_01J9")

8. Configuration​

  • Environment variables: EAI_DATABASE_URL, EAI_TEMPORAL_URL, EAI_TEMPORAL_NAMESPACE, EAI_TEMPORAL_TASK_QUEUE (optional Temporal adapter)
  • Helm keys: controlPlane.config, worker.enabled

9. Security controls​

  • mTLS between the server worker, the Control Plane and, when the adapter is enabled, Temporal
  • Payloads encrypted by codec (E-04)
  • Isolation by namespace and tenant_id
  • Every request carries the tenant resolved from the credential; PostgreSQL RLS
  • Every mutation and policy decision writes a hash-chained audit event

10. Observability​

  • Prometheus metrics: eai_workflow_retries_total, eai_workflow_resume_total, temporal_workflow_failed
  • Traces: OpenTelemetry spans with GenAI semantic conventions (gen_ai.operation.name, gen_ai.agent.id, gen_ai.tool.name), attributes eai.tenant_id, eai.run_id, traceparent propagation.
  • Audit actions: run.transition

11. FinOps units​

  • Worker activity duration, retry count.
  • Emitted as Usage Event (POST /v1/finops/usage) with tenant, project, agent and run.

12. Failure modes and resilience​

  • Process crash: resume from the last checkpoint with AgentLoop.resume(run_id)
  • Temporal not configured: PostgreSQL durability stays active, with no external dependency
  • No heartbeat beyond EAI_HEARTBEAT_STALE_SECONDS: the run shows up in GET /v1/heartbeats and the worker reports it

13. Standalone adoption (client of a client)​

  • enterprise_ai.agent + enterprise_ai.loop + Control Plane + PostgreSQL; optional Temporal (EAI_TEMPORAL_URL).
  • SDK: enterprise_ai.agent (AgentsClient, RunsClient, SessionsClient)
  • Minimal deployment: Helm chart with controlPlane.enabled=true and PostgreSQL; disabled external components use their in-process fallback.

14. Acceptance tests​

  1. Kill the worker mid step: the run resumes
  2. Cancel delivered as signal: state cancelled
  3. Cross-tenant access refused
  4. Error returned in fr with Accept-Language: fr

Kit L3 proof​

Real Temporal: the adapter eai.integrations.temporal starts a durable workflow through the HTTP API (start_workflow, describe_workflow, workflow_result). Proof: tests/test_runtime_temporal_durable.py runs temporal server start-dev on free ports, kills the worker in the middle of step 2, a new worker resumes from the history: step 1 ran once, final result checked. Toolchain: tools/runtime/fetch.sh (temporal CLI 1.9.1, temporalio 1.18.2 in /tmp/toolchains/temporal-venv). NATS JetStream: publish_event sets Nats-Msg-Id (duplicate dropped inside the window), IdempotentHandler applies each event once even after a redelivery; proof tests/test_runtime_nats_jetstream.py (nats-server 2.15.2).