Skip to main content

C-08 Heartbeat Monitor

Brick C: Agents, autonomy, execution. French version: C-08-heartbeat-monitor

1. Role and responsibilities​

Detects stuck or orphan runs from heartbeats. AgentRunner sends POST /v1/runs/{id}/heartbeat before every model or tool call (and enterprise_ai.heartbeat.Heartbeat in the background); the action=stop answer stops the turn. The server worker python -m eai.worker detects heartbeats stale beyond EAI_HEARTBEAT_STALE_SECONDS, exposed by GET /v1/heartbeats (Prometheus format with ?format=prometheus).

2. Position in the architecture and interactions​

  • Emitters: AgentRunner, Heartbeat, TypeScript (startHeartbeat), Java (client.heartbeats().beat) and Go (client.Heartbeats.Beat) runtimes. Readers: server worker, Prometheus, portal.

3. Open source component and developed part​

  • Engine: Navin architecture of the Python SDK, with no declared graph. enterprise_ai.loop.AgentLoop takes in messages, prepares the context, manages sessions, resumes a run after a crash, handles stop, cron and heartbeat, and exposes serve(); enterprise_ai.runtime.AgentRunner runs one turn; enterprise_ai.subagents.SubagentManager runs sub-agents in parallel; enterprise_ai.loop.plan.RunPlan carries the plan; enterprise_ai.runtime.bus.MessageBus links channels and engine.
  • Optional adapters: LangGraph (enterprise_ai.runtime.langgraph.build_graph) and Temporal (enterprise_ai.runtime.temporal.build_workflow) reuse the same steps; neither is required.
  • Server side, python -m eai.worker runs the periodic jobs: scheduler tick, event outbox drain, quota reservation expiry, stale heartbeat detection, retention.

4. Data model​

Tables (schema eai, tenant_id RLS):

  • eai.runs (columns last_heartbeat_at, heartbeat)

Common columns: id uuid, tenant_id uuid NOT NULL, created_at, updated_at; RLS policy current_setting('app.tenant_id', true)::uuid.

5. Contract​

  • Applied contract: execution-contract
  • JSON Schema: contracts/v1/execution-contract.schema.json
  • Key fields: state, updated_at
  • Validation: POST /v1/contracts/{name}/validate; manifest apply: POST /v1/manifests/apply.

6. API endpoints​

  • POST /v1/runs/{id}/heartbeat
  • GET /v1/heartbeats

Request POST /v1/runs/{id}/heartbeat:

{
  "step_index": 4,
  "worker": "agent-runtime-5d9f"
}

Response:

{
  "run_id": "run_01J9",
  "received_at": "2025-01-10T10:00:05Z",
  "stale_after_s": 60
}

7. SDK examples​

Python:

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_xxx_secret")
client.runs.heartbeat("run_01J9", step=4, progress=0.4)
stale = client.runs.stale_heartbeats()

TypeScript:

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY });
await client.heartbeats.send("run_01J9", { step: "4", progress: 0.4 });

Java:

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .build();
client.heartbeats().beat("run_01J9", 4);

Go:

c, _ := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")))
c.Heartbeats.Heartbeat(ctx, "run_01J9", map[string]any{"step": "4", "progress": 0.4})

8. Configuration​

  • Environment variables: EAI_HEARTBEAT_STALE_SECONDS=60, EAI_DATABASE_URL
  • Helm values keys: controlPlane.config.EAI_HEARTBEAT_STALE_SECONDS, prometheus.enabled

9. Security controls​

  • Only the run's agent identity can emit its heartbeat
  • Every request carries the tenant resolved from the credential; PostgreSQL RLS
  • Every mutation and policy decision writes a hash-chained audit event

10. Observability​

  • Prometheus metrics: eai_heartbeat_age_seconds, eai_runs_stale
  • Traces: OpenTelemetry spans with GenAI semantic conventions (gen_ai.operation.name, gen_ai.agent.id, gen_ai.tool.name), attributes eai.tenant_id, eai.run_id, traceparent propagation.
  • Audit actions: run.transition

11. FinOps units​

  • Negligible.
  • Emitted as Usage Event (POST /v1/finops/usage) with tenant, project, agent and run.

12. Failure modes and resilience​

  • Frozen client process: no more heartbeat, the run shows up as stale and can be resumed with AgentLoop.resume(run_id)
  • Agent suspended or killed: the heartbeat answer carries action=stop with the reason

13. Standalone adoption (client of a client)​

  • enterprise_ai.agent + Prometheus.
  • SDK: enterprise_ai.agent (AgentsClient, RunsClient, SessionsClient)
  • Minimal deployment: Helm chart with controlPlane.enabled=true and PostgreSQL; disabled external components use their in-process fallback.

14. Acceptance tests​

  1. Without heartbeat for 60 s, the run appears in GET /v1/heartbeats
  2. Cross-tenant access refused
  3. Error returned in fr with Accept-Language: fr