Skip to main content

ADR-0155: Worker planes, operation_id of long operations and compensation of external actions

Context​

The architecture reference asks for audit-plane, soc-plane and telemetry-plane deployment units (A5.2) scaled separately (B6.2), an operation_id for every long operation to follow it, read its result and avoid duplicates (B6.3), and compensations for actions on external systems (B13.5). The Control Plane worker ran every job in one deployment, the Control Plane returned the id of the tracked resource and accepted Idempotency-Key on runs and approvals only, and an external action committed then cancelled had no reverse action.

Decision​

  • python -m eai.worker --plane audit|soc|telemetry|core runs only the jobs of that plane (eai.worker.PLANES; core takes the others) under its own advisory lock (EAI_WORKER_LOCK_KEY + the plane offset). With worker.planes.enabled the Helm chart creates one Deployment per plane with its own resources and replicas; the network policies and the PodMonitor select the label eai.io/workload-class: control-plane-worker, carried by every plane. Without the option a single worker runs everything (unchanged behaviour).
  • The domain eai.domains.operations records every long creation (runs, evaluation and red team runs, marketplace installations and upgrades, knowledge base reindex, connector sync, video generation) in eai.operations (V188). The response carries Operation-Id and Operation-Location; GET /v1/operations/{operation_id} returns the accepted answer, the tracked resource and its live state, read through the resource route with the caller's credentials. With Idempotency-Key a retry by the same principal with the same body replays the first answer without running again; another body gets 409 idempotency_conflict, a retry while the first one runs gets 409 operation_in_progress. A refused request leaves no operation.
  • eai.domains.soc.transactions.compensate and compensate_run undo a committed external action with its registered reverse action (payment.create to payment.refund...), at most once: key compensation:<key>:<attempt>, durable intent (compensating, V185) before the call, compensated after; an unknown outcome waits for the SOC reconciliation, never a blind retry.

Consequences​

Each plane keeps a single active replica (advisory lock): extra replicas are standbys. Telemetry ingestion by the API follows the Control Plane HPA. Enabling the planes on an existing installation does not change the selector of the main worker.

Evidence​

services/control-plane/tests/test_worker_planes.py, services/control-plane/tests/test_operations.py, services/control-plane/tests/test_side_effect_compensation.py, and in the SDKs sdks/python/tests/test_operations.py, sdks/typescript/packages/sdk/test/operations.test.ts, sdks/go/operations_test.go, sdks/java/enterprise-ai-java/src/test/java/io/enterpriseai/OperationsClientTest.java.