SOC, AI-SDR, Kill Switch and recovery runbook

Version française : ../soc-runbook.md

Kit section 11. Code: services/control-plane/src/eai/domains/soc (routes /v1/soc/*, see the API contract, section SOC, AI-SDR and Kill Switch), risk engine eai/domains/risk/engine.py, Response Policy Engine deploy/policies/opa/soc_response.rego, migrations sql/postgres/V120 to V122, contracts security-incident and containment-order, tests services/control-plane/tests/test_soc_ai_sdr.py and test_soc_runtime_rules.py.

Components

Component	Where	What it does
Detection Ingestor	soc/detections.py, soc/webhooks.py, soc/jobs.py (job soc_scan)	Normalizes every source into one detection: internal (guard, DLP, egress, quotas, plugins, runtime, authz, policy, audit, knowledge, MCP, read from the audit chain after a per tenant watermark) and external (Falco, Tetragon, OpenSearch Security Analytics through HMAC signed webhooks)
Event Correlator	soc/detections.py	Same user, else same agent, else tenant level detections within EAI_SOC_CORRELATION_WINDOW_SECONDS share one open incident (category soc)
Risk Engine	domains/risk/engine.py	Severity, confidence, score and the list of reasons, stored on the detection and on the incident timeline
Incident Manager	domains/incidents	Lifecycle open, investigating, contained, resolved, closed, assignee, timeline, SIEM forwarding
Response Policy Engine	deploy/policies/opa/soc_response.rego, soc/response.py	Allowlisted reaction and target, blast radius cap, proportionality guard. OPA when EAI_OPA_URL is set, else the same rules in process (a test keeps both identical)
Kill Switch Registry	eai.kill_switches (PostgreSQL)	Source of truth: desired_state SUSPENDED or RELEASED, observed_state PENDING, APPLIED_PENDING_RECEIPT, SUSPENDED_VERIFIED, VERIFICATION_FAILED, RELEASED
Enforcement Adapters	soc/adapters.py	One fixed code path per typed reaction, never a command
Containment Verifier	soc/jobs.py (job soc_verify)	Re-probes every active switch, re-applies a drifted containment, records a receipt each time
Recovery Controller	soc/recovery.py	RECOVER: two distinct approvers who are not the requester, then re-authorization of the target

Where a suspension is enforced

• Policy decision point (eai.policy.evaluate, every gateway, runtime, connector, model and tool decision): a decision whose subject, resource, run, session, agent, application, provider or tenant is suspended is denied before any engine runs, with the reason kill_switch:<REACTION>:<type>:<ref>:<id>. An OPA outage or a permissive tenant rule cannot lift it.
• Authentication (eai.auth.current_principal): REVOKE_CREDENTIAL (401 credential_revoked, API keys are also revoked in eai.api_keys), SUSPEND_IDENTITY (403 identity_suspended, every credential type including a still valid JWT), SUSPEND_TENANT (403 tenant_suspended; a platform administrator still enters for the recovery).
• Agents runtime: steps and sub-agents call eai.policy.kill_switch_guard first; the refusal is 403 policy_denied with a recorded decision_id and the kill_switch_ids. CANCEL_RUN and CANCEL_SUBAGENTS cancel the runs; SUSPEND_AGENT suspends the agent and POST /v1/agents/{id}/resume answers 409 kill_switch_active until the recovery.
• Cluster (ISOLATE_NETWORK, TERMINATE_WORKLOAD): the order renders a CiliumNetworkPolicy (ingress and egress deny) or a scale to zero patch for the GitOps path; the switch stays APPLIED_PENDING_RECEIPT until an in-cluster reporter posts a signed receipt to POST /v1/soc/receipts.

Everything is read from PostgreSQL on every decision: a restart of any replica never resumes a suspended target (test_kill_switches_persist_across_restart).

Proportionality guard (criteria defined in advance)

Values in deploy/policies/opa/data.json (soc_response), mirrored by soc/response.py:

• automatic only for BLOCK_REQUEST, RATE_LIMIT, REQUIRE_APPROVAL, CANCEL_RUN, CANCEL_SUBAGENTS, SUSPEND_AGENT, DISABLE_TOOL, QUARANTINE_CONNECTOR, REVOKE_CREDENTIAL, DISABLE_MODEL_ROUTE, with a confidence of at least 0.9 and a blast radius of at most 5;
• a manual order above a blast radius of 25 waits for a second person;
• EMERGENCY_LOCKDOWN and SUSPEND_TENANT always wait for a second person;
• an order above its max_blast_radius is refused (409 containment_refused, reason blast_radius_exceeded);
• OPA unreachable: the in-process rules decide and no order is automatic (reason policy_engine_unavailable).

The blast radius is counted, never guessed: an agent is 1 plus its active runs, a run is 1 plus its active sub-agents, a tenant is 1 plus its members and agents.

Planes kept apart (SECU-021)

• Alerting (SIEM forwarding, Alertmanager) is never on the enforcement path: a SIEM down does not stop an incident or a containment.
• Enforcement does not depend on OPA: the Kill Switch Registry is read before any engine.
• Audit is written in the same transaction as each change: if the audit write fails, the containment is rolled back with it, never applied unaudited.

Roles and access

The SOC roles are set by eai.domains.soc.catalog: SOC analyst security_officer or soc_analyst (role added by migration V161; triage, orders, recovery requests, evidence sealing); SOC lead security_officer or tenant_admin (approvals, recovery, reconciliation); SOC console readers security_officer, tenant_admin, auditor, soc_analyst; evidence readers security_officer, tenant_admin, auditor; detection reporters security_officer, agent, soc_analyst (scope soc:detections). Four eyes always means two distinct people. Each read of the SOC consoles is audited (soc.access, soc.evidence.access), each refusal too (soc.access.deny, which the scanner turns into a privilege_escalation detection).

Evidence and chain of custody (AUDI-017)

Every detection, order, receipt and recovery of an incident is sealed in eai.incident_evidence (content hash, previous hash, chain per incident). Every seal, read and export is a eai.incident_custody entry (second chain: who, when, what). Both tables refuse UPDATE and DELETE (trigger). GET /v1/soc/incidents/{id}/evidence and /custody recompute both chains. POST /v1/soc/incidents/{id}/evidence/export scans the bundle for credentials first: 422 secret_detected (type and path only, never the value) and a secret_leak detection.

Transaction safety (SECU-023, RELE-007)

Side effecting calls use eai.domains.soc.transactions.execute with an idempotency key and a transaction id: kill switch check, durable intent (in_flight) committed before the call, then the result. A retry with the same key returns the stored result and never calls the provider again. A containment that cancels a run or suspends an agent marks its in flight calls interrupted: the SOC lead reconciles them (POST /v1/soc/transactions/{id}/reconcile, outcome committed or not_executed) with the provider reference; a blind retry is refused (409 transaction_needs_reconciliation).

Runbook: contain

1. Read the incident: GET /v1/incidents/{id} (timeline, risk reasons), GET /v1/soc/detections?incident_id=.
2. Pending automatic orders: GET /v1/soc/orders?status=pending_approval; approve with a second person: POST /v1/soc/orders/{id}/approve, or reject.
3. Manual containment: POST /v1/soc/orders with the narrowest reaction and target, and a max_blast_radius.
4. Check the proof: the order carries its switch; observed_state must be SUSPENDED_VERIFIED. For ISOLATE_NETWORK and TERMINATE_WORKLOAD, commit manifest to the GitOps repository; the state becomes verified only with the signed in-cluster receipt.
5. Re-probe at any time: POST /v1/soc/kill-switches/{id}/verify. VERIFICATION_FAILED means the effect is not proven: escalate.
6. In flight transactions: GET /v1/soc/transactions?status=interrupted, reconcile each one with the provider.

Runbook: recover (four eyes)

1. Request: POST /v1/soc/recoveries {kill_switch_id, reason}.
2. Two SOC leads, both different from the requester, approve: POST /v1/soc/recoveries/{id}/approve.
3. One of them re-authorizes: POST /v1/soc/recoveries/{id}/reauthorize. The switch is released, the target is re-evaluated by the policy decision point (403 reauthorization_denied keeps the switch), then restored (an agent is resumed). A revoked API key stays revoked: issue a new key.
4. Close the incident: PATCH /v1/incidents/{id} {status: resolved} then closed; export the evidence if required.

Settings

Setting	Default	Purpose
EAI_SOC_WEBHOOK_SECRET	empty	HMAC secret of the SOC webhooks and of the in-cluster receipts (empty: every delivery refused). 32 characters minimum in prod (checked at startup)
EAI_SOC_CORRELATION_WINDOW_SECONDS	900	Correlation window of the event correlator (60 to 86400, checked at startup in prod)
EAI_WORKER_SOC_SCAN_SECONDS	30	Interval of the soc_scan worker job
EAI_WORKER_SOC_VERIFY_SECONDS	120	Interval of the soc_verify worker job (Containment Verifier)

Falco and Tetragon: BLOCKED without a cluster

deploy/runtime-security/falco/eai-platform-rules.yaml (shell, secret mount read, public egress of agents, privilege escalation, binary drift in the sandbox) and deploy/runtime-security/tetragon/eai-tracing-policies.yaml (observe in enterprise-ai, Sigkill in eai-sandbox) are validated offline only (test_soc_runtime_rules.py: structure, references, namespaces, mapping to the detection kinds). No kubeconform schema exists offline for them. Their runtime effect (an alert raised by a real syscall, a process killed) needs a real cluster with Falco and Tetragon: BLOCKED (owner: platform operator of the target cluster; exit criterion: a test alert of each rule received by POST /v1/soc/webhooks/falco and a Sigkill measured by Tetragon). Falcosidekick and the OpenSearch alerting webhook do not compute the X-EAI-Signature HMAC themselves: an in-cluster signing relay is a deployment prerequisite, and the in-cluster receipt reporter of the Cilium and workload adapters as well; both are BLOCKED with the cluster.

Typed reactions and targets

The allowed reactions and targets are closed lists of eai.domains.soc.catalog; an order outside the list is refused. RECOVER is not an order: it is the four eyes recovery workflow.

Reaction	Effect	Automatic possible
BLOCK_REQUEST	Refuses the targeted request	yes
RATE_LIMIT	Rate limits the target	yes
REQUIRE_APPROVAL	Requires a human approval before the action	yes
CANCEL_RUN	Cancels the run	yes
CANCEL_SUBAGENTS	Cancels the sub-agents of a run	yes
SUSPEND_AGENT	Suspends the agent until recovery	yes
SUSPEND_APPLICATION	Suspends a governed application	no
DISABLE_TOOL	Disables a tool	yes
QUARANTINE_CONNECTOR	Quarantines a connector	yes
REVOKE_CREDENTIAL	Revokes a credential (401 credential_revoked)	yes
DISABLE_MODEL_ROUTE	Disables a model route	yes
ISOLATE_NETWORK	CiliumNetworkPolicy denying ingress and egress (GitOps, signed receipt)	no
TERMINATE_WORKLOAD	Scale to zero replicas (GitOps, signed receipt)	no
SUSPEND_IDENTITY	Suspends an identity, whatever the credential (403 identity_suspended)	no
SUSPEND_TENANT	Suspends the tenant (403 tenant_suspended), always with a second person	no
EMERGENCY_LOCKDOWN	Emergency lockdown, always with a second person	no

Targets: request, session, run, agent, subagent, application, tool, plugin, mcp, connector, knowledge_base, model, credential, identity, tenant, workload, service. The full catalogue is read with GET /v1/soc/catalog.

Read the Kill Switch Registry

GET /v1/soc/kill-switches lists every kill switch with desired_state and observed_state. An active kill switch whose observed state is not SUSPENDED_VERIFIED is not proven: run the verification again (POST /v1/soc/kill-switches/{switch_id}/verify) and, for a cluster containment, check that the signed receipt arrived (POST /v1/soc/receipts).

Related pages

• Approval workflows and four eyes rules: approvals.md.
• Evidence, seals and chain of custody: audit-evidence.md, risk-compliance.md.
• Blocked files turned into detections: file-security-pipeline.md.
• Governance overview: governance.md.