SIEM, mutation audit, WORM and erasure

Sending to the SIEM (siem_forward job)

The worker (python -m eai.worker) reads the hash chained audit trail (eai.audit_events) of every tenant after a watermark (eai.siem_cursors, migration V096) and sends the security events with eai.domains.incidents.siem.forward: RFC 5424 syslog (UDP or TCP) and an HTTP JSON webhook.

• Selection: EAI_SIEM_FORWARD_ACTIONS, comma separated patterns (prefix.* or an exact name, * for the whole trail). Default: security.* (authentication failures, prompt injections, egress denials, rate limiting), policy.*, incident.*, approval.*, secret.*, api_key.*, audit.*, dlp.*, member.*, authz.write, authz.delete, and the kill switches provider.disable, model.disable, agent.kill, agent.suspend. Denials (decision.result = deny) and failures (outcome = failure or error) are always sent.
• Resume: the watermark only moves past the events every target accepted. A failure stops the tenant batch, the error is recorded in eai.siem_cursors.last_error and eai.worker_jobs, and the next run resumes at the first event not sent. With several targets, a target already served may get the same event again (at least once per target); event_id allows deduplication.
• Bounds: EAI_SIEM_FORWARD_BATCH (500) events per tenant and per run, EAI_WORKER_SIEM_FORWARD_SECONDS (30) between runs.
• Format: <pri>1 <timestamp> <host> eai-control-plane - audit - {json}, the JSON starts with "schema":"eai.audit.v1" then carries the Audit Event v1 (action, outcome, decision, actor, tenant_id, resource, evidence, hash). Syslog severity: error for failures, denials and blocks, warning for security.*, incident.*, secret.*, audit.*, notice otherwise. The webhook receives {"event": ..., "sourcetype": "eai:audit"}.

Wazuh

Decision: the platform does not deploy Wazuh (neither in the chart nor in the ApplicationSet). It is the customer's SIEM, reached in TCP syslog through EAI_SIEM_WAZUH_HOST / EAI_SIEM_WAZUH_PORT / EAI_SIEM_WAZUH_PROTO=tcp. A SIEM is only worth something outside the perimeter it watches; running it in the same cluster would lose the trail of a compromise of that cluster. Other SIEMs (Splunk HEC, Elastic) use EAI_SIEM_WEBHOOK_URL.

Files in deploy/wazuh:

• manager-ossec.conf: remote syslog listener (<remote>) to merge into /var/ossec/etc/ossec.conf, with allowed-ips restricted to the worker egress range.
• decoders/eai_audit_decoders.xml: prematch eai-control-plane - <msgid> - followed by {"schema":"eai.audit.v1", then JSON_Decoder from the end of the prematch.
• rules/eai_audit_rules.xml (100800 to 100814): kill switches, failures and failure bursts per tenant, prompt injection, key revocation or rotation, audit chain break (audit.chain_verify_failed, emitted by the verify_chain job), access granted, rejected approval, authentication failures and brute force per address, egress denial, incident response, security configuration change, rate limiting.
• agent-ossec.conf: agent of the OpenBao hosts only (OpenBao file audit device).

tests/test_siem_audit_worm.py checks that every rule targets an action the code really emits and that the expected outcome values exist (success, failure, error).

Auditing every mutation

tests/test_audit_mutation_coverage.py lists every POST, PUT, PATCH and DELETE route and fails when one of them does not reach audit() (or security_event()) in its call graph. Exemptions are listed with their reason (read only routes: resolve, simulate, context/plan, render, validate; inbound webhook journaled in eai.webhook_deliveries).

High frequency routes:

Route	Event	Volume
POST /v1/runs/{run_id}/heartbeat	run.heartbeat	state changes only: first heartbeat of the run, or a stop answer
POST /v1/sessions/{session_id}/messages	session.message.add	one per message, role and length, never the content
POST /v1/finops/usage	finops.usage.ingest	one per call, without the payload
POST /v1/finops/usage/batch	finops.usage.ingest_batch	aggregated: one per batch with its size
POST /v1/quotas/reserve, /commit, /release	finops.quota.reserve, .commit, .release	one per call
POST /v1/finops/gpu/samples	finops.gpu.samples_ingest	aggregated: one per batch of samples

High throughput usage producers should use the batch route. Internal reservations of the LLM gateway do not go through these routes and add no event.

Periodic WORM export (audit_worm job)

Every complete UTC day of each tenant chain not exported yet gives a signed NDJSON and manifest, written with S3 object lock (EAI_AUDIT_WORM_RETENTION_DAYS, 365) under tenants/<tenant>/audit-worm/<export_id>. The export is recorded in eai.audit_exports and eai.audit_worm_days (a day is never exported twice) and audited (audit.export). At most 200 exports per run, the backlog continues on the next run. The logic is shared with POST /v1/audit/export (eai.domains.observability.audit_api.write_export).

Signature (EAI_AUDIT_SIGNING):

• transit: OpenBao Transit sign over the SHA-256 digest with the key EAI_OPENBAO_AUDIT_SIGNING_KEY (eai-audit-export, type ed25519 or ecdsa-p256), checked with transit/verify. The private key never leaves OpenBao. Creation: bao write transit/keys/eai-audit-export type=ed25519, control plane policy limited to transit/sign/eai-audit-export and transit/verify/eai-audit-export.
• hmac: HMAC-SHA256 derived from the master key, for development and tests only.
• auto (default): Transit when OpenBao is configured, HMAC otherwise.

In production the startup guard (eai/security/startup.py) refuses to start without EAI_OPENBAO_URL and EAI_OPENBAO_TOKEN, or with EAI_AUDIT_SIGNING=hmac.

Erasing a person

POST /v1/retention/erase also covers the external stores, reported under external:

• Opik: traces whose eai.subject_hash metadata is the pseudonymous hash of the person, listed with GET /v1/private/traces (metadata filter) then removed with POST /v1/private/traces/delete (eai/integrations/opik.py, at most 10,000 traces per request, truncated otherwise).
• ClickHouse: lightweight DELETE of the usage rows (user_id) and of the spans of the person's sessions; values travel as typed parameters. The PostgreSQL usage row is kept as a financial record.
• Status per store: not_configured, dry_run, completed or error. An error makes the request incomplete and the retention.erase event is written with outcome = failure: run the request again.
• A legal hold on the person blocks the whole request (blocked_legal_hold), external stores included.

Settings

Variable	Default	Purpose
EAI_SIEM_SYSLOG_HOST, _PORT, _PROTO		syslog collector
EAI_SIEM_WAZUH_HOST, _PORT, _PROTO		Wazuh manager syslog listener
EAI_SIEM_WEBHOOK_URL, _TOKEN, _AUTH_SCHEME		HTTP JSON webhook
EAI_SIEM_FORWARD_ACTIONS	see above	event selection
EAI_SIEM_FORWARD_BATCH	500	events per tenant and per run
EAI_WORKER_SIEM_FORWARD_SECONDS	30	siem_forward job interval
EAI_WORKER_AUDIT_WORM_SECONDS	3600	audit_worm job interval
EAI_AUDIT_WORM_RETENTION_DAYS	365	object lock duration
EAI_AUDIT_SIGNING	auto	auto, transit or hmac
EAI_OPENBAO_AUDIT_SIGNING_KEY	eai-audit-export	Transit signing key

Audit Gateway and Evidence Service (kit section 11)

• Worker jobs: audit_gateway (outbox to EAI_AUDIT_GATEWAY_HTTP_URL and NATS JetStream), audit_seal (Ed25519 seals), audit_integrity (recompute and alerts), audit_retention (purge of encrypted payloads outside legal holds). Settings: configuration.md.
• Fetch the test binaries: bash tools/audit/fetch.sh (nats-server 2.15.2, pinned sha256, into /tmp/toolchains).
• Offline verification: GET /v1/evidence/bundle then python -m eai.domains.evidence.verify bundle.json (exit 0: valid, 1: tampering).
• Integrity alert: GET /v1/evidence/alerts?status=open; never repair the row, open an incident and keep the evidence.
• Security model: ../../security/en/audit-evidence.md.