Skip to main content

Security incident runbooks

Français : ../runbooks.md

The events below are emitted by eai.security.security_event and written to the tenant audit chain. General operations runbooks are in operations/en/runbooks.md.

SSRF attempts (security.egress.denied)​

  1. Search security.egress.denied events (emitted by eai.integrations.outbound.deny_event): tenant, principal, redacted URL, reason.
  2. Identify the tool, MCP server or connector; a series of denials to 169.254.169.254, private addresses or unusual ports means an attack.
  3. Disable the resource, revoke the principal's API key if needed (see below).
  4. Restrict the tenant with egress_allowed_hosts (PUT /v1/security/settings).
  5. Never widen EAI_EGRESS_PRIVATE_ALLOWLIST to "unblock" a tenant: this list is for platform calls only.

Prompt injection campaign​

  1. Follow security.prompt_injection.detected and security.prompt_injection.blocked (LLM gateway and knowledge ingestion) and security.output.link_stripped.
  2. If the tenant is in detect, set prompt_injection_mode to block.
  3. For indirect injection, find the document or source (knowledge base, tool result) and remove or reindex it.
  4. Keep untrusted_content_wrapping=true and output_links=strip_untrusted.
  5. Add the observed payloads to the garak and promptfoo corpora (deploy/ci/promptfoo.yaml).

Brute force and credential stuffing​

  1. Follow security.auth.failed (per IP) and security.rate_limited.
  2. Above EAI_RATE_LIMIT_AUTH_FAILURES_PER_MINUTE failures per minute the IP gets 429 (check_auth_lockout).
  3. Check that EAI_TRUSTED_PROXIES is right: otherwise every request seems to come from the gateway.
  4. Block hostile ranges at the WAF (waf-gateway), enable MFA at the IdP.

Leaked API key​

  1. Rotate the key: POST /v1/api-keys/{key_id}/rotate (the new value is shown once), or revoke it: DELETE /v1/api-keys/{key_id}.
  2. Search the audit for every action of the key since the leak.
  3. Run gitleaks on the repository or the leak location, purge history if needed.
  4. If the key gave access to tool or connector secrets, rotate them too.

WAF false positive​

Follow waf-gateway: find the CRS rule in the Coraza log, add the narrowest exclusion (route and parameter) in deploy/apisix/apisix.yaml, never a global disable.

Rate limit tuning​

  1. Measure 429 per bucket in security.rate_limited.
  2. Tune EAI_RATE_LIMIT_PER_MINUTE or EAI_RATE_LIMIT_GATEWAY_PER_MINUTE in Helm controlPlane.config.
  3. With several replicas, set EAI_REDIS_URL (rediss://) for shared counters; without Redis every replica counts on its own.

Compromised tenant tool or MCP server​

  1. Disable the tool or MCP server, revoke its OAuth tokens and rotate its secrets in OpenBao.
  2. Review security.egress.denied and the audit of the tool calls.
  3. Restrict egress_allowed_hosts, check that results were treated as untrusted (wrapping on).

CSP violation​

  1. Read the browser violation report: directive, blocked source.
  2. An inline script without nonce or an unknown domain means XSS or an extension: do not widen the CSP without analysis.
  3. A legitimate resource is added in buildCsp (portal/src/lib/security.ts), never with unsafe-inline.

Vulnerable dependency found by CI​

Jobs pip-audit, npm-audit, trivy-fs, trivy-image (GitLab) or security-deps-python, security-deps-node (Forgejo) fail. Upgrade the dependency, otherwise document a dated exception. Details: ci-security-gates.

Key rotation in OpenBao​

  1. Write the new version (bao kv put under kv/eai/prod/control-plane), following controlPlane.secretFormats.
  2. Let External Secrets sync, then restart the Control Plane.
  3. For data encryption keys: POST /v1/encryption/keys/{key_id}/rotate.
  4. Revoke the old version once the service is checked. Details: data-services-hardening.

Audit chain verification failure​

  1. GET /v1/audit/verify (eai.domains.observability.audit_api.verify) reports a break.
  2. Freeze privileged write access, keep a copy of the database.
  3. Compare with the last signed export (verify_export) to date the tampering.
  4. Handle it as an insider or database compromise incident.

File blocked by the scan (security.file.blocked)​

  1. Read the scan: GET /v1/file-security/scans?status=blocked, then GET /v1/file-security/scans/{scan_id} (verdict, reasons, engine, source, author).
  2. malware or suspicious: the SOC received a detection; check the open incident and the origin workstation or source.
  3. Suspected false positive: scan the retained sample again (POST /v1/file-security/scans/{scan_id}/rescan) after the signature update; the original upload is never released, the user uploads the file again.
  4. A family of legitimate files blocked by a YARA-X rule: publish a rules version where that rule carries action = "report", never disable an engine.
  5. Purge the sample once the analysis is closed: DELETE /v1/file-security/scans/{scan_id}/sample. Details: file-security-pipeline.md.

Scanning engines unavailable (security.file.scan_unavailable)​

  1. GET /v1/file-security/status: engine, endpoint, signature age.
  2. Uploads are refused (scan_error verdict, fail closed): this is intended. Restore the ClamAV gateway or the YARA-X worker, check the Bearer key and the CA (EAI_FILE_SCAN_CA_FILE).
  3. Never set EAI_FILE_SCAN_MODE back to dev in production: the startup guard refuses it.

Suspicious reveal in the IT vault (it_vault.reveal)​

  1. Search the it_vault.reveal events in GET /v1/audit/events: who, when, which service.
  2. An unexplained reveal counts as a leak: rotate the service secret at its source (OpenBao, External Secrets), then update the entry (PATCH /v1/platform-manager/vault/entries/{entry_id}).
  3. Check the platform_admin members (GET /v1/members) and the groups of EAI_OIDC_PLATFORM_ADMIN_GROUPS in the directory. Details: secrets-vault.md.

Break glass grant open​

  1. List the grants: GET /v1/break-glass, then GET /v1/break-glass/{grant_id} (requester, beneficiary, roles, justification, approvers, expiry).
  2. Check that the justification matches a real incident; otherwise revoke: POST /v1/break-glass/{grant_id}/revoke. Delegations created under the grant never outlive it.
  3. Review in the audit trail every action taken during the grant (events whose delegation.break_glass_id is the grant id). Details: identity-context-delegation.md.

Repeated separation of duties denials​

  1. Search approval.decide with decision deny and reason separation_of_duties: a requester trying to approve its own request.
  2. The same principal insisting, or an agent API key trying to approve, points to a bypass attempt: suspend the identity with a SUSPEND_IDENTITY SOC order (soc-runbook.md). Details: approvals.md.