Skip to main content

Runbooks

French version: ../runbooks.md.

Each runbook gives the symptom (alert or signal), the diagnosis, the steps and the verification. The alert names are those of deploy/observability/prometheus/rules/eai-slo-cost.yaml (copy deploy/observability/k8s/prometheusrule-eai.yaml), of the chart (templates/control-plane/monitoring.yaml) and of the Wazuh rules deploy/wazuh/rules/eai_audit_rules.xml. Their runbook_url points to the headings below: keep the heading names.

RunbookAlerts and signals
provider-outageEaiProviderFailing, EaiGatewayErrorRatio, EaiSloFastBurn, EaiControlPlaneHighErrorRate
cost-spikeEaiCostSpike, EaiQuotaRejectionsHigh
key-leakWazuh 100805, secret scanner, notice of a provider
prompt-injectionWazuh 100804, EaiSecurityEventsSpike
gpu-saturationEaiGpuSaturated, EaiVllmTtftSlo, EaiGpuXidError, EaiGpuMemoryFull
control-plane-downEaiControlPlaneDown, EaiControlPlaneHighLatency, EaiControlPlaneLatencySlo
worker-downEaiWorkerDown, EaiWorkerJobFailing, EaiStaleHeartbeats
media-backlogEaiMediaQueueBacklog
database-unavailable/readyz answers 503 with database: false
kill-switch-activeEaiKillSwitchActive, Wazuh 100801
Audit chainEaiAuditChainBroken, EaiAuditChainNotVerified: Alerting
MCP RuntimeEaiMcpRuntimeFailures, EaiMcpRuntimeInstancesFailed, EaiMcpCatalogStale, EaiMcpCallErrorRatio: MCP Runtime

provider-outage​

Symptom: EaiProviderFailing (a provider fails above 50 percent for 5 minutes), EaiGatewayErrorRatio (more than 10 percent of gateway requests fail), EaiSloFastBurn or EaiControlPlaneHighErrorRate (5xx ratio above 2 percent).

Diagnosis

  1. Find the provider: label provider of the alert, Gateway dashboard, metric litellm_deployment_failure_responses_total.
  2. Tell a provider outage from a local fault: if every provider fails, check LiteLLM (kubectl -n enterprise-ai logs deploy/enterprise-ai-litellm) and the egress (hubble observe --namespace enterprise-ai --verdict DROPPED).

Steps

  1. LiteLLM fallbacks switch automatically (router_settings.fallbacks). Check that the fallback stays compliant (classification, region) in the OPA policies.
  2. If the outage lasts, disable the provider: POST /v1/providers/{pid}/disable (audited provider.disable, Wazuh 100801), or add it to kill_switch.providers of the OPA data. Tell the tenants.

Verification: the error ratio of EaiGatewayErrorRatio goes back under 10 percent. Recovery: POST /v1/providers/{pid}/enable, remove it from the kill switch, watch the error rate for 30 minutes.

cost-spike​

Symptom: EaiCostSpike (hourly gateway cost three times the 7 day average for 30 minutes) or EaiQuotaRejectionsHigh (more than 30 quota or budget rejections per minute).

Diagnosis

  1. FinOps dashboard: tenant, provider, model and API key behind the cost (eai_gateway_cost_total, eai_gateway_tokens_total).
  2. Look for an agent loop (agent_autonomy settings max_steps and max_depth of the OPA data) or a compromised key (calls from an unusual address in GET /v1/audit/events).

Steps

  1. Lower model_invocation.max_cost_per_request in the OPA data, or the budget of the tenant.
  2. Suspend the agent: POST /v1/agents/{agent_id}/suspend (/kill to stop it at once).
  3. Revoke a compromised key: DELETE /v1/api-keys/{key_id}; or set the tenant kill switch.

Verification: eai:gateway_cost:rate1h goes back to its usual level. Reconcile afterwards with the invoice: POST /v1/finops/reconciliation/run.

key-leak​

Symptom: Wazuh rule 100805 (credential revoked or rotated), an alert of the secret scanner, a notice of a provider.

Diagnosis: identify the credential (tenant API key, provider key, OpenBao token, Transit key) and the first use that is not legitimate in GET /v1/audit/events?actor=<id>.

Steps

  1. Revoke at once: tenant API key (DELETE /v1/api-keys/{key_id}, or POST /v1/api-keys/{key_id}/rotate), provider key in the console of the provider.
  2. Replace it in OpenBao (bao kv put kv/eai/prod/<path> ...); External Secrets resynchronizes the Secret (refresh one hour), or restart LiteLLM.
  3. Suspected Transit key: bao write -f transit/keys/<key>/rotate, then rewrap the data.
  4. Declare the incident (POST /v1/incidents); personal data involved: GDPR notification within 72 hours.

Verification: the old credential is refused (401); GET /v1/audit/verify shows a valid chain.

prompt-injection​

Symptom: Wazuh rule 100804 (audit actions security.prompt_injection.blocked or .detected), EaiSecurityEventsSpike for the type of the security event.

Diagnosis: find the run, the agent and the source (RAG document, tool, user) with GET /v1/audit/correlations?trace_id=<trace>.

Steps

  1. Suspend the agent (POST /v1/agents/{agent_id}/suspend).
  2. Quarantine the document (remove its ACL) or block the tool in the policy.
  3. Check that no irreversible action ran without approval (agent_autonomy policy, approval requests).
  4. Raise the guard of the tenant to block (prompt_injection_mode of the tenant settings, default EAI_PROMPT_INJECTION_DEFAULT_MODE).
  5. Add the case to the promptfoo and garak tests of the CI.

Verification: the same input is now refused (security.prompt_injection.blocked in the audit).

gpu-saturation​

Symptom: EaiGpuSaturated (GPU above 95 percent with more than 20 queued requests for 10 minutes), EaiVllmTtftSlo (time to first token p95 above 2 seconds), EaiGpuXidError, EaiGpuMemoryFull.

Diagnosis

  1. GPU dashboard: use, memory, queue vllm:num_requests_waiting per model.
  2. Did KEDA reach maxReplicas of the model? kubectl -n enterprise-ai get scaledobject. If not, are GPU nodes free? (GPU Operator, XID errors).

Steps

  1. Raise maxReplicas of the model in vllm.models, or add GPU nodes.
  2. Enable a LiteLLM fallback to a smaller model or to an allowed provider.
  3. Limit the rate of the tenant in APISIX.
  4. Repeated XID error: cordon the node (kubectl cordon <node>) and open a hardware ticket.

Verification: vllm:num_requests_waiting goes back under the threshold of vllm.keda.threshold; the alert resolves.

control-plane-down​

Symptom: EaiControlPlaneDown (no healthy target for 5 minutes), EaiControlPlaneHighLatency or EaiControlPlaneLatencySlo (p95 above 1 second).

Diagnosis

  1. Kubernetes: kubectl -n enterprise-ai get pods -l app.kubernetes.io/component=control-plane and the logs of a pod. Packages: systemctl status navin-platform-api and journalctl -u navin-platform-api -n 200.
  2. unsafe production settings: in the logs: a production rule is broken. password authentication failed or a timeout: see database-unavailable.
  3. curl -fsS <api-url>/readyz tells the state of the database and of the migrations.

Steps

  1. Fix the setting named by the log (Configuration reference), then restart (kubectl rollout restart deploy/enterprise-ai-control-plane, or systemctl restart navin-platform-api).
  2. High latency only: check the HPA (kubectl -n enterprise-ai get hpa) and the CPU limits; raise controlPlane.autoscaling.maxReplicas.

Verification: /readyz answers 200 ready; the alert resolves.

worker-down​

Symptom: EaiWorkerDown (no tick for 5 minutes), EaiWorkerJobFailing (more than 2 failures of a job in 30 minutes), EaiStaleHeartbeats (active runs with a stale heartbeat).

Diagnosis

  1. Is a worker running? kubectl -n enterprise-ai get pods -l app.kubernetes.io/component=control-plane-worker, or systemctl status navin-platform-worker.
  2. A worker that stays in activating or in a failed init: a migration is pending (python -m eai.migrate --check).
  3. A failing job: label job_name of eai_worker_job_failures_total, then the logs of the worker.

Steps

  1. Pending migration: apply it (navin-platform-db migrate, or the migration Job of the chart).
  2. Restart the worker. Only one replica is active (PostgreSQL advisory lock EAI_WORKER_LOCK_KEY): a second one waits.
  3. Stale heartbeats: once the worker ticks again, its heartbeats job flags the runs past the stale threshold (EAI_HEARTBEAT_STALE_SECONDS) and moves them to timed_out past EAI_WORKER_HEARTBEAT_TIMEOUT_SECONDS.

Verification: eai_worker_last_tick_timestamp_seconds{worker="worker"} is recent; eai_heartbeat_stale_runs goes back to 0.

media-backlog​

Symptom: EaiMediaQueueBacklog (more than 100 queued media jobs, or the oldest older than 15 minutes).

Diagnosis: eai_media_queue_depth and eai_media_queue_oldest_age_seconds; are media workers running (kubectl -n enterprise-ai get pods, systemctl status navin-platform-media-worker) and did KEDA scale them?

Steps

  1. Start or scale the media workers (maxReplicas of workers.items, lagThreshold).
  2. Jobs stuck in running: the media_requeue job requeues a job whose lease expired (EAI_MEDIA_JOB_LEASE_SECONDS) and fails it after EAI_MEDIA_JOB_MAX_ATTEMPTS claims.

Verification: the queue depth decreases; the alert resolves.

database-unavailable​

Symptom: /readyz answers 503 with "database": false; API and worker logs show connection errors.

Diagnosis

  1. Kubernetes: kubectl -n eai-data get cluster eai-postgres (CloudNativePG) shows the primary and the replicas. Packages: systemctl status postgresql and sudo navin-platform-db status.
  2. TLS errors: check the CA of sslrootcert and the sslmode of EAI_DATABASE_URL.

Steps

  1. CloudNativePG promotes a replica automatically; wait for the new primary, the Service eai-postgres-rw follows it.
  2. Disk full: grow the volume (storage.size of the cluster) or free the WAL archive.
  3. Data loss: restore (PostgreSQL, High availability and disaster recovery).

Verification: /readyz answers 200 with database: true and migrations: true.

kill-switch-active​

Symptom: EaiKillSwitchActive (a provider, model or tenant kill switch active for 5 minutes), Wazuh 100801 (provider.disable, model.disable, agent.kill, agent.suspend).

Diagnosis: label kind of eai_kill_switches_active; GET /v1/soc/kill-switches lists the switches and their verification receipts; the audit event names who set it.

Steps

  1. Confirm that the switch is intended (incident in progress, containment order of the SOC).
  2. Lift it through the same channel once the cause is fixed: POST /v1/providers/{pid}/enable, remove the entry of kill_switch in the OPA data, or the four eyes recovery of the SOC for a switch set by a containment order (POST /v1/soc/recoveries, then POST /v1/soc/recoveries/{recovery_id}/approve by a second person).

Verification: eai_kill_switches_active goes back to 0 for that kind.