Monitoring and alerting

This page describes the platform Prometheus signals, the alert rules and how the Control Plane receives alerts.

Metrics

Source	Exposition	Families
API (eai.main)	GET /metrics, bearer token EAI_METRICS_TOKEN	eai_http_*, eai_gateway_requests_total, eai_gateway_tokens_total, eai_gateway_cost_total, eai_policy_decisions_total, eai_quota_rejections_total, eai_security_events_total, eai_alerts_received_total
Worker (python -m eai.worker)	port EAI_WORKER_METRICS_PORT (9464)	eai_worker_job_duration_seconds, eai_worker_job_failures_total, eai_worker_job_last_success_timestamp_seconds, eai_worker_last_tick_timestamp_seconds, global gauges
Media worker (python -m eai.media_worker)	port EAI_MEDIA_WORKER_METRICS_PORT (9465)	eai_media_worker_jobs_total, eai_worker_last_tick_timestamp_seconds{worker="media_worker"}

Model Gateway counters are fed by usage recording (gateway.record) and failures (gateway.audit_failure). Every recorded policy decision increments eai_policy_decisions_total, every quota rejection eai_quota_rejections_total.

Global gauges are computed from the database by the worker job gauges (the API has several replicas): eai_runs_active, eai_heartbeat_stale_runs, eai_kill_switches_active{kind}, eai_media_queue_depth, eai_media_queue_oldest_age_seconds. The job verify_chain (every EAI_WORKER_VERIFY_CHAIN_INTERVAL_SECONDS, one hour by default) recomputes the audit chain of every tenant and publishes eai_audit_chain_valid{tenant} and eai_audit_chain_broken_tenants. A broken chain writes the audit and security event audit.chain_verify_failed and forwards it to the SIEM.

The worker job label is job_name: job is reserved by Prometheus for the target.

Access to /metrics

Outside dev and test, EAI_METRICS_TOKEN (32 characters minimum) is required by the startup check. Without the header Authorization: Bearer <token>, GET /metrics answers 401. In the Helm chart:

• the ServiceMonitor reads the token from the secret global.monitoring.metricsTokenSecret (default: the Control Plane secret, key EAI_METRICS_TOKEN);
• a PodMonitor scrapes the worker on the metrics port;
• the CiliumNetworkPolicies open these ports to the monitoring namespace only.

Alert rules

Rules live in deploy/observability/prometheus/rules/eai-slo-cost.yaml and their copy deploy/observability/k8s/prometheusrule-eai.yaml. The group eai-operations adds:

Alert	Condition
EaiAuditChainBroken	at least one tenant with a broken audit chain
EaiAuditChainNotVerified	no successful verification for 3 hours
EaiKillSwitchActive	a provider, model or tenant kill switch active for 5 minutes
EaiMediaQueueBacklog	more than 100 queued media jobs or the oldest older than 15 minutes
EaiWorkerDown	worker missing or last tick older than 5 minutes
EaiWorkerJobFailing	more than 2 failures of a job in 30 minutes
EaiSecurityEventsSpike	security event rate three times above the daily average

tests/test_platform_metrics.py checks that every metric used by rules and dashboards is a declared family, and tests/test_observability_signals.py that every eai.metrics helper is called by the code.

Receiving alerts

Alertmanager sends every alert (receiver default included) to POST /v1/incidents/alerts with a service API key explicitly granted the scope incidents:alerts (or incidents:*; the global wildcard * of an interactive session does not count, so a viewer cannot open an incident) or the role alert_receiver, read through http_config.authorization.credentials_file from the secret eai-alert-receiver (key api-key).

• A firing alert opens an incident of category alert or updates the open incident with the same fingerprint.
• A resolved alert resolves that incident.
• Every change is audited (incident.alert.open, incident.alert.update, incident.alert.resolve) and forwarded to the SIEM.

Tenant: alerts land in the tenant of the key (operations tenant), so platform alerts without a tenant label always land there. A tenant=<uuid> label is honoured only when the key belongs to a platform administrator.

Broken audit chain

1. Find the tenant in eai_audit_chain_valid == 0 or in the open incident.
2. GET /v1/audit/verify on that tenant gives the sequence and the reason (hash mismatch, prev_hash mismatch, sequence gap).
3. Handle it as a security incident: freeze database write access and compare with the WORM export.