SLO, error budgets and trace propagation

Kit sections 11 and 15, requirements OBSE-001, OBSE-005 and OBSE-012. French version: ../slo.md.

Targets

Service	Routes	Availability	Latency
control_plane_api	every other /v1 route	99.5 %	95 % of requests under 1 s
model_gateway	/v1/gateway/*	99.5 %	95 % under 1 s
tool_gateway	/v1/tools/*	99.5 %	95 % under 1 s
realtime	/v1/realtime*	99.5 %	95 % under 1 s
worker	jobs of python -m eai.worker	99 % successful jobs

The route targets are read from the eai.service_slos table (V169, ADR-0157), written by PUT /v1/telemetry/slo/targets/{service} (platform_admin only, body {availability, latency_seconds, latency_objective, reason?}, audited telemetry.slo.target.update, 404 slo_service_unknown); each service of the response says target_source: database | settings. A service without a row keeps the defaults set by EAI_SLO_AVAILABILITY_TARGETS, EAI_SLO_LATENCY_TARGETS_SECONDS, EAI_SLO_LATENCY_OBJECTIVE; the window is bounded by EAI_SLO_RETENTION_MINUTES (configuration.md). An invalid value is refused at production startup (eai.security.startup). The Prometheus rules carry the defaults above: changing a target also means updating deploy/observability/slo/eai-slo-rules.yaml and its promtool tests.

Budgets and burn rate

• Allowed budget = (1 - target) x requests of the window. Budget remaining = 1 - bad requests / allowed budget (negative once exceeded).
• Burn rate = bad ratio / (1 - target): 1 spends exactly the budget over the period.
• Bad request: 5xx status for availability, duration above the threshold for latency.

Burn rate alerts (multi window)

Alert	Condition	Severity
EaiSloAvailabilityFastBurn	error ratio > 14.4 x 0.5 % over 1 h and 5 min, for 2 min	critical
EaiSloAvailabilitySlowBurn	error ratio > 6 x 0.5 % over 6 h and 1 h, for 15 min	warning
EaiSloLatencyFastBurn	slow requests > 14.4 x 5 % over 1 h and 5 min, for 2 min	critical
EaiSloWorkerFastBurn	failed jobs > 14.4 x 1 % over 1 h and 5 min, for 2 min	critical

Checks: promtool check rules deploy/observability/slo/eai-slo-rules.yaml and promtool test rules deploy/observability/slo/tests/eai-slo-rules.test.yaml (tests/test_observability_slo.py).

Portal screen

The platform ships no dashboard server (ADR-0169). The budgets are read in the SLO and error budgets screen of the portal (/observability/slo, guide), fed by the route below; the fleet view over 30 days comes from the eai:* recording rules and the burn alerts, read in Prometheus and Alertmanager. A Grafana run by the client can read the same rules (EAI_GRAFANA_URL).

Route GET /v1/telemetry/slo

Parameters: window_minutes (1 to EAI_SLO_RETENTION_MINUTES, default 60) and tenant_id (platform_admin only). Roles: platform_admin (whole platform or one tenant), tenant_admin, auditor and viewer (their own tenant only). Every read is audited (telemetry.slo.read). Error codes: forbidden, tenant_required, slo_scope_forbidden, slo_window_invalid, slo_targets_invalid.

Response per service: requests, errors, over_latency_target, latency_target_seconds, then availability and latency with target, bad, ratio, budget_remaining and burn_rate.

Source: the API HTTP middleware counts every served request (route template, status, duration, caller tenant) in one minute buckets kept in memory by each replica. The route therefore gives the view of the replica that answers; the fleet view over 30 days comes from the Prometheus rules, computed on the same measurements (eai_http_requests_total, eai_http_request_duration_seconds).

Trace propagation

• HTTP: the API server span continues the received traceparent; eai.llm.route, eai.llm.provider_call and eai.tool.invoke are its descendants, and the upstream (LLM provider, tool) receives the traceparent of the current span.
• Queue: scheduler.publish opens eai.event.publish and writes traceparent and tracestate into the event headers (NATS or eai.event_outbox, media worker wake ups included). The outbox job opens eai.event.deliver as a child of that traceparent, then eai.event.dispatch, and audits written during delivery carry the same trace_id.
• The worker initialises OpenTelemetry with the API EAI_OTEL_* settings and flushes the exporter on exit.
• Evidence: services/control-plane/tests/test_trace_end_to_end.py with the real otelcol-contrib (deploy/observability/tools/fetch-toolchain.sh otelcol).