Skip to main content

ADR-0157: Bounded span summary in PostgreSQL and SLO objectives in the database

Context​

The architecture reference keeps large traces in Opik or Jaeger, logs in OpenSearch and files in the client object storage (A6.1-23); PostgreSQL keeps trace references only (trace_refs, A6.1-18) and OpenTelemetry traces are not copied into it (B7.1-18). The observability domain also names the service_slos table.

Before this decision, eai.telemetry_spans (V065) received every full span: all its attributes, with no list and no size limit, so also a prompt (gen_ai.prompt), a model output (gen_ai.completion) or tool arguments sent by an SDK. SLO objectives only existed in configuration (EAI_SLO_*) and in memory.

Decision​

  1. eai.telemetry_spans becomes a bounded summary of each span, not a copy of the trace:
    • trace reference (trace_id, span_id, parent_span_id), run, session, service, name (at most 256 characters), duration, status, provider cost, input and output tokens (input_tokens, output_tokens);
    • only the attributes of the ALLOWED_ATTRIBUTES allow list of services/control-plane/src/eai/domains/observability/telemetry.py (run identifiers, model, token counts, cost, HTTP method and route, error type), scalar values only, each string cut to 128 characters, the whole under 2048 bytes; attributes_dropped counts the attributes left out;
    • no prompt, no model output, no business content enters PostgreSQL. The telemetry_spans_summary_bounded constraint (V169) refuses in the database a summary larger than 4096 bytes or a name longer than 256 characters; V169 reduced the existing rows to the same list.
  2. The full span, with every attribute, goes to the OpenTelemetry collector (EAI_OTEL_EXPORTER_OTLP_ENDPOINT, then Jaeger or Opik), as before. GET /v1/telemetry/overview and GET /v1/telemetry/traces/{trace_id} read the summary and give the Opik link to the full trace.
  3. The summary does not live longer than needed: the tenant telemetry retention policy purges it, and the retention worker job applies a ceiling EAI_TELEMETRY_SPAN_MAX_DAYS to every tenant (90 days by default, 1 to 3650).
  4. eai.service_slos (V169) carries the objectives of each platform service (availability, latency target, share of requests under the latency target). GET /v1/telemetry/slo reads them (30 second cache cleared by a write); EAI_SLO_* are only the defaults of a service without a row. PUT /v1/telemetry/slo/targets/{service} (platform_admin only, audited telemetry.slo.target.update) writes them. The table belongs to the installation: every authenticated context reads it, only the platform context writes it (forced RLS).

A bounded summary was preferred to references alone: the overview (requests, errors, p50 and p95, cost per day), FinOps cost per trace and the ClickHouse export need it without depending on Opik or Jaeger being available.

Consequences​

  • An attribute outside the list stays readable in Jaeger or Opik, no longer in PostgreSQL. Adding an attribute to the summary means changing ALLOWED_ATTRIBUTES and checking that it carries no content.
  • The Prometheus rules of deploy/observability/slo/eai-slo-rules.yaml keep the defaults: an objective changed in the database must also be reported there for the 30 day fleet view.

Evidence​

services/control-plane/tests/test_observability_data_model.py (bounded summary and full OTel export, database refusal, retention by policy and by ceiling, SLO read from the database, write RLS), services/control-plane/tests/test_data_model_mapping.py, services/control-plane/tests/test_observability_slo.py, services/control-plane/tests/test_data_quality_governance.py, services/control-plane/tests/test_tracing.py.