Skip to main content

G-01 Distributed tracing

Brick G of the Enterprise AI Platform. Technology: OpenTelemetry SDKs and semantic conventions (gen_ai.*), W3C traceparent propagation. French version: ../../fr/bricks/G-01-distributed-tracing.md.

Role and responsibilities​

  • Give one trace id to a whole request: portal or SDK -> Control Plane -> gateway -> provider -> tools -> async workers.
  • Carry eai.* attributes (tenant, project, agent, run, task, decision id) on every span.

Position and interactions​

  • Execution path: every path (realtime, batch, event, workflow).
  • traceparent accepted on every API call and copied into NATS headers and Temporal memo for async paths.
  • X-Request-Id returned and stored in Usage and Audit events for correlation (G-14).

Open source and developed parts​

  • Open source reused: OpenTelemetry SDKs (Python, JS, Java, Go), auto-instrumentation for FastAPI, httpx, psycopg.
  • Developed in the platform: eai.* attribute convention, propagation through NATS and Temporal, SDK helpers.

Data model​

Control Plane tables (schema eai): eai.telemetry_spans.

Every tenant table has tenant_id uuid NOT NULL and a row level security policy on current_setting('app.tenant_id', true)::uuid. ClickHouse usage_events is the analytics replica of eai.usage_events (ReplacingMergeTree keyed by event_id).

Contract: contracts/v1/execution-contract.schema.json (schema_version 1.0). Usage is always reported with contracts/v1/usage-event.schema.json, audit with contracts/v1/audit-event.schema.json.

API endpoints​

MethodPath
POST/v1/telemetry/spans
GET/v1/telemetry/traces/{trace_id}

POST /v1/telemetry/spans​

Request:

{
  "spans": [
    {
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
      "span_id": "00f067aa0ba902b7",
      "name": "gen_ai.chat",
      "kind": "client",
      "start_time": "2026-10-29T10:00:00.120Z",
      "end_time": "2026-10-29T10:00:01.940Z",
      "attributes": {
        "gen_ai.request.model": "gpt-4o-mini",
        "gen_ai.usage.input_tokens": 812,
        "eai.agent_id": "agt_ap",
        "eai.run_id": "run_42"
      }
    }
  ]
}

Response:

{
  "accepted": 1,
  "rejected": 0
}

GET /v1/telemetry/traces/{trace_id}​

Response:

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "spans": [
    {
      "span_id": "00f067aa0ba902b7",
      "name": "gen_ai.chat",
      "duration_ms": 1820
    }
  ],
  "links": {
    "opik": {
      "url": "https://<opik-host>/default/redirect/projects?name=eai-platform",
      "metadata_key": "eai.trace_id",
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
    }
  }
}

Trace propagation (HTTP and event queue)​

  • HTTP: the API server span continues the received W3C traceparent; the Model Gateway (eai.llm.route, eai.llm.provider_call) and the Tool Gateway (eai.tool.invoke) open child spans and the outbound call receives the traceparent of the current span.
  • Queue: scheduler.publish opens eai.event.publish and writes traceparent and tracestate into the event headers (NATS or eai.event_outbox). The worker outbox job opens eai.event.deliver as a child of that traceparent, then eai.event.dispatch for the triggers.
  • The worker (python -m eai.worker) initialises OpenTelemetry with the same EAI_OTEL_* settings as the API.
  • Evidence: services/control-plane/tests/test_trace_end_to_end.py with the real otelcol-contrib. Details: docs/operations/en/slo.md.

SDK examples​

Python​

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_live_xxx", tenant="acme", locale="en")
client.telemetry.ingest(spans=spans)
client.telemetry.trace(trace_id="4bf92f3577b34da6a3ce929d0e0e4736")

TypeScript​

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY, tenant: "acme", locale: "en" });
await client.telemetry.ingestSpans({ spans: spans });
await client.telemetry.trace({ traceId: "4bf92f3577b34da6a3ce929d0e0e4736" });

Java​

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .tenant("acme")
    .locale(Locale.ENGLISH)
    .build();
client.telemetry().ingestSpans(Map.of("spans", spans));
client.telemetry().trace("4bf92f3577b34da6a3ce929d0e0e4736");

Go​

client, err := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")), eai.WithTenant("acme"), eai.WithLocale("en"))
if err != nil { log.Fatal(err) }
ctx := context.Background()
_, err = client.Telemetry.IngestSpans(ctx, map[string]any{"spans": spans})
_, err = client.Telemetry.Trace(ctx, "4bf92f3577b34da6a3ce929d0e0e4736")

Configuration​

Environment variables: EAI_OTEL_EXPORTER_OTLP_ENDPOINT.

Helm values keys (deploy/helm/enterprise-ai/values.yaml): controlPlane.config.EAI_OTEL_EXPORTER_OTLP_ENDPOINT, global.monitoring.

Security​

  • No prompt or completion content in OTel attributes by default; content goes to Opik with redaction (G-03).
  • Tenant resolved from the credential, never from the body; PostgreSQL row level security on every tenant table.
  • Every mutation and policy decision writes a hash chained audit event (G-11).
  • Errors use the standard envelope with message_i18n en and fr.

Observability​

  • Self monitoring: span drop counters in the SDK exporters.
  • W3C traceparent propagated, X-Request-Id returned, spans carry eai.tenant_id, eai.project, eai.agent_id, eai.run_id.

FinOps units​

  • Trace volume per tenant measured to allocate observability storage cost.

Failure modes and resilience​

  • Collector down -> SDK buffers then drops spans, never blocks the business call.

Standalone adoption​

Any client app can send spans to POST /v1/telemetry/spans or directly to its own collector.

Acceptance tests​

  1. A run calling a tool through MCP shows a single trace with gateway, tool and model spans.
  2. A NATS triggered job keeps the parent trace id.

Implementation roadmap