Skip to main content

G-05 Metrics

Brick G of the Enterprise AI Platform. Technology: Prometheus (kube-prometheus-stack), ServiceMonitors, recording rules. French version: ../../fr/bricks/G-05-metrics.md.

Role and responsibilities​

  • Collect RED metrics of every service, GPU metrics (DCGM), queue depths, SLO burn rates.

Position and interactions​

  • Execution path: pull scrape.
  • Control Plane exposes /metrics; recording rules feed the portal screens (G-07) and Alertmanager (G-08); KEDA reads queue metrics (H-05).

Open source and developed parts​

  • Open source reused: Prometheus, kube-state-metrics, node-exporter, DCGM exporter.
  • Developed in the platform: eai_* metric catalog with bounded label cardinality (no user ids as labels).

Data model​

Control Plane tables (schema eai): eai.gpu_samples.

Every tenant table has tenant_id uuid NOT NULL and a row level security policy on current_setting('app.tenant_id', true)::uuid. ClickHouse usage_events is the analytics replica of eai.usage_events (ReplacingMergeTree keyed by event_id).

Contract: contracts/v1/usage-event.schema.json (schema_version 1.0). Usage is always reported with contracts/v1/usage-event.schema.json, audit with contracts/v1/audit-event.schema.json.

API endpoints​

MethodPath
GET/v1/telemetry/overview
GET/readyz

GET /v1/telemetry/overview​

Response:

{
  "window": "24h",
  "requests": 18234,
  "error_rate": 0.012,
  "p95_latency_ms": 2140,
  "tokens": {
    "input": 9120044,
    "output": 1203400
  },
  "top_models": [
    {
      "model_id": "gpt-4o-mini",
      "share": 0.61
    }
  ]
}

GET /readyz​

Response:

{
  "status": "ready",
  "checks": {
    "postgres": "ok",
    "redis": "ok",
    "clickhouse": "ok"
  }
}

SLO metrics​

eai_http_request_duration_seconds{method,route} and eai_http_requests_total feed the recording rules deploy/observability/slo/eai-slo-rules.yaml (p50, p95, p99, availability, budget remaining) and the SLO and error budgets screen of the portal (/observability/slo). See docs/operations/en/slo.md.

SDK examples​

Python​

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_live_xxx", tenant="acme", locale="en")
client.telemetry.overview(days=1)
client.system.readyz()

TypeScript​

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY, tenant: "acme", locale: "en" });
await client.telemetry.overview({ days: 1 });
await client.meta.ready();

Java​

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .tenant("acme")
    .locale(Locale.ENGLISH)
    .build();
client.telemetry().overview(Map.of("days", 1));
client.meta().ready();

Go​

client, err := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")), eai.WithTenant("acme"), eai.WithLocale("en"))
if err != nil { log.Fatal(err) }
ctx := context.Background()
_, err = client.Telemetry.Overview(ctx, eai.WithQuery(map[string]string{"days": "1"}))
_, err = client.Meta.Ready(ctx, nil)

Configuration​

Environment variables: EAI_OTEL_ENABLED, EAI_OTEL_EXPORTER_OTLP_ENDPOINT (OTLP/HTTP, collector port 4318, /v1/traces appended), EAI_OTEL_EXPORTER_OTLP_CERTIFICATE (internal CA for an https collector), EAI_LOG_LEVEL (DEBUG, INFO, WARNING, ERROR or CRITICAL).

Traces: with EAI_OTEL_ENABLED=1 the Control Plane builds a TracerProvider (service.name, service.version, deployment.environment), instruments FastAPI (except /metrics, /healthz, /readyz) and adds spans without personal data: authentication, policy, DLP, routing, provider call, tool, MCP, RAG and multimodal. A missing OpenTelemetry package refuses the start. traceparent is propagated to outbound calls and the trace_id is stored in audit events.

Logs: one JSON line per event on stdout (timestamp, level, logger, trace_id, span_id, request_id) for the API, the worker and the media worker. The eai.access line never holds the query string, Authorization, cookies or bodies.

Helm values keys (deploy/helm/enterprise-ai/values.yaml): global.monitoring.serviceMonitor, controlPlane.port.

Security​

  • /metrics only reachable from the monitoring namespace (Cilium policy).
  • Tenant resolved from the credential, never from the body; PostgreSQL row level security on every tenant table.
  • Every mutation and policy decision writes a hash chained audit event (G-11).
  • Errors use the standard envelope with message_i18n en and fr.

Observability​

  • Prometheus self metrics and TSDB size.
  • W3C traceparent propagated, X-Request-Id returned, spans carry eai.tenant_id, eai.project, eai.agent_id, eai.run_id.

FinOps units​

  • Metrics are operational signals, not invoices; FinOps uses gpu_samples derived from DCGM through H-04.

Failure modes and resilience​

  • High cardinality -> relabeling drops the label and an alert fires.

Standalone adoption​

Existing Prometheus can federate the eai_* metrics.

Acceptance tests​

  1. eai_http_requests_total and eai_gateway_tokens_total present after a gateway call.

Implementation roadmap