Platform integration: control plane and infrastructure

How the infrastructure (deploy/) is aligned with the control plane code (services/control-plane/src/eai). The backend code is the source of truth: on any mismatch the deployment side is fixed. Every point is checked by a services/control-plane/tests/test_platform_*.py test.

Background worker control-plane-worker

python -m eai.worker runs the periodic platform jobs: scheduler tick, event outbox drain, quota reservation expiry, stale heartbeat detection, retention. A PostgreSQL advisory lock keeps a single active instance.

• Helm: templates/control-plane/worker.yaml, enabled by worker.enabled: true (default). One replica, Recreate strategy, same image, ConfigMap, Secret (External Secrets or OpenBao agent) and ServiceAccount as the API. No PodDisruptionBudget.
• NetworkPolicy (Cilium or Kubernetes): no ingress; egress to PostgreSQL (5432) and NATS (4222) in the data namespace, OpenBao (8200), the OTLP/HTTP collector (4318) and DNS.
• Compose: control-plane-worker service, same variables as control-plane (YAML anchor x-control-plane-env), no published port.
• Monitoring: EaiStaleHeartbeats fires when eai_heartbeat_stale_runs stays above zero for 10 minutes (worker down or runtimes stuck).

Prometheus metrics

GET /metrics (not in the OpenAPI document) serves the prometheus_client text format. eai/metrics.py defines the families and safe helpers (no exception ever reaches the request):

Metric	Type	Labels	Source
eai_http_requests_total	counter	method, route, status	HTTP middleware
eai_http_request_duration_seconds	histogram	method, route	HTTP middleware
eai_gateway_requests_total	counter	provider, model, outcome	metrics.gateway_request()
eai_gateway_tokens_total	counter	type	metrics.gateway_tokens()
eai_gateway_cost_total	counter	currency	metrics.gateway_cost()
eai_policy_decisions_total	counter	result (allow, deny)	metrics.policy_decision()
eai_quota_rejections_total	counter	none	metrics.quota_rejection()
eai_runs_active	gauge	none	metrics.set_runs_active(), run_started(), run_finished()
eai_heartbeat_stale_runs	gauge	none	metrics.set_heartbeat_stale_runs()

The route label is the route template (/v1/tenants/{tenant_id}), never the raw path, so cardinality is bounded and no identifier leaks. Requests without a route use route="unmatched". *_created series are disabled.

/metrics is not authenticated: APISIX does not publish it and the control plane NetworkPolicy only accepts the gateway, the portal and the observability namespace.

Alignment:

• ServiceMonitor templates/control-plane/monitoring.yaml: selects app.kubernetes.io/component: control-plane, port http, path /metrics.
• Rules: deploy/observability/prometheus/rules/eai-slo-cost.yaml and its copy deploy/observability/k8s/prometheusrule-eai.yaml (identical, checked by a test), plus the chart alerts. Recording rules eai:http_requests:error_ratio_{5m,1h,6h}, eai:http_request_duration_seconds:p95_5m, eai:gateway_cost:rate1h, eai:gateway_tokens:rate5m.
• The platform ships no dashboards (ADR-0169): the portal screens and the Navin Manager read the API and ClickHouse; a Grafana of the client reading this Prometheus only uses the eai_* metrics above, the recording rules and the external exporters vllm:* (vLLM), DCGM_FI_* (DCGM), litellm_* (LiteLLM), kube_*, container_*, node_*, up. Per tenant panels were removed: the control plane has no tenant label (cardinality); per tenant figures stay in the FinOps API and ClickHouse.

OPA

For every active engine: opa policy of the tenant, the control plane (eai/policy.py) sends:

POST {EAI_OPA_URL}/v1/data/eai/<package>/decision
{"input": {"subject": {...}, "action": "...", "resource": {...}, "context": {...}}}

input is exactly the Policy Contract request (contracts/v1/policy-contract.schema.json); resource.tenant_id is always the caller tenant. The package comes from the package eai.<package> header of the policy rego field. Every package of deploy/policies/opa returns:

{"allow": bool, "reasons": ["code", ...], "obligations": [{"type": "dlp" | "approval" | "log_prompts", "params": {...}}]}

Package	Action	Attributes read
eai.model_invocation	model.*	resource.attributes.provider, regions, data_classification_max, status, provider_retention_days, local; context.data_classification, region, cost_estimate.amount; subject.clearance
eai.tool_invocation	tool.*	resource.attributes.risk_level, side_effects, required_scope, network_egress; context.attributes.allowed_tools, sandbox_runtime_class; subject.scopes (globs)
eai.gateway_access	mcp.call, knowledge.query, memory.write, memory.search, realtime.session.create, connector.sync, connector.test, channel.inbound, multimodal.submit, agent.a2a.send	context.data_classification; data.eai.config.kill_switch.gateways; gateway_access.human_only_actions (actions refused to agents); called by policy.gate before any outbound call
eai.data_access	knowledge.*, data.*	resource.tenant_id, resource.attributes.data_classification, acl.users, acl.groups; subject.groups, subject.clearance
eai.agent_autonomy	agent.*	resource.attributes.data_classification_max, regions, risk_level; context.attributes.autonomy_level, depth, steps, action_name, action_kind, amount

Configuration lives in deploy/policies/opa/data.json (data.eai.config): kill switch (global, tenants, providers, models, tools, agents), classification ranks (public to secret, restricted included), per package thresholds and per tenant overrides in tenants.<uuid>.<package>. The approval obligation is the one domains/tools/service.py reads.

Enable a package for a tenant: create an engine: opa policy whose rego declares the package, for example package eai.model_invocation then import rego.v1. The control plane uploads that module (PUT /v1/policies/<tenant>-<name>), which extends the package already loaded. When OPA is unreachable the decision is a deny (policy engine unavailable: <name>).

OpenFGA

deploy/policies/openfga/model.fga declares, for every resource type (tenant, project, agent, model, tool, knowledge_base, collection, document, dataset, prompt, connector, secret), the relations written and checked by /v1/authz and the builtin fallback: owner, editor, viewer, can_use, member, with owner implying editor implying viewer and owner implying can_use (openfga.IMPLIED). Richer relations (admin, can_invoke, can_read, from tenant / from project inheritance) remain available.

Isolation: OpenFGA has a single store, so the control plane prefixes objects and non user subjects with the tenant UUID (agent:<tenant>/invoice, group:<tenant>/finance#member). Users stay global (user:<OIDC sub>), user:* is a wildcard. Writes use on_duplicate: ignore and deletes on_missing: ignore, like the builtin table.

Bootstrap:

fga store create --name enterprise-ai --model deploy/policies/openfga/model.fga --api-url http://openfga:8080 --api-token "$OPENFGA_PRESHARED_KEY"
bao kv patch kv/eai/prod/control-plane EAI_OPENFGA_STORE_ID=<store> EAI_OPENFGA_MODEL_ID=<model>

EAI_OPENFGA_API_TOKEN (optional) is sent as Authorization: Bearer (OpenFGA preshared key). Without EAI_OPENFGA_STORE_ID the control plane uses the eai.authz_tuples table. Model tests: python3 tools/policies/fga_build.py --check.

Keycloak

deploy/keycloak/realm-enterprise-ai.json produces the tokens eai/auth.py reads:

• Issuer https://<keycloak>/realms/enterprise-ai (EAI_OIDC_ISSUER), audience control-plane (EAI_OIDC_AUDIENCE) added by the audience-control-plane mapper.
• basic scope: sub claim (required). eai-tenant scope: tenant (tenant slug, user attribute), clearance, roles and realm_access.roles. profile scope: preferred_username, shown as the principal display name.
• Realm roles: platform_admin, tenant_admin, security_officer, finops, developer, auditor, agent, viewer; default composite role default-roles-enterprise-ai grants viewer.
• Declarative user profile: only administrators may edit tenant and clearance; clearance is limited to the backend classifications.
• Clients: portal (public, PKCE S256, redirects http://localhost:3000/auth/callback and https://<portal-host>/auth/callback, replace with the production host), control-plane (bearer only, carries the audience), cli (public, device flow).
• The platform-admin account has no tenant attribute: a token with an unknown tenant is refused (unknown_tenant).

Environment variables

Every EAI_* variable set by the chart (ConfigMap, Secret, env) or by Compose (environment, .env.example) is a real setting (config.py or an integration adapter). Fixes:

Before	After
EAI_OPENBAO_ADDR	EAI_OPENBAO_URL
EAI_TEMPORAL_ADDRESS (gRPC 7233)	EAI_TEMPORAL_URL (HTTP API 7243)
EAI_AUDIT_BUCKET	EAI_S3_BUCKET
EAI_ENV, EAI_LOG_LEVEL, EAI_REDIS_URL, EAI_REGION, EAI_SPIFFE_TRUST_DOMAIN, EAI_POD_NAME	removed (never read)
EAI_WORKER_STREAM, EAI_WORKER_CONSUMER (GPU workers)	--stream, --consumer arguments
EAI_REGISTRY, EAI_VERSION (Compose)	IMAGE_REGISTRY, IMAGE_TAG

Added: EAI_OTEL_ENABLED, EAI_CORS_ORIGINS, EAI_OPENBAO_KV_MOUNT=kv, EAI_MASTER_KEY, EAI_OPENBAO_TOKEN, EAI_OPENFGA_STORE_ID, EAI_OPENFGA_MODEL_ID, EAI_OPENFGA_API_TOKEN, EAI_QDRANT_API_KEY, EAI_CLICKHOUSE_USER, EAI_CLICKHOUSE_PASSWORD. EAI_CORS_ORIGINS accepts a comma separated list (https://<a-host>,https://<b-host>) or a JSON list ('["https://<portal-host>"]'), like every EAI_* list setting.

CORS exposes X-Request-Id, traceparent, X-EAI-Decision-Id, X-EAI-Cost-Estimated, X-EAI-Cost-Measured and Retry-After.

OpenBao policy deploy/openbao/policies/control-plane.hcl: paths kv/data/tenants/* and transit keys tenant-<uuid>, the ones integrations/openbao.py uses.

Checks

/tmp/toolchains/opa test deploy/policies/opa -v
python3 tools/policies/fga_build.py --check
helm lint deploy/helm/enterprise-ai -f deploy/helm/enterprise-ai/values-prod.yaml
helm template eai deploy/helm/enterprise-ai -f deploy/helm/enterprise-ai/values-onprem-gpu.yaml
terraform -chdir=deploy/terraform/envs/eks-example init -backend=false && terraform -chdir=deploy/terraform/envs/eks-example validate
services/control-plane/.venv/bin/python -m pytest services/control-plane/tests/test_platform_*.py -q

The OPA and OpenFGA tests start the real binaries (/tmp/toolchains/opa, /tmp/toolchains/openfga, /tmp/toolchains/fga CLI, or EAI_TEST_OPA_BIN, EAI_TEST_OPENFGA_BIN, EAI_TEST_FGA_CLI) and are skipped when a binary is missing.