Skip to main content

B05 - Model Router

GroupTechnologyStatusServiceRoadmap phaseContracts
B - LLM management and Model GatewayDevelopedto developllm2-

Back to index - Français

Role and responsibilities​

Routing rules and fallback chains by purpose, classification, region, cost and latency.

Position in the architecture and interactions​

Called synchronously by the gateway for every request; POST /v1/routes/simulate explains the choice without calling a model. Rules match on purpose, classification, region, effort and cost/latency targets, and output an ordered chain.

Data model​

Tables (schema eai, every tenant table carries tenant_id uuid NOT NULL with an RLS policy on current_setting('app.tenant_id', true)::uuid): eai.llm_routes, eai.audit_events.

API endpoints​

  • GET /v1/routes
  • POST /v1/routes
  • PUT /v1/routes/{id}
  • DELETE /v1/routes/{id}
  • POST /v1/routes/simulate
POST /v1/routes HTTP/1.1
Host: <control-plane-host>
X-API-Key: eai_xxx_secret
Accept-Language: en
Content-Type: application/json

{"name": "support-eu", "match": {"purpose": "support"}, "strategy": "priority", "candidates": ["fake-gpt"], "priority": 100, "enabled": true}
{"id": "417d22b5-f725-4a5c-a6fe-128310c3d083", "tenant_id": "18de5695-21f9-4053-a4ea-beb4bf2a5932", "name": "support-eu", "match": {"purpose": "support"}, "strategy": "priority", "candidates": ["fake-gpt"], "fallbacks": [], "constraints": {}, "priority": 100, "enabled": true, "created_at": "2026-10-30T02:36:57.212559+01:00", "updated_at": "2026-10-30T02:36:57.212559+01:00", "target_model": "fake-gpt"}

Errors use the common envelope {"error": {"code", "message", "message_i18n", "details", "request_id"}}; lists are paginated with limit and cursor.

SDK usage​

Python (enterprise_ai):

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_xxx_secret")
items = client.routes.list()

TypeScript (@enterprise-ai/sdk):

import { EnterpriseAI } from "@enterprise-ai/sdk";

const eai = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY });
const items = await eai.routes.list();

Java (enterprise-ai-java):

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .build();
var items = client.routes().list();

Go (enterprise-ai-go):

client, _ := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")))
items, err := client.Routes.List(ctx)

Configuration​

  • Environment: EAI_DATABASE_URL, EAI_OTEL_EXPORTER_OTLP_ENDPOINT.
  • Helm values: controlPlane.enabled, controlPlane.replicas, controlPlane.resources, controlPlane.config, litellm.enabled, litellm.config.

Security controls​

  • Authentication by OIDC JWT, tenant API key with scopes, or SPIFFE mTLS; tenant resolved from the credential, never from the body.
  • Authorization through OpenFGA and OPA decisions; every mutation and decision writes a hash-chained eai.audit_events row.
  • Secrets never returned by the API; TLS everywhere; Cilium network policies restrict who can reach the service.
ControlEnforcement
Provider / model / region allowlistsPer tenant and project in eai.policies; the Router only selects allowed triples (provider, model, region)
Data classificationRequest eai.data_classification must be <= model data_classification_max; very sensitive (restricted, secret) data stays on local models (vLLM)
DLPDLP Gateway (E02) inspects and masks or tokenizes prompts before any remote call
Secret isolationCredentials write-only, stored in OpenBao, injected into LiteLLM at sync, never returned by the API or logged
TLSTLS 1.2+ to every provider, mTLS SPIFFE inside the cluster
Provider retentionRegistry records the provider retention / zero data retention terms; models without acceptable terms are restricted
Fallback policyFallback chains are filtered with the same allowlists: never a forbidden region, model or classification
Cost guardrailsEstimate and quota reservation before execution (POST /v1/gateway/estimate, POST /v1/quotas/reserve), streaming cut-off during execution when the budget is reached
Logging without exposurePrompts and completions logged only according to policy, redacted, never with credentials
Kill switchPOST /v1/providers/{id}/disable and POST /v1/models/{model_id}/disable take effect at the next call

Remote models are allowed for some uses (public or internal data, approved purposes) while very sensitive data stays on local models.

Observability​

  • Metrics: eai_model_router_requests_total, eai_model_router_errors_total, eai_model_router_latency_seconds (Prometheus).
  • Traces: OpenTelemetry spans eai.routes.* with W3C traceparent propagation, AI spans forwarded to Opik.
  • Audit actions: route.create, route.update, route.delete, policy.evaluate.

FinOps aspects​

Operating cost is allocated per tenant and project through labels on Usage Events and Kubernetes resource requests; no direct per-call cost.

Failure modes and resilience​

  • Stateless replicas behind the API Gateway, PostgreSQL HA for state; readiness probe GET /readyz removes unhealthy pods.
  • External system unavailable: fail closed for security decisions, explicit error code, no silent bypass.

Standalone adoption (client of a client)​

A client of a client can adopt this brick alone with the Python SDK sub-client client.routes (TS eai.routes, Java client.routes(), Go client.Routes). Minimal deployment: Control Plane API + PostgreSQL + Keycloak (or API keys), the Control Plane loads every domain of services/control-plane/src/eai/domains; the optional chart components are switched off in deploy/helm/enterprise-ai/values.yaml (vllm.enabled, qdrant.enabled, nats.enabled, workers.enabled).

Acceptance tests​

  1. A tenant cannot read or modify another tenant's resources (RLS test).
  2. Every mutation produces an audit event verifiable by GET /v1/audit/verify.
  3. Errors are returned in English and French according to Accept-Language.

Canary and isolated modes (LLM_-016, LLM_-015)​

The router serves the canary version to its traffic share (eai.model_version in the answer). In local_only or air_gapped mode (PUT /v1/llm/isolation, per tenant or per project, V139) only local providers (vllm, ollama, lm_studio) and the declared internal hosts are routable: an external route is refused in preflight (403 isolation_external_route) before any outbound call, external fallbacks are dropped, and the outbound host is checked a second time (403 isolation_egress_denied). Fallbacks stay forbidden towards an unauthorized classification or region (LLM_-007). Tests: tests/test_llm_isolation.py, tests/test_llm_router.py.