B01 - Unified LLM Proxy

Group	Technology	Status	Service	Roadmap phase	Contracts
B - LLM management and Model Gateway	LiteLLM Proxy	open source integrated	llm	2	model-manifest, usage-event

Back to index - Français

Role and responsibilities

Unified OpenAI-compatible gateway to local and remote models behind the EAI Model Gateway.

LiteLLM vs vLLM. vLLM (brick B02) is a local inference engine: it loads open weight models on our GPUs and serves them. LiteLLM (brick B01) is the unified gateway: it translates one OpenAI-compatible call to any upstream, local (vLLM, Ollama, LM Studio) or remote (OpenAI, Bedrock, Vertex...). vLLM is one upstream of LiteLLM, never a replacement for it, and LiteLLM never runs a model itself.

Position in the architecture and interactions

Sync path: Agent Runtime (C01), Capability Gateway, Knowledge Gateway and SDK clients call POST /v1/gateway/* on the Control Plane Model Gateway. Sequence: authenticate -> Policy Enforcement (A07) -> DLP (E02) -> Capability Resolver (B06) and Effort Policy (B07) -> Context plan (B08) -> estimate and quota reservation (H03, H01) -> Router (B05) -> LiteLLM (B01) -> upstream (vLLM B02 or remote provider) -> Token Accounting (B09) -> Usage Event and audit. Streaming uses SSE. Async paths: batch jobs and video generation go through specialized adapters and Temporal workflows; Usage Events are also emitted on NATS for analytics (ClickHouse).

Open source component vs developed part

Open source: LiteLLM Proxy (routing, provider translation, fallbacks, retries). Developed: the EAI Model Gateway in front of it (policy, DLP, quotas, cost, audit), the generator of the LiteLLM config from the registries (GET /v1/litellm/config, POST /v1/litellm/sync), credential injection from OpenBao, and the specialized adapters.

LiteLLM stays internal: it only receives the providers and models pushed by the platform. Team keys, access and quotas are held by the platform (memberships, API keys, quotas and the directory group mappings of A03), never as LiteLLM virtual keys or teams.

Supported providers

Provider id	Provider	Path through LiteLLM	Admin configuration fields
openai	OpenAI	native	api_key, organization, base_url
anthropic	Anthropic	native	api_key, base_url
google_ai_studio	Google AI Studio / Gemini	native (gemini/)	api_key
vertex_ai	Vertex AI	native (vertex_ai/)	project, location, service account JSON (or workload identity)
azure_openai	Azure OpenAI	native (azure/)	endpoint, deployment name, api_version, api_key or Entra ID
azure_ai_foundry	Azure AI Foundry	adapter selected by endpoint type (OpenAI style or model inference style)	endpoint, api_key or Entra ID, api_version
aws_bedrock	AWS Bedrock	native (bedrock/)	aws_region_name, aws_access_key_id + aws_secret_access_key, or IAM role (aws_role_arn / IRSA), or Bedrock API bearer token (aws_bearer_token)
mistral	Mistral	native	api_key
cohere	Cohere	native (chat, embed, rerank)	api_key
deepseek	DeepSeek	native	api_key
groq	Groq	native	api_key
together_ai	Together AI	native	api_key
fireworks_ai	Fireworks AI	native	api_key
openrouter	OpenRouter	native	api_key
ollama	Ollama	native	base_url
vllm	vLLM (local, brick B02)	OpenAI compatible (hosted_vllm/)	base_url, optional api_key
qwen	Qwen (DashScope)	OpenAI compatible	api_key, base_url
zai	Z.AI (GLM)	OpenAI compatible	api_key, base_url
minimax	MiniMax	OpenAI compatible	api_key, base_url
lm_studio	LM Studio	OpenAI compatible (lm_studio/)	base_url
openai_codex	OpenAI Codex	OpenAI compatible, OAuth account	OAuth token (brokered)
github_copilot	GitHub Copilot	native (github_copilot/)	OAuth device token (brokered)
huggingface	Hugging Face	native (huggingface/)	api_key, endpoint (Inference Endpoints)
custom	Custom OpenAI compatible	openai/ with custom base_url	base_url, api_key, headers
custom_anthropic	Custom Anthropic compatible	anthropic/ with custom base_url	base_url, api_key

Functional compatibility limits. LiteLLM normalizes chat, embeddings and rerank well, but it does not guarantee universal functional compatibility. Video generation, realtime duplex voice, image generation and editing, and batch APIs differ deeply between providers. They are served by specialized adapters listed by GET /v1/adapters, registered under the same Model Registry, the same policies, the same quotas and the same cost system (Usage Events, price cards). A capability not supported by the selected adapter is rejected explicitly.

Data model

Tables (schema eai, every tenant table carries tenant_id uuid NOT NULL with an RLS policy on current_setting('app.tenant_id', true)::uuid): eai.llm_models, eai.llm_providers, eai.usage_events, eai.audit_events.

API endpoints

• POST /v1/gateway/chat/completions
• POST /v1/gateway/embeddings
• POST /v1/gateway/rerank
• GET /v1/litellm/config
• POST /v1/litellm/sync
• GET /v1/adapters

POST /v1/gateway/chat/completions HTTP/1.1
Host: <control-plane-host>
X-API-Key: eai_xxx_secret
Accept-Language: en
Content-Type: application/json

{"model": "claude-sonnet-eu", "messages": [{"role": "user", "content": "Bonjour"}], "eai": {"effort": "medium", "data_classification": "internal", "purpose": "support", "region_allow": ["eu"]}}

{"id": "chatcmpl-...", "choices": [{"message": {"role": "assistant", "content": "Bonjour !"}}], "usage": {"prompt_tokens": 12, "completion_tokens": 4}}

Errors use the common envelope {"error": {"code", "message", "message_i18n", "details", "request_id"}}; lists are paginated with limit and cursor.

SDK usage

Python (enterprise_ai):

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_xxx_secret")
items = client.models.list()

TypeScript (@enterprise-ai/sdk):

import { EnterpriseAI } from "@enterprise-ai/sdk";

const eai = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY });
const items = await eai.models.list();

Java (enterprise-ai-java):

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .build();
var items = client.models().list();

Go (enterprise-ai-go):

client, _ := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")))
items, err := client.Models.List(ctx)

Chat call with the eai extension (Python): client.gateway.chat(model="claude-sonnet-eu", messages=[...], eai={"effort": "medium", "data_classification": "internal"}).

Configuration

• Environment: EAI_DATABASE_URL, EAI_OTEL_EXPORTER_OTLP_ENDPOINT.

• Helm values: controlPlane.enabled, controlPlane.replicas, controlPlane.resources, controlPlane.config, litellm.enabled, litellm.config.

• EAI_LITELLM_URL, EAI_LITELLM_MASTER_KEY (from OpenBao).

• Helm: litellm.enabled, litellm.replicas, litellm.config.fromControlPlane: true, gateway.adapters.video.enabled.

Security controls

• Authentication by OIDC JWT, tenant API key with scopes, or SPIFFE mTLS; tenant resolved from the credential, never from the body.
• Authorization through OpenFGA and OPA decisions; every mutation and decision writes a hash-chained eai.audit_events row.
• Secrets never returned by the API; TLS everywhere; Cilium network policies restrict who can reach the service.

Control	Enforcement
Provider / model / region allowlists	Per tenant and project in eai.policies; the Router only selects allowed triples (provider, model, region)
Data classification	Request eai.data_classification must be <= model data_classification_max; very sensitive (restricted, secret) data stays on local models (vLLM)
DLP	DLP Gateway (E02) inspects and masks or tokenizes prompts before any remote call
Secret isolation	Credentials write-only, stored in OpenBao, injected into LiteLLM at sync, never returned by the API or logged
TLS	TLS 1.2+ to every provider, mTLS SPIFFE inside the cluster
Provider retention	Registry records the provider retention / zero data retention terms; models without acceptable terms are restricted
Fallback policy	Fallback chains are filtered with the same allowlists: never a forbidden region, model or classification
Cost guardrails	Estimate and quota reservation before execution (POST /v1/gateway/estimate, POST /v1/quotas/reserve), streaming cut-off during execution when the budget is reached
Logging without exposure	Prompts and completions logged only according to policy, redacted, never with credentials
Kill switch	POST /v1/providers/{id}/disable and POST /v1/models/{model_id}/disable take effect at the next call

Remote models are allowed for some uses (public or internal data, approved purposes) while very sensitive data stays on local models.

Observability

• Metrics: eai_unified_llm_proxy_requests_total, eai_unified_llm_proxy_errors_total, eai_unified_llm_proxy_latency_seconds (Prometheus).
• Traces: OpenTelemetry spans eai.gateway.* with W3C traceparent propagation, AI spans forwarded to Opik.
• Audit actions: model.invoke, policy.evaluate.

FinOps aspects

Each call is estimated before execution (X-EAI-Cost-Estimated), measured after (X-EAI-Cost-Measured) and later billed by reconciliation (B11). Agents must never call LiteLLM directly: Cilium policies only allow the Control Plane to reach it, so no call escapes accounting.

Failure modes and resilience

• Upstream 429/5xx: retries with backoff, then Provider Failover (B14) within allowed chains.
• LiteLLM unavailable: the gateway returns 503 upstream_unavailable; several replicas and a PodDisruptionBudget.
• Config drift: POST /v1/litellm/sync is idempotent and the generated config is versioned.

Standalone adoption (client of a client)

A client of a client can adopt this brick alone with the Python SDK sub-client client.gateway (TS eai.gateway, Java client.gateway(), Go client.Gateway). Minimal deployment: Control Plane API + PostgreSQL + Keycloak (or API keys), the Control Plane loads every domain of services/control-plane/src/eai/domains; the optional chart components are switched off in deploy/helm/enterprise-ai/values.yaml (vllm.enabled, qdrant.enabled, nats.enabled, workers.enabled).

Acceptance tests

1. A call to a blocked model returns 403 policy_denied and an audit event.
2. A streaming call emits SSE chunks and a final Usage Event with measured cost.
3. A restricted request is routed only to a vLLM model.
4. GET /v1/litellm/config never contains a secret.
5. A direct pod to LiteLLM connection from an agent namespace is refused by Cilium.

Real LiteLLM proxy (LLM_-001)

bash tools/litellm/setup.sh creates the separate venv /tmp/toolchains/litellm-venv with the packages of the Navin LiteLLM image: the hashed lock deploy/litellm/requirements.lock (litellm 1.106.5 without its proxy extra, which requires the proprietary package litellm-enterprise), installed with pip install --require-hashes --no-deps (pip --require-virtualenv, resolved versions in resolved-versions.txt); a venv that already holds litellm-enterprise is recreated. tests/test_llm_litellm_proxy_real.py starts the real proxy on a free port with the configuration generated by litellm_config.py, in front of a local OpenAI compatible upstream, then calls /v1/gateway/chat/completions with EAI_LITELLM_URL: answer, usage, usage event and model.invoke audit are checked. Without the venv the test fails with a clear message.