Skip to main content

B02 - Local Inference

GroupTechnologyStatusServiceRoadmap phaseContracts
B - LLM management and Model GatewayvLLMopen source integrateddeploy/vllm4model-manifest

Back to index - Français

Role and responsibilities​

Local GPU inference engine serving open weight models; one upstream of LiteLLM.

LiteLLM vs vLLM. vLLM (brick B02) is a local inference engine: it loads open weight models on our GPUs and serves them. LiteLLM (brick B01) is the unified gateway: it translates one OpenAI-compatible call to any upstream, local (vLLM, Ollama, LM Studio) or remote (OpenAI, Bedrock, Vertex...). vLLM is one upstream of LiteLLM, never a replacement for it, and LiteLLM never runs a model itself.

Position in the architecture and interactions​

vLLM servers are declared like any other LiteLLM provider: POST /v1/providers with provider_type: vllm, the endpoint (api_base) and an optional key, exactly as Ollama or LM Studio. POST /v1/providers/{id}/models/discover reads the served models from GET {api_base}/models, POST /v1/models/import registers the selected ones and POST /v1/litellm/sync publishes them to LiteLLM. The servers themselves run on GPU nodes (B13, Helm chart vllm.models). Only LiteLLM calls vLLM (OpenAI-compatible HTTP, sync and streaming). KEDA scales replicas on queue depth.

Open source component vs developed part​

Open source: vLLM (PagedAttention, continuous batching, tensor parallelism). Developed: provider registration, model discovery and import, model manifests, GPU placement, sovereignty rules (restricted data served only here).

Data model​

Tables (schema eai, every tenant table carries tenant_id uuid NOT NULL with an RLS policy on current_setting('app.tenant_id', true)::uuid): eai.llm_providers, eai.llm_models, eai.gpu_samples, eai.audit_events.

API endpoints​

  • GET /v1/providers/catalog
  • POST /v1/providers
  • POST /v1/providers/{id}/models/discover
  • POST /v1/models/import
POST /v1/providers/7fcf4489-04d6-4992-9187-0ccca12986e6/models/discover HTTP/1.1
Host: <control-plane-host>
X-API-Key: eai_xxx_secret
Accept-Language: en
{"provider_id": "7fcf4489-04d6-4992-9187-0ccca12986e6", "provider_type": "vllm", "kind": "openai", "mode": "api", "count": 1, "registered_count": 0, "items": [{"provider_model_id": "meta-llama/Llama-3.1-8B-Instruct", "model_id": "meta-llama-llama-3.1-8b-instruct", "display_name": "meta-llama/Llama-3.1-8B-Instruct", "category": "text", "context_window": 131072, "registered": false, "registered_model_id": null, "hints_applied": []}]}

Errors use the common envelope {"error": {"code", "message", "message_i18n", "details", "request_id"}}; lists are paginated with limit and cursor.

SDK usage​

Python (enterprise_ai):

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_xxx_secret")
items = client.providers.discover_models("7fcf4489-04d6-4992-9187-0ccca12986e6")

TypeScript (@enterprise-ai/sdk):

import { EnterpriseAI } from "@enterprise-ai/sdk";

const eai = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY });
const items = await eai.providers.discoverModels("7fcf4489-04d6-4992-9187-0ccca12986e6");

Java (enterprise-ai-java):

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .build();
var items = client.providers().discoverModels("7fcf4489-04d6-4992-9187-0ccca12986e6");

Go (enterprise-ai-go):

client, _ := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")))
items, err := client.Providers.DiscoverModels(ctx, "7fcf4489-04d6-4992-9187-0ccca12986e6")

Configuration​

  • Environment: EAI_DATABASE_URL, EAI_OTEL_EXPORTER_OTLP_ENDPOINT.
  • Helm values: controlPlane.enabled, controlPlane.replicas, controlPlane.resources, controlPlane.config.

Security controls​

  • Authentication by OIDC JWT, tenant API key with scopes, or SPIFFE mTLS; tenant resolved from the credential, never from the body.
  • Authorization through OpenFGA and OPA decisions; every mutation and decision writes a hash-chained eai.audit_events row.
  • Secrets never returned by the API; TLS everywhere; Cilium network policies restrict who can reach the service.

Observability​

  • Metrics: eai_local_inference_requests_total, eai_local_inference_errors_total, eai_local_inference_latency_seconds (Prometheus).
  • Traces: OpenTelemetry spans eai.providers.* with W3C traceparent propagation, AI spans forwarded to Opik.
  • Audit actions: provider.create, provider.discover, model.import, litellm.sync, policy.evaluate.

FinOps aspects​

Cost is GPU time, not tokens: DCGM samples (POST /v1/finops/gpu/samples) are allocated per tenant by GET /v1/finops/gpu/allocation and allocation policies; an internal price card converts it into an equivalent cost per token.

Failure modes and resilience​

  • Stateless replicas behind the API Gateway, PostgreSQL HA for state; readiness probe GET /readyz removes unhealthy pods.
  • External system unavailable: fail closed for security decisions, explicit error code, no silent bypass.

Standalone adoption (client of a client)​

A client of a client can adopt this brick alone with the Python SDK sub-client client.providers (TS eai.providers, Java client.providers(), Go client.Providers). Minimal deployment: Control Plane API + PostgreSQL + Keycloak (or API keys), the Control Plane loads every domain of services/control-plane/src/eai/domains; the optional chart components are switched off in deploy/helm/enterprise-ai/values.yaml (vllm.enabled, qdrant.enabled, nats.enabled, workers.enabled).

Acceptance tests​

  1. A tenant cannot read or modify another tenant's resources (RLS test).
  2. Every mutation produces an audit event verifiable by GET /v1/audit/verify.
  3. Errors are returned in English and French according to Accept-Language.