Skip to main content

B08 - Context Window Manager

GroupTechnologyStatusServiceRoadmap phaseContracts
B - LLM management and Model GatewayDevelopedto developllm3context-contract

Back to index - Français

Role and responsibilities​

Plans context budget, compression, selection, summarization and overflow prevention.

Open source component vs developed part​

Planning formula: B_context = W_model - B_output - B_reasoning - B_safety, where W_model is the documented window, B_output the reserved output, B_reasoning the reserved reasoning budget and B_safety a margin for tokenizer differences and system prompts. This is a planning model: real accounting depends on the provider (tokenizer, whether reasoning tokens count against the window, cached tokens), and the measured values come back from Token Accounting (B09).

Strategies applied in order when segments exceed B_context: selection (top-k, deduplication, ACL), compression (removing boilerplate, extractive compression), summarization of old history (by a cheap model through the gateway), prompt cache reuse for stable prefixes, and overflow prevention: the call is rejected with context_overflow rather than truncated silently by the provider.

Data model​

Plans follow contracts/v1/context-contract.schema.json (budget, segments, classification_max, compression_actions, cache).

Tables (schema eai, every tenant table carries tenant_id uuid NOT NULL with an RLS policy on current_setting('app.tenant_id', true)::uuid): eai.audit_events.

API endpoints​

  • POST /v1/context/plan
POST /v1/context/plan HTTP/1.1
Host: <control-plane-host>
X-API-Key: eai_xxx_secret
Accept-Language: en
Content-Type: application/json

{"model_id": "claude-sonnet-eu", "window": 200000, "output": 8000, "reasoning": 16000, "safety": 4000, "segments": [{"kind": "rag", "tokens": 190000}]}
{"b_context": 172000, "overflow": true, "actions": [{"type": "select", "segment": "rag", "target_tokens": 150000}, {"type": "summarize", "segment": "history"}]}

Errors use the common envelope {"error": {"code", "message", "message_i18n", "details", "request_id"}}; lists are paginated with limit and cursor.

SDK usage​

Python (enterprise_ai):

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_xxx_secret")
plan = client.context.plan(window=128000, output=4000, reasoning=2000)

TypeScript (@enterprise-ai/sdk):

import { EnterpriseAI } from "@enterprise-ai/sdk";

const eai = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY });
const plan = await eai.context.plan({ window: 128000, output: 4000, reasoning: 2000, segments: [] });

Java (enterprise-ai-java):

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .build();
var plan = client.context().plan(Map.of("window", 128000, "output", 4000, "reasoning", 2000));

Go (enterprise-ai-go):

client, _ := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")))
plan, err := client.Context.Plan(ctx, map[string]any{"window": 128000, "output": 4000, "reasoning": 2000})

Configuration​

  • Environment: EAI_DATABASE_URL, EAI_OTEL_EXPORTER_OTLP_ENDPOINT.
  • Helm values: controlPlane.enabled, controlPlane.replicas, controlPlane.resources, controlPlane.config, litellm.enabled, litellm.config.

Security controls​

  • Authentication by OIDC JWT, tenant API key with scopes, or SPIFFE mTLS; tenant resolved from the credential, never from the body.
  • Authorization through OpenFGA and OPA decisions; every mutation and decision writes a hash-chained eai.audit_events row.
  • Secrets never returned by the API; TLS everywhere; Cilium network policies restrict who can reach the service.

Observability​

  • Metrics: eai_context_window_manager_requests_total, eai_context_window_manager_errors_total, eai_context_window_manager_latency_seconds (Prometheus).
  • Traces: OpenTelemetry spans eai.context.* with W3C traceparent propagation, AI spans forwarded to Opik.
  • Audit actions: model.invoke, policy.evaluate.

FinOps aspects​

Operating cost is allocated per tenant and project through labels on Usage Events and Kubernetes resource requests; no direct per-call cost.

Failure modes and resilience​

  • Stateless replicas behind the API Gateway, PostgreSQL HA for state; readiness probe GET /readyz removes unhealthy pods.
  • External system unavailable: fail closed for security decisions, explicit error code, no silent bypass.

Standalone adoption (client of a client)​

A client of a client can adopt this brick alone with the Python SDK sub-client client.context (TS eai.context, Java client.context(), Go client.Context). Minimal deployment: Control Plane API + PostgreSQL + Keycloak (or API keys), the Control Plane loads every domain of services/control-plane/src/eai/domains; the optional chart components are switched off in deploy/helm/enterprise-ai/values.yaml (vllm.enabled, qdrant.enabled, nats.enabled, workers.enabled).

Acceptance tests​

  1. A tenant cannot read or modify another tenant's resources (RLS test).
  2. Every mutation produces an audit event verifiable by GET /v1/audit/verify.
  3. Errors are returned in English and French according to Accept-Language.