Skip to main content

H-05 Autoscaling

Brick H of the Enterprise AI Platform. Technology: Kubernetes HPA (CPU, memory, custom metrics) + KEDA (NATS JetStream lag, Prometheus queries). French version: ../../fr/bricks/H-05-autoscaling.md.

Role and responsibilities​

  • Scale stateless services on CPU and request rate, GPU workers on queue lag and pending requests, scale to zero for idle GPU pools.

Position and interactions​

  • Execution path: control loop.
  • vllm.keda and workers.items[].keda values define ScaledObjects.

Open source and developed parts​

  • Open source reused: HPA, KEDA, Cluster Autoscaler or Karpenter.
  • Developed in the platform: Scaling profiles per workload, warm pool policy for low latency models.

Data model​

Control Plane tables (schema eai): eai.gpu_samples.

Every tenant table has tenant_id uuid NOT NULL and a row level security policy on current_setting('app.tenant_id', true)::uuid. ClickHouse usage_events is the analytics replica of eai.usage_events (ReplacingMergeTree keyed by event_id).

Contract: contracts/v1/usage-event.schema.json (schema_version 1.0). Usage is always reported with contracts/v1/usage-event.schema.json, audit with contracts/v1/audit-event.schema.json.

API endpoints​

MethodPath
GET/v1/providers/catalog
GET/v1/telemetry/overview

GET /v1/providers/catalog​

Response:

{
  "items": [
    {
      "id": "vllm",
      "label": "vLLM (local inference engine)",
      "local": true,
      "base_url": "http://vllm-llama-70b:8000/v1",
      "provider_id": "7fcf4489-04d6-4992-9187-0ccca12986e6",
      "enabled": true,
      "discovery": "api"
    }
  ]
}

GET /v1/telemetry/overview​

Response:

{
  "window": "24h",
  "requests": 18234,
  "error_rate": 0.012,
  "p95_latency_ms": 2140,
  "tokens": {
    "input": 9120044,
    "output": 1203400
  },
  "top_models": [
    {
      "model_id": "gpt-4o-mini",
      "share": 0.61
    }
  ]
}

SDK examples​

Python​

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_live_xxx", tenant="acme", locale="en")
client.providers.catalog()
client.telemetry.overview(days=1)

TypeScript​

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY, tenant: "acme", locale: "en" });
await client.providers.catalog();
await client.telemetry.overview({ days: 1 });

Java​

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .tenant("acme")
    .locale(Locale.ENGLISH)
    .build();
client.providers().catalog();
client.telemetry().overview(Map.of("days", 1));

Go​

client, err := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")), eai.WithTenant("acme"), eai.WithLocale("en"))
if err != nil { log.Fatal(err) }
ctx := context.Background()
_, err = client.Providers.Catalog(ctx)
_, err = client.Telemetry.Overview(ctx, eai.WithQuery(map[string]string{"days": "1"}))

Configuration​

No EAI_* variable: configured through Helm values and the tool's own configuration.

Helm values keys (deploy/helm/enterprise-ai/values.yaml): controlPlane.autoscaling, portal.autoscaling, litellm.autoscaling, vllm.keda, workers.items[].keda.

Security​

  • KEDA reads metrics with a read-only service account.
  • Tenant resolved from the credential, never from the body; PostgreSQL row level security on every tenant table.
  • Every mutation and policy decision writes a hash chained audit event (G-11).
  • Errors use the standard envelope with message_i18n en and fr.

Observability​

  • Replica count versus target, scaling events.
  • W3C traceparent propagated, X-Request-Id returned, spans carry eai.tenant_id, eai.project, eai.agent_id, eai.run_id.

FinOps units​

  • Scale to zero and max replicas are cost controls; GPU idle time is visible in H-04.

Failure modes and resilience​

  • Metric source down -> HPA keeps the current replicas (no scale down storm).

Standalone adoption​

Helm values are reusable in the client cluster.

Acceptance tests​

  1. Load test raises control plane replicas within 2 minutes and returns to minimum after.

Implementation roadmap