Scalability
At a glance
The challenge
An agent run lasts seconds or hours, calls dozens of tools and uses GPUs in bursts. An architecture designed for short requests breaks at the first scale-up.
Scaling out
Requests and events
API, webhooks, channels, scheduler
NATS JetStream
Durable queues and triggers
Workers and agents
Scaled by KEDA and HPA
Checkpoints
PostgreSQL checkpoints, recovery after lease
Inference
vLLM on GPU pools, remote providers
Holding the load
Agents, real-time voice, batch and media processing, periodic tasks and inference scale separately.
HPA and KEDA size services by load and by the depth of NATS JetStream queues.
Agent runs record checkpoints and resume after an incident, without starting over.
Every run is bounded in time, memory, calls and tokens; a heartbeat detects stuck runs.
Quotas and budgets isolate tenants: heavy usage does not affect the others.
NVIDIA GPU Operator, DCGM and GPU accounting for models served by vLLM, with dedicated pools per tenant when needed.
LiteLLM and the model router fail over to a fallback provider or model during an outage.
High availability is declared service by service; Velero handles backups and disaster recovery.
The proof
Usage, cost and performance metrics are available per tenant, project and model, to size the infrastructure before it saturates.
Go further
Autoscaling with HPA and KEDA, event triggers on NATS JetStream, durable checkpoints, quotas and GPU accounting: how Navin Enterprise AI runs many agents without losing work or control, with high availability declared per service.
We estimate with you the capacity your use cases need, from pilot to production.