Skip to main content

ADR-0168: Two managed local inference engines, vLLM and Ollama, behind LiteLLM

Context​

Navin Enterprise must host models on the infrastructure of the customer, on CPU hosts as well as GPU hosts, in a sandbox as well as in a cluster. Three engines were compared: vLLM (Apache-2.0), Ollama (MIT) and LM Studio (llmster daemon). Until now vLLM and Ollama were only existing endpoints declared as LiteLLM providers; the Manager did not install them.

Decision​

The Manager installs, configures, monitors and administers two engines only, both described in its catalog: navin-manager/catalog/services/vllm.yaml and navin-manager/catalog/services/ollama.yaml.

  • vLLM is the main production engine: one model per installation, mode single (sandbox), replicated (at least two independent instances on distinct hosts, high availability) and distributed (one large model split across GPU hosts with Ray, pipeline parallelism). The accelerator parameter picks an NVIDIA GPU host (CUDA image, GPUs given through CDI nvidia.com/gpu, tensor parallelism) or a CPU host (x86 CPU image).
  • Ollama is the complementary engine of light and quantized models: modes single and replicated, CPU or GPU host, commands to download, import offline GGUF files, list and delete models, run on every instance.
  • Weights are hosted on the host (/var/lib/navin/vllm/models, /var/lib/navin/ollama/models); vLLM starts offline (HF_HUB_OFFLINE=1) by default.
  • Applications and agents never call an engine directly: the optional vllm and ollama bindings of the LiteLLM service generate one hosted_vllm or ollama_chat entry per instance, under the same model name, for balancing, retries, fallbacks and quotas. The vLLM API key stays in the environment of the proxy.
  • On Kubernetes the deploy/helm/enterprise-ai chart serves vLLM (one Deployment per model, GPU or CPU, replicas on distinct nodes, PodDisruptionBudget) and Ollama (StatefulSet, one model store per replica), reachable only from LiteLLM and the Control Plane.
  • LM Studio is excluded: its licence is not an open source licence and restricts redistribution and serving third parties.

Consequences​

Splitting a model over several GPUs is not high availability: only the replicated mode (or several Kubernetes replicas) provides it. LiteLLM does not isolate GPU memory between tenants: a sensitive tenant gets its own instances and dedicated GPUs (gpu_devices parameter).

Evidence​

navin-manager/server/tests/test_catalog_inference.py, navin-manager/catalog/templates/vllm/start.sh.j2, navin-manager/catalog/templates/litellm/config.yaml.j2, deploy/helm/enterprise-ai/templates/ollama/statefulset.yaml.