LiteLLM gateway (litellm)

OpenAI compatible model gateway in front of the providers (cloud providers, the vLLM and Ollama engines installed by the Manager, registered automatically through the vllm and ollama bindings, and existing endpoints declared as LiteLLM providers): routing, load balancing between replicas, fallbacks, Redis cache and spend tracking. Provider credentials stay in its sensitive files and never reach the agents.

Generated from catalog/services/litellm.yaml: do not edit by hand.

Summary

Version	1.106.5
Use	install and configure, configure and connect
Layer	functional (AI catalog)
Category	Gateways and load balancing
Installation methods	python
External connection	no
Sandbox only	no

Licence

SPDX identifier	MIT
Licence policy	core licence (MIT or Apache-2.0)
Licence exception	-
Licence note	MIT core only. The Manager installs litellm without its proxy extra, which requires the proprietary package litellm-enterprise (LicenseRef-Proprietary), and adds the other requirements of that extra one by one: the proxy and its router run without it. The LiteLLM Enterprise features (enterprise/ directory and litellm-enterprise package: SSO of the admin UI, audit logs, advanced guardrails and budgets, the background mode of the Responses API) are under a commercial licence and are neither installed nor configured. No podman method: the official image ghcr.io/berriai/litellm installs the proxy extra, hence litellm-enterprise, and copies the enterprise/ directory.
Platform component	litellm

Packages and artifacts

A package the project of the service publishes (its image, its binary archive, its Python or OS package) carries the licence of the service; a third-party package carries the licence written on its row, read from the official upstream metadata.

Kind	Package	Version	Licence	Details
Python package	litellm	1.106.5	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	gunicorn	>=23.0.0,<24.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	uvicorn	>=0.33.0,<1.0 (range)	BSD-3-Clause	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	granian	>=2.7.4,<3.0 (range)	BSD-3-Clause	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	uvloop	>=0.22.1,<1.0 (range)	MIT OR Apache-2.0	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	fastapi	>=0.136.3,<1.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	starlette	>=1.0.1,<2.0 (range)	BSD-3-Clause	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	backoff	>=2.2.1,<3.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	pyyaml	>=6.0.3,<7.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	rq	>=2.7.0,<3.0 (range)	BSD-2-Clause	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	orjson	>=3.11.6,<4.0 (range)	MPL-2.0 AND (Apache-2.0 OR MIT)	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	hiredis	>=3.0.0,<4.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	apscheduler	>=3.11.2,<4.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	fastapi-sso	>=0.19.0,<1.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	pyjwt	>=2.13.0,<3.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	python-multipart	>=0.0.27,<1.0 (range)	Apache-2.0	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	cryptography	>=49.0.0,<51.0 (range)	Apache-2.0 OR BSD-3-Clause	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	pynacl	>=1.6.2,<2.0 (range)	Apache-2.0	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	websockets	>=15.0.1,<16.0 (range)	BSD-3-Clause	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	boto3	>=1.43.117,<2.0 (range)	Apache-2.0	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	azure-identity	>=1.25.2,<2.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	azure-storage-blob	>=12.28.0,<13.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	mcp	>=2.2.0,<3 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	httpx2	>=2.5.0,<3 (range)	BSD-3-Clause	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	pydantic	>=2.12.0,<3 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	litellm-proxy-extras	0.4.108	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	restrictedpython	>=8.5,<9.0 (range)	ZPL-2.1	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	rich	>=13.9.4,<14.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	inquirerpy	>=0.3.4,<1.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	tomlkit	>=0.13.3,<1.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	polars	>=1.38.1,<2.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	soundfile	>=0.12.1,<1.0 (range)	BSD-3-Clause	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	pyroscope-io	>=0.8.16,<1.0 (range)	Apache-2.0	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	expression	>=5.6.0,<6.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	prisma	>=0.11 (range)	Apache-2.0	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	opik	>=1.0 (range)	Apache-2.0	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	opentelemetry-exporter-otlp-proto-http	>=1.25 (range)	Apache-2.0	interpreters, in order of preference python3.12, python3.11, python3.10
Python package	prometheus_client	>=0.20,<1.0 (range)	MIT	interpreters, in order of preference python3.12, python3.11, python3.10

Supported systems and architectures

Method	Operating systems	Architectures
python	RHEL family (dnf), Debian family (apt), Arch Linux (pacman)	x86_64, aarch64 (not restricted by the descriptor)

Certified host systems: RHEL 8/9/10, Rocky Linux 8/9/10, AlmaLinux 8/9/10, Oracle Linux 8/9, CentOS Stream 9/10, Ubuntu 22.04/24.04, Debian 12, Arch Linux (rolling). The host preflight accepts these systems (other releases of a known family give a warning, end of life releases are refused) and the x86_64 and aarch64 architectures.

Prerequisites

Host

• at least 2048 MB of memory (blocking, memory).

Network and ports

Component	Port name	Port	Opened in the host firewall
litellm-proxy	api	4000/tcp	yes

Services and connections needed

• database: postgresql.
• cache: redis (optional).
• vllm: vllm (optional).
• ollama: ollama (optional).
• opik: opik (optional).
• otel: otel-collector (optional).

Details and resolution: see Dependencies.

Installation methods and modes

Method	Default	What the method uses
python	yes	a Python virtual environment holding the Python packages of the descriptor

Components

Component	Kind	Installation methods	Ports
LiteLLM proxy (litellm-proxy)	master	python	4000/tcp

Single proxy (single) (default)

One proxy instance.

What the mode deploys:

Component	Placement
litellm-proxy	min 1, max 1

Replicated (replicated)

Several proxies behind HAProxy, sharing the database and the Redis cache.

What the mode deploys:

Component	Placement
litellm-proxy	min 2, recommended 3

Install this service

The install wizard of the Navin Manager plans, validates and runs the installation; you need the services:install permission on the cluster.

1. Open the wizard. In Clusters, choose the row action Install services of the target cluster. The wizard opens on its first step, Service.

2. Choose the service. On the Service step, select LiteLLM gateway. The card shows the licence MIT and the dependencies the installation needs.

3. Choose the hosts and the mode. On the Hosts step, tick the hosts, then choose the Mode: single (default), replicated. Each mode deploys the components listed in Installation methods and modes.

4. Bind the dependencies. On the Bindings step, choose for each dependency (database, cache, vllm, ollama, opik, otel) Install with the platform or Use an existing connection; an optional dependency can stay on No binding.

5. Configure. On the Configure step, review the parameters (see Configuration parameters). Passwords left empty with a generated default are generated and sealed by the Manager.

6. Review and install. On the Install step the Manager validates the plan and shows the installation order; when it says "The plan is valid. Ready to install.", choose Install. A highly available mode is installed only when its HA checks pass.

7. Verify. The operation ends when the health checks of every instance pass (see Health checks); the service then shows as running in Services.

Connect an existing service

Schemes	https, http
Default port	4000
Authentication	required
Health check	/health/liveliness

Authentication method	Fields	Description
bearer	token	Master key or virtual key (sk-...), sent as Authorization: Bearer.

Official documentation: https://docs.litellm.ai/docs/proxy/virtual_keys

High availability

Cluster	replicas (replication)
High availability	external (external)
Failover	load balancer (load-balancer)
Replication	shared store (shared-store)
Quorum	none (none)
Discovery	none (none)
Entry point	load balancer (load-balancer)
Official documentation	https://docs.litellm.ai/docs/proxy/prod

Prerequisites

Prerequisite	Must be highly available	Modes	Description
database	yes	modes of level ha or cluster	Highly available PostgreSQL (keys, budgets, spend).
cache	yes	replicated	Highly available Redis, mandatory with several instances (shared limits and cache).

Topology per mode

Modes	Highest catalog level	Distribution	Quorum component	Fault domains	Description
single	standalone (standalone)	single (single)	-	1	One instance, no fault tolerance.
replicated	ha (ha)	active-active (active-active)	-	2	Several proxies behind HAProxy, sharing PostgreSQL and the Redis cache.

Notes

• Redis is no longer optional with several instances: without it the rate limits and budgets are counted per instance.
• With TLS on, the HAProxy in front of the replicated proxies must reach them in TLS too (backend servers with ssl and the Manager CA as ca-file), or pass the TLS stream through in tcp mode.

Configuration parameters

Network

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
API port (port)	port	4000	-	no	no	yes	Port of the proxy API (behind HAProxy in replicated mode).
TLS (tls_enabled)	bool	false	-	no	no	yes	Serve the proxy API, the admin console /ui and /metrics/ over TLS with the certificate the Manager issues to each host from its CA (SSL_CERTFILE_PATH and SSL_KEYFILE_PATH of the proxy); the binding of its consumers then uses https. Independently of this parameter, LiteLLM calls in https every engine of the Manager whose binding is https (vLLM with its TLS parameter) and verifies its certificate against the Manager CA.
Workers (workers)	int	4	min 1, max 64	no	no	yes	Uvicorn worker processes of the proxy. (recommended clamp(host.cpu_count, 1, 16))

Models and routing

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
Model list (model_list)	json	[]	-	no	no	no	model_list of the LiteLLM configuration (JSON); secrets only as os.environ/NAME references to the provider credentials.
Router settings (router_settings)	json	{"routing_strategy": "simple-shuffle", "num_retries": 2, "timeout": 120, "allowed_fails": 3, "cooldown_time": 60}	-	no	no	no	router_settings of the configuration (fallbacks, retries, timeouts).
Request timeout (request_timeout)	int	120	min 5, max 3600	no	no	no	Seconds before a provider call is abandoned.

Security

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
Master key (master_key)	password	generated	-	no	yes	no	Admin key of the proxy (EAI_LITELLM_MASTER_KEY of the Control Plane only); prefixed with sk-.
Consumer key (consumer_key)	password	generated	-	no	yes	no	Non admin virtual key navin-consumers (prefixed with sk-), registered by the node at startup with POST /key/generate (LLM routes and the models of model_list only) and published to the consumers through the api binding; never equal to the master key. One shared key: spend is not tracked per consumer service.
Salt key (salt_key)	password	generated	-	no	yes	no	Encryption key of the credentials stored in the database (never change it after the first start).
Admin console user (ui_username)	string	admin	^[A-Za-z0-9_.@-]{3,64}$	no	no	no	Login of the admin console /ui (UI_USERNAME).
Admin console password (ui_password)	password	generated	-	no	yes	no	Password of the admin console /ui (UI_PASSWORD), generated at each install and kept in the Manager vault.
Provider credentials (provider_env)	json	{}	-	no	yes	no	Map NAME: value of the provider credentials and endpoints referenced as os.environ/NAME, for example VLLM_API_BASE and VLLM_API_KEY of an existing vLLM server or OLLAMA_API_BASE of an existing Ollama server (written to the sensitive file providers.yaml, 0640).

Observability

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
Opik project (opik_project)	string	navin-gateway	-	no	no	no	Opik project of the traces (when bound).
Log level (log_level)	enum	INFO	DEBUG, INFO, WARNING, ERROR	no	no	no	Verbosity of the proxy log.

Dependencies

For each active dependency the Manager takes the first of: the binding chosen in the wizard; a service of the same installation that provides it; a service already installed in the cluster; the only connection of the expected type. Otherwise an optional dependency stays unbound, and a required one is added to the installation in its default mode (the column below names the catalog service added).

Binding	Provided by	Connection type	Endpoint	Optional	When nothing is bound	Description
database	postgresql	postgresql	sql	no	postgresql is added to the installation	Database of the virtual keys, spend logs and teams.
cache	redis	redis	redis	yes	stays unbound (optional)	Response cache and router state shared by the replicas.
vllm	vllm	-	api	yes	stays unbound (optional)	Managed vLLM engine: one hosted_vllm entry per instance under the served model name (LiteLLM balances the replicas); its API key stays in the proxy environment.
ollama	ollama	-	api	yes	stays unbound (optional)	Managed Ollama engine: one ollama_chat entry per published model and per instance.
opik	opik	-	api	yes	stays unbound (optional)	Tracing of the LLM calls in Opik.
otel	otel-collector	-	otlp_http	yes	stays unbound (optional)	OpenTelemetry traces (OTLP over HTTP).

Endpoints

Endpoint	Scheme	Component	Port	Path	TLS parameter	Binding parameters	Binding secrets	Provisioning command
api	http	litellm-proxy	4000 (port)	-	tls_enabled	-	token	-
admin	http	litellm-proxy	4000 (port)	-	tls_enabled	-	token	-

Client configuration files

Files rendered for the client configuration download:

• litellm.env (client.env.j2)

Commands

Command	Component	Target	Description
Readiness (readiness)	litellm-proxy	every host	Checks the database and cache connections of every proxy.
Recent logs (logs)	litellm-proxy	chosen host	Last lines of the proxy journal.

Health checks

Liveness (per instance)

Component	Check	Details
litellm-proxy	HTTP {{ 'https' if config.tls_enabled else 'http' }}://{{ host.fqdn if config.tls_enabled else (host.ip or '127.0.0.1') }}:{{ config.port }}/health/liveliness	Expected 200; 60 retries; interval 2 s; timeout 5 s
litellm-proxy	{{ paths.opt }}/venv/bin/python {{ paths.etc }}/register.py --check --env {{ paths.etc }}/consumer-key.env	60 retries; interval 3 s; timeout 30 s

Readiness (service check)

Check	Details
HTTP {{ 'https' if config.tls_enabled else 'http' }}://{{ host.fqdn if config.tls_enabled else (host.ip or '127.0.0.1') }}:{{ config.port }}/health/readiness	Expected 200; 10 retries; interval 3 s; timeout 10 s

Preflight checks

Check	Type	Meaning	Severity	Components
memory	memory_mb {"min": 2048}	at least 2048 MB of memory	blocking	every component
api_port_free	port_free {"param": "port"}	port 4000 free on the host	blocking	every component

Uninstall

Paths removed when the data is purged: {{ paths.log }}.

Quick links

• Admin console: {{ 'https' if config.tls_enabled else 'http' }}://{{ host.fqdn }}:{{ config.port }}/ui
• Metrics: {{ 'https' if config.tls_enabled else 'http' }}://{{ host.fqdn }}:{{ config.port }}/metrics/

Upstream documentation

• Homepage: https://docs.litellm.ai
• Source: https://github.com/BerriAI/litellm
• High availability: https://docs.litellm.ai/docs/proxy/prod
• Authentication: https://docs.litellm.ai/docs/proxy/virtual_keys