vLLM (vllm)

Production inference engine with an OpenAI compatible API: one model per installation, served on NVIDIA GPU hosts (tensor parallelism on several GPUs of a host, Ray pipeline parallelism across hosts) or on CPU hosts (x86 AVX512 or AVX2 build). Model weights are hosted on the host (Hugging Face cache or imported folders), never downloaded at start in offline mode. Registered automatically in LiteLLM.

Generated from catalog/services/vllm.yaml: do not edit by hand.

Summary

Version	0.33.0
Use	install and configure, configure and connect
Layer	functional (AI catalog)
Category	AI serving
Installation methods	podman
External connection	no
Sandbox only	no

Licence

SPDX identifier	Apache-2.0
Licence policy	core licence (MIT or Apache-2.0)
Licence exception	-
Platform component	vllm

Packages and artifacts

A package the project of the service publishes (its image, its binary archive, its Python or OS package) carries the licence of the service; a third-party package carries the licence written on its row, read from the official upstream metadata.

Kind	Package	Version	Licence	Details
container image	docker.io/vllm/vllm-openai	v0.33.0	Apache-2.0	image key vllm, used by server, ray-worker, digest sha256:d825baf9ce4b1b6bfc5ecc7b79880c7e6748c65a72d697ce3d47607aafa84c6e
container image	docker.io/vllm/vllm-openai-cpu	v0.33.0-x86_64	Apache-2.0	image key vllm_cpu, used by server, ray-worker, digest sha256:4154467a886f8b7ffa9dbab32f873f5ca4528409d042a6ec866e8888223263d3

Images are pinned by tag in the descriptor, and by digest when the descriptor declares one (the offline bundle then refuses a pulled image that does not carry it); the digest is recorded in the mirror of the Manager when the offline bundle is imported, and every pull or load is checked against it.

Supported systems and architectures

Method	Operating systems	Architectures
podman	RHEL family (dnf), Debian family (apt), Arch Linux (pacman)	x86_64, aarch64 (not restricted by the descriptor)

Certified host systems: RHEL 8/9/10, Rocky Linux 8/9/10, AlmaLinux 8/9/10, Oracle Linux 8/9, CentOS Stream 9/10, Ubuntu 22.04/24.04, Debian 12, Arch Linux (rolling). The host preflight accepts these systems (other releases of a known family give a warning, end of life releases are refused) and the x86_64 and aarch64 architectures.

GPU: NVIDIA GPU given to the component as CDI devices (nvidia.com/gpu): server (when the configuration asks for it), ray-worker (when the configuration asks for it).

Prerequisites

Host

• at least 8192 MB of memory (blocking, memory).
• at least 1 NVIDIA GPU with 8000 MB of memory, only when accelerator is gpu (blocking, gpu).
• at least 8 CPU, only when accelerator is cpu (warning, cpu_cores).
• the program podman installed on the host (blocking, podman).
• at least 51200 MB free for /var/lib/navin (warning, model_disk).

Programs the commands and checks of the service start on the host, outside the managed roots: add them to allowed_commands of the agent (see Security of the Navin agent), else the agent refuses them with command_not_allowed: /usr/bin/nvidia-smi, /usr/bin/podman.

Network and ports

Component	Port name	Port	Opened in the host firewall
server	api	8000/tcp	yes
server	ray	6379/tcp	yes
server	ray_node	6381-6382/tcp	yes
server	ray_workers	20000-20199/tcp	yes
ray-worker	ray_node	6381-6382/tcp	yes
ray-worker	ray_workers	20000-20199/tcp	yes

Services and connections needed

This service has no dependency.

Installation methods and modes

Method	Default	What the method uses
podman	yes	the container images of the descriptor, run by Podman under systemd

Components

Component	Kind	Installation methods	Ports
vLLM server (server)	master	podman	8000/tcp, 6379/tcp, 6381-6382/tcp, 20000-20199/tcp
Ray worker (ray-worker)	worker	podman	6381-6382/tcp, 20000-20199/tcp

Single instance (sandbox) (single) (default)

One engine on one CPU or GPU host.

What the mode deploys:

Component	Placement
server	min 1, max 1
ray-worker	not used

Replicated (high availability) (replicated)

Independent instances of the same model on distinct hosts; LiteLLM balances, retries and fails over between them.

What the mode deploys:

Component	Placement
server	min 2, recommended 2
ray-worker	not used

Multi node (large model) (distributed)

One large model split across GPU hosts with Ray (pipeline parallelism): one server host and Ray worker hosts. Not high availability by itself.

What the mode deploys:

Component	Placement
server	min 1, max 1
ray-worker	min 1, recommended 1

Install this service

The install wizard of the Navin Manager plans, validates and runs the installation; you need the services:install permission on the cluster.

1. Open the wizard. In Clusters, choose the row action Install services of the target cluster. The wizard opens on its first step, Service.

2. Choose the service. On the Service step, select vLLM. The card shows the licence Apache-2.0.

3. Choose the hosts and the mode. On the Hosts step, tick the hosts, then choose the Mode: single (default), replicated, distributed. Each mode deploys the components listed in Installation methods and modes. With several components, place each one on its hosts in Component placement (Suggest placement proposes a placement that respects the cardinalities).

4. Bindings. This service has no dependency: the Bindings step has nothing to bind.

5. Configure. On the Configure step, review the parameters (see Configuration parameters); the required ones are model. Passwords left empty with a generated default are generated and sealed by the Manager.

6. Review and install. On the Install step the Manager validates the plan and shows the installation order; when it says "The plan is valid. Ready to install.", choose Install. A highly available mode is installed only when its HA checks pass.

7. Verify. The operation ends when the health checks of every instance pass (see Health checks); the service then shows as running in Services.

Connect an existing service

Schemes	https, http
Default port	8000
Authentication	optional
Health check	/health

Authentication method	Fields	Description
bearer	token	Key of --api-key (or VLLM_API_KEY), sent as Authorization: Bearer on the OpenAI compatible API.
none	-	Without --api-key the OpenAI compatible API of vLLM is open to anyone who reaches it.

Official documentation: https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html

High availability

Cluster	replicas (replication)
High availability	redundancy (redundancy)
Failover	client side (client)
Replication	none (none)
Quorum	none (none)
Discovery	static (static)
Entry point	any node (any-node)
Official documentation	https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html
References	https://docs.vllm.ai/en/latest/deployment/k8s.html, https://docs.litellm.ai/docs/routing

Prerequisites

Prerequisite	Must be highly available	Modes	Description
litellm	yes	replicated	Highly available LiteLLM (two proxies or more): it balances, retries and fails over between the replicas; agents and applications never call vLLM directly.

Topology per mode

Modes	Highest catalog level	Distribution	Quorum component	Fault domains	Description
single	standalone (standalone)	single (single)	-	1	One engine on one CPU or GPU host, no fault tolerance.
replicated	ha (ha)	active-active (active-active)	-	2	Independent replicas of the same model on distinct hosts, each with its own GPUs and its own copy of the weights; LiteLLM fails over between them.
distributed	standalone (standalone)	single (single)	-	1	One large model split across GPU hosts with Ray: every host is needed and the loss of one stops the model.

Notes

• Splitting a model across GPUs or hosts is not high availability: a critical model needs the replicated mode, or a second independent installation under the same served model name behind LiteLLM.
• The TLS parameter protects the OpenAI compatible API of the server (with /health, /metrics and /docs). The Ray ports of the distributed mode (head, node manager, object manager and worker ports) and the NCCL traffic between the pipeline stages stay in clear text: keep the hosts of the distributed mode on a trusted internal network. The /docs page loads Swagger UI from a public CDN: on a host without Internet add --enable-offline-docs to extra_args.

Configuration parameters

Model and hosting

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
Model (model)	string	Qwen/Qwen3-0.6B	-	yes	no	yes	Hugging Face identifier of the model (source cache, kept in the Hugging Face cache of the host below /var/lib/navin/vllm/models/hf) or folder name of imported weights below /var/lib/navin/vllm/models/local (source local).
Model source (model_source)	enum	cache	cache, local	no	no	yes	cache: Hugging Face cache of the host (filled by the download_model command or copied); local: validated weights imported in a folder of the host.
Served model name (served_model_name)	string	qwen-local	^[A-Za-z0-9][A-Za-z0-9._/-]{0,127}$	no	no	yes	Model name of the API and of the LiteLLM entry; replicas of the same model share it so that LiteLLM balances between them.
Runner (runner)	enum	auto	auto, generate, pooling	no	no	yes	generate for chat and completion models, pooling for embedding and reranking models (registered as embedding in LiteLLM).
Vision model (VLM) (vision)	bool	false	-	no	no	no	The model reads images and video (for example Qwen3-VL, or a VLM fine-tuned with LLaMA-Factory): LiteLLM declares it with supports_vision. The number of images or videos per request goes into extra_args, for example --limit-mm-per-prompt then {"image": 4, "video": 1}.
Offline (offline)	bool	true	-	no	no	yes	Never download at start (HF_HUB_OFFLINE=1): the weights must already be on the host. Required for a production host without Internet.
Model repository mirror (hf_endpoint)	string	(empty)	-	no	no	no	Internal Hugging Face compatible mirror used by download_model (HF_ENDPOINT), empty for huggingface.co.
Repository token (hf_token)	password	-	-	no	yes	no	Token of the gated models of the repository (HF_TOKEN), optional.

CPU or GPU host

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
Host type (accelerator)	enum	gpu	gpu, cpu	no	no	yes	gpu: NVIDIA GPU host (driver and NVIDIA Container Toolkit CDI specification required); cpu: x86 CPU host (AVX512 recommended), for small models and sandboxes.
GPUs (gpu_devices)	string	all	^(all|[A-Za-z0-9-]+(,[A-Za-z0-9-]+)*)$	no	no	yes	GPUs given to the engine: all, or a comma list of indexes or UUIDs (dedicated GPUs of a tenant pool).
Minimum VRAM per GPU (min_gpu_memory_mb)	int	8000	min 0, max 1048576	no	no	no	Preflight: VRAM a GPU of the host must have (MB).
CPU KV cache (cpu_kvcache_space_gb)	int	8	min 1, max 1024	no	no	yes	KV cache of a CPU host in GB (VLLM_CPU_KVCACHE_SPACE).
CPU cores (cpu_threads_bind)	string	auto	-	no	no	yes	Cores of the OpenMP threads of a CPU host (VLLM_CPU_OMP_THREADS_BIND): auto, or a range such as 0-31.

Serving and memory

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
GPUs per instance (tensor_parallel_size)	int	1	min 1, max 16	no	no	yes	Tensor parallelism: the model is split on this number of GPUs of each host. Splitting is not high availability: use the replicated mode for that.
GPU memory share (gpu_memory_utilization)	float	0.85	min 0.1, max 0.98	no	no	yes	Share of the VRAM of each GPU vLLM reserves for the weights and the KV cache.
Context length (max_model_len)	int	8192	min 256, max 1048576	no	no	yes	Maximum tokens of a request (prompt and answer).
Concurrent sequences (max_num_seqs)	int	128	min 1, max 4096	no	no	yes	Requests batched together by the scheduler.
Data type (dtype)	enum	auto	auto, bfloat16, float16, float32	no	no	yes	Precision of the weights and activations (bfloat16 on CPU hosts).
Quantization (quantization)	string	(empty)	^(|awq|gptq|fp8|compressed-tensors|awq_marlin|gptq_marlin)$	no	no	yes	Quantization method of the weights, empty to detect it from the model.
Shared memory (shm_size)	string	8g	^[0-9]+[mg]$	no	no	yes	Size of /dev/shm of the container (tensor parallelism uses it).
Extra arguments (extra_args)	list	[]	-	no	no	yes	Further vllm serve arguments, one per item (for example --enable-prefix-caching).

Multi node (Ray)

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
Pipeline stages (pipeline_parallel_size)	int	2	min 2, max 64	no	no	yes	Hosts the model is split across (server plus Ray workers); usually the number of hosts of the distributed mode. (distributed)
Ray head port (ray_port)	port	6379	-	no	no	yes	Port of the Ray head (GCS) on the server host. (distributed)
Ray node port (ray_node_port)	port	6381	-	no	no	yes	Node manager port of every Ray node (the object manager uses the next port). (distributed)
First Ray worker port (ray_worker_port_start)	port	20000	-	no	no	yes	First port of the Ray worker processes (allowed between the hosts of the service). (distributed)
Last Ray worker port (ray_worker_port_end)	port	20199	-	no	no	yes	Last port of the Ray worker processes. (distributed)

Network

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
API port (port)	port	8000	-	no	no	yes	Port of the OpenAI compatible API, /health and /metrics; reached by LiteLLM only.

Security

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
API key (api_key)	password	generated	-	no	yes	no	Key required on /v1 (VLLM_API_KEY); given to LiteLLM through the binding, never to the agents.
TLS (tls_enabled)	bool	false	-	no	no	yes	Serve the OpenAI compatible API, /health, /metrics and /docs over TLS with the certificate the Manager issues to each host from its CA (--ssl-certfile, --ssl-keyfile, --ssl-ca-certs, reloaded on renewal by --enable-ssl-refresh); LiteLLM then calls the engine in https and verifies the Manager CA. The Ray ports of the distributed mode stay in clear text between the hosts of the service.
Log level (log_level)	enum	INFO	DEBUG, INFO, WARNING, ERROR	no	no	no	Verbosity of the engine log (VLLM_LOGGING_LEVEL).

Endpoints

Endpoint	Scheme	Component	Port	Path	TLS parameter	Binding parameters	Binding secrets	Provisioning command
api	http	server	8000 (port)	/v1	tls_enabled	model, mode, engine, vision	api_key	-

Commands

Command	Component	Target	Description
Served models (models)	server	every host	Lists the models every instance serves (GET /v1/models with the API key).
Download a model (download_model)	server	every host	Downloads the weights of a Hugging Face model into the cache of every host of the service (from huggingface.co or the internal mirror); then set the model parameter.
Download a model on the Ray workers (download_model_workers)	ray-worker	every host	Same download on every Ray worker host of the distributed mode.
Hosted models (hosted_models)	server	every host	Models stored on every host: Hugging Face cache and imported folders, with their size.
Delete a hosted model (delete_model)	server	every host	Removes the weights of a model that is not served any more from the cache and the imported folders of every host. (asks for confirmation)
GPU status (gpu_status)	server	every host	Usage, memory and temperature of the GPUs of every host (nvidia-smi).
Ray cluster status (ray_status)	server	first host	Nodes and resources of the Ray cluster of the distributed mode.
Recent logs (logs)	server	chosen host	Last lines of the engine journal.

Command parameters

Command	Parameter	Type	Default	Required	Description
download_model	Repository (repo)	string	-	yes	Hugging Face identifier, for example Qwen/Qwen3-8B.
download_model	Revision (revision)	string	main	no	Branch, tag or commit of the validated weights.
download_model_workers	Repository (repo)	string	-	yes	Hugging Face identifier, for example Qwen/Qwen3-8B.
download_model_workers	Revision (revision)	string	main	no	Branch, tag or commit of the validated weights.
delete_model	Model (repo)	string	-	yes	Hugging Face identifier (cache) or imported folder name.

Health checks

Liveness (per instance)

Component	Check	Details
server	HTTP {{ 'https' if config.tls_enabled else 'http' }}://{{ host.fqdn if config.tls_enabled else (host.ip or '127.0.0.1') }}:{{ config.port }}/health	Expected 200; 120 retries; interval 10 s; timeout 5 s
ray-worker	TCP {{ host.ip or '127.0.0.1' }}:{{ config.ray_node_port }}	60 retries; interval 5 s; timeout 5 s

Readiness (service check)

Check	Details
HTTP {{ 'https' if config.tls_enabled else 'http' }}://{{ host.fqdn if config.tls_enabled else (host.ip or '127.0.0.1') }}:{{ config.port }}/health	Expected 200; 10 retries; interval 5 s; timeout 10 s

Preflight checks

Check	Type	Meaning	Severity	Components
memory	memory_mb {"min": 8192}	at least 8192 MB of memory	blocking	every component
gpu	gpu {"min_count": 1, "min_memory_mb": 8000}	at least 1 NVIDIA GPU with 8000 MB of memory, only when accelerator is gpu	blocking	every component
cpu_cores	cpu_count {"min": 8}	at least 8 CPU, only when accelerator is cpu	warning	every component
podman	binary {"name": "podman"}	the program podman installed on the host	blocking	every component
model_disk	disk_free_mb {"path": "/var/lib/navin", "min": 51200}	at least 51200 MB free for /var/lib/navin	warning	every component
api_port_free	port_free {"param": "port"}	port 8000 free on the host	blocking	server

Uninstall

Paths removed when the data is purged: {{ paths.data }}, {{ paths.log }}.

Quick links

• Metrics: {{ 'https' if config.tls_enabled else 'http' }}://{{ host.fqdn }}:{{ config.port }}/metrics
• API documentation: {{ 'https' if config.tls_enabled else 'http' }}://{{ host.fqdn }}:{{ config.port }}/docs

Upstream documentation

• Homepage: https://docs.vllm.ai
• Source: https://github.com/vllm-project/vllm
• High availability: https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html
• High availability: https://docs.vllm.ai/en/latest/deployment/k8s.html
• High availability: https://docs.litellm.ai/docs/routing
• Authentication: https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html