Ollama (ollama)

Inference engine for light and quantized models with an OpenAI compatible and a native API, on CPU hosts or NVIDIA GPU hosts. Several models per instance (loaded on demand), parallel requests and queue settings, models hosted on the host (pulled from a registry or imported from validated GGUF files). Registered automatically in LiteLLM.

Generated from catalog/services/ollama.yaml: do not edit by hand.

Summary

Version	0.40.14
Use	install and configure, configure and connect
Layer	functional (AI catalog)
Category	AI serving
Installation methods	podman
External connection	no
Sandbox only	no

Licence

SPDX identifier	MIT
Licence policy	core licence (MIT or Apache-2.0)
Licence exception	-
Platform component	ollama

Packages and artifacts

A package the project of the service publishes (its image, its binary archive, its Python or OS package) carries the licence of the service; a third-party package carries the licence written on its row, read from the official upstream metadata.

Kind	Package	Version	Licence	Details
container image	docker.io/ollama/ollama	0.40.14	MIT	image key ollama, used by server, digest sha256:ebaef4f4c5d93660e1f25f8ad085a112f51ccfb0e8458073e71793959dba91c9
container image	docker.io/nginxinc/nginx-unprivileged	1.30.6-alpine-slim	MIT	image key tls_front, used by tls-front, digest sha256:5d8bb9437d74a600ec5dbf90c035451c724dc33eac8d3a71078f6f8612c345f6

Images are pinned by tag in the descriptor, and by digest when the descriptor declares one (the offline bundle then refuses a pulled image that does not carry it); the digest is recorded in the mirror of the Manager when the offline bundle is imported, and every pull or load is checked against it.

Supported systems and architectures

Method	Operating systems	Architectures
podman	RHEL family (dnf), Debian family (apt), Arch Linux (pacman)	x86_64, aarch64 (not restricted by the descriptor)

Certified host systems: RHEL 8/9/10, Rocky Linux 8/9/10, AlmaLinux 8/9/10, Oracle Linux 8/9, CentOS Stream 9/10, Ubuntu 22.04/24.04, Debian 12, Arch Linux (rolling). The host preflight accepts these systems (other releases of a known family give a warning, end of life releases are refused) and the x86_64 and aarch64 architectures.

GPU: NVIDIA GPU given to the component as CDI devices (nvidia.com/gpu): server (when the configuration asks for it).

Prerequisites

Host

• at least 4096 MB of memory (blocking, memory).
• at least 1 NVIDIA GPU with 4000 MB of memory, only when accelerator is gpu (blocking, gpu).
• at least 4 CPU, only when accelerator is cpu (warning, cpu_cores).
• the program podman installed on the host (blocking, podman).
• at least 20480 MB free for /var/lib/navin (warning, model_disk).

Programs the commands and checks of the service start on the host, outside the managed roots: add them to allowed_commands of the agent (see Security of the Navin agent), else the agent refuses them with command_not_allowed: /usr/bin/nvidia-smi, /usr/bin/podman.

Network and ports

Component	Port name	Port	Opened in the host firewall
server	api	11434/tcp	yes
server	api_internal	11434/tcp	no
tls-front	api	11434/tcp	yes

Services and connections needed

This service has no dependency.

Installation methods and modes

Method	Default	What the method uses
podman	yes	the container images of the descriptor, run by Podman under systemd

Components

Component	Kind	Installation methods	Ports
Ollama server (server)	master	podman	11434/tcp, 11434/tcp
TLS front (tls-front)	master	podman	11434/tcp

Single instance (sandbox) (single) (default)

One engine on one CPU or GPU host.

What the mode deploys:

Component	Placement
server	min 1, max 1
tls-front	min 1

Replicated (high availability) (replicated)

Independent instances with the same models on distinct hosts; LiteLLM balances and fails over between them. The model commands run on every instance.

What the mode deploys:

Component	Placement
server	min 2, recommended 2
tls-front	min 1

Install this service

The install wizard of the Navin Manager plans, validates and runs the installation; you need the services:install permission on the cluster.

1. Open the wizard. In Clusters, choose the row action Install services of the target cluster. The wizard opens on its first step, Service.

2. Choose the service. On the Service step, select Ollama. The card shows the licence MIT.

3. Choose the hosts and the mode. On the Hosts step, tick the hosts, then choose the Mode: single (default), replicated. Each mode deploys the components listed in Installation methods and modes. With several components, place each one on its hosts in Component placement (Suggest placement proposes a placement that respects the cardinalities).

4. Bindings. This service has no dependency: the Bindings step has nothing to bind.

5. Configure. On the Configure step, review the parameters (see Configuration parameters). Passwords left empty with a generated default are generated and sealed by the Manager.

6. Review and install. On the Install step the Manager validates the plan and shows the installation order; when it says "The plan is valid. Ready to install.", choose Install. A highly available mode is installed only when its HA checks pass.

7. Verify. The operation ends when the health checks of every instance pass (see Health checks); the service then shows as running in Services.

Connect an existing service

Schemes	https, http
Default port	11434
Authentication	none
Health check	/api/version

Authentication method	Fields	Description
none	-	Ollama has no authentication of its own: keep it on the internal network or behind an authenticating gateway.

Official documentation: https://github.com/ollama/ollama/blob/main/docs/faq.md

High availability

Cluster	replicas (replication)
High availability	redundancy (redundancy)
Failover	client side (client)
Replication	none (none)
Quorum	none (none)
Discovery	static (static)
Entry point	any node (any-node)
Official documentation	https://docs.ollama.com/faq
References	https://docs.ollama.com/docker, https://docs.litellm.ai/docs/routing

Prerequisites

Prerequisite	Must be highly available	Modes	Description
litellm	yes	replicated	Highly available LiteLLM (two proxies or more): it balances and fails over between the instances; agents and applications never call Ollama directly.

Topology per mode

Modes	Highest catalog level	Distribution	Quorum component	Fault domains	Description
single	standalone (standalone)	single (single)	-	1	One engine on one CPU or GPU host, no fault tolerance.
replicated	ha (ha)	active-active (active-active)	-	2	Independent instances with the same models on distinct hosts; LiteLLM fails over between them.

Notes

• The models are stored on each host: pull or import them on every instance (the model commands run on all of them) before publishing them in LiteLLM.
• Ollama serves its API in plain HTTP only (OLLAMA_HOST takes no certificate). With TLS on, a TLS front (nginx with the certificate of the Manager) placed next to each engine takes the API port, the engine stays on the network of the service and LiteLLM verifies the certificate with the CA of the Manager. Turning TLS on or off adds or removes the fronts at the next configuration apply; LiteLLM is then shown with changes to apply.

Configuration parameters

Models and hosting

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
Models published in LiteLLM (litellm_models)	list	["llama3.2"]	-	no	no	no	Models of this engine LiteLLM exposes (one ollama_chat entry per model and per instance); pull or import them on every instance first.
Keep loaded (keep_alive)	string	5m	^(-1|0|[0-9]+[smh])$	no	no	yes	How long an idle model stays in memory (OLLAMA_KEEP_ALIVE): 5m, 1h, -1 forever, 0 unload at once.

CPU or GPU host

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
Host type (accelerator)	enum	gpu	gpu, cpu	no	no	yes	gpu: NVIDIA GPU host (driver and NVIDIA Container Toolkit CDI specification required); cpu: CPU only host (quantized models).
GPUs (gpu_devices)	string	all	^(all|[A-Za-z0-9-]+(,[A-Za-z0-9-]+)*)$	no	no	yes	GPUs given to the engine: all, or a comma list of indexes or UUIDs (GPU pool distinct from the vLLM pool).

Concurrency and memory

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
Parallel requests per model (num_parallel)	int	4	min 1, max 64	no	no	yes	OLLAMA_NUM_PARALLEL: requests a loaded model serves at once (the context memory grows with it).
Loaded models (max_loaded_models)	int	2	min 1, max 32	no	no	yes	OLLAMA_MAX_LOADED_MODELS: models kept loaded at the same time, within the VRAM or the RAM.
Queue (max_queue)	int	512	min 1, max 100000	no	no	yes	OLLAMA_MAX_QUEUE: waiting requests before the engine answers busy (LiteLLM then retries another instance).
Context length (context_length)	int	4096	min 512, max 1048576	no	no	yes	OLLAMA_CONTEXT_LENGTH: default context window of the loaded models.
Flash attention (flash_attention)	bool	true	-	no	no	yes	OLLAMA_FLASH_ATTENTION: lower memory use of the context on supported GPUs.
KV cache precision (kv_cache_type)	enum	f16	f16, q8_0, q4_0	no	no	yes	OLLAMA_KV_CACHE_TYPE (needs flash attention): q8_0 halves the context memory.

Network

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
API port (port)	port	11434	-	no	no	yes	Port of the API, bound to the host address and reached by LiteLLM only (served by the TLS front when TLS is on).

Security

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
TLS of the API (tls_enabled)	bool	false	-	no	no	yes	Serves the API in HTTPS with the certificate of the Manager: Ollama has no TLS of its own, a TLS front (nginx) placed next to each engine takes the API port and the engine is no longer published on the host. LiteLLM then verifies the certificate with the CA of the Manager.

Endpoints

Endpoint	Scheme	Component	Port	Path	TLS parameter	Binding parameters	Binding secrets	Provisioning command
api	http	server	11434 (port)	-	tls_enabled	models, engine	-	-

Commands

Command	Component	Target	Description
Hosted models (list_models)	server	every host	Models stored on every instance, with their size (ollama list).
Loaded models (running_models)	server	every host	Models loaded in memory on every instance, with the CPU and GPU share (ollama ps).
Download a model (pull_model)	server	every host	Downloads a model on every instance from the Ollama registry or an internal registry (full reference registry/namespace/model:tag).
Import a GGUF model (import_model)	server	every host	Creates a model on every instance from a validated GGUF file copied to /var/lib/navin/ollama/import of each host (offline installation, no download).
Import a fine-tuned model (import_safetensors)	server	every host	Creates a model on every instance from a Safetensors folder of /var/lib/navin/ollama/import (merged export of LLaMA-Factory, copied by its publish_to_ollama command), for the architectures Ollama imports (Llama, Mistral, Gemma, Qwen). Then add the model name to litellm_models.
Delete a model (delete_model)	server	every host	Removes a model from every instance (ollama rm). (asks for confirmation)
GPU status (gpu_status)	server	every host	Usage, memory and temperature of the GPUs of every host (nvidia-smi).
Recent logs (logs)	server	chosen host	Last lines of the engine journal.

Command parameters

Command	Parameter	Type	Default	Required	Description
pull_model	Model (model)	string	-	yes	Model reference, for example llama3.2:3b or registry.example.internal/library/qwen3:8b.
import_model	Model name (model)	string	-	yes	Name of the created model, for example qwen3-8b-q4.
import_model	GGUF file (file)	string	-	yes	File name in the import folder of the host.
import_safetensors	Model name (model)	string	-	yes	Name of the created model, for example support-qwen3-8b.
import_safetensors	Folder (folder)	string	-	yes	Folder name in the import folder of the host.
delete_model	Model (model)	string	-	yes	Model reference as shown by Hosted models.

Health checks

Liveness (per instance)

Component	Check	Details
server	HTTP http://{{ host.ip or '127.0.0.1' }}:{{ config.port }}/api/version	Expected 200; 60 retries; interval 2 s; timeout 5 s
server	/usr/bin/podman exec navin-{{ service.name }}-server ollama list	60 retries; interval 2 s; timeout 10 s
tls-front	HTTP https://{{ host.fqdn }}:{{ config.port }}/api/version	Expected 200; 60 retries; interval 2 s; timeout 5 s

Readiness (service check)

Check	Details
HTTP {{ 'https://' ~ host.fqdn if config.tls_enabled else 'http://' ~ (host.ip or '127.0.0.1') }}:{{ config.port }}/api/tags	Expected 200; 10 retries; interval 3 s; timeout 10 s

Preflight checks

Check	Type	Meaning	Severity	Components
memory	memory_mb {"min": 4096}	at least 4096 MB of memory	blocking	every component
gpu	gpu {"min_count": 1, "min_memory_mb": 4000}	at least 1 NVIDIA GPU with 4000 MB of memory, only when accelerator is gpu	blocking	every component
cpu_cores	cpu_count {"min": 4}	at least 4 CPU, only when accelerator is cpu	warning	every component
podman	binary {"name": "podman"}	the program podman installed on the host	blocking	every component
model_disk	disk_free_mb {"path": "/var/lib/navin", "min": 20480}	at least 20480 MB free for /var/lib/navin	warning	every component
api_port_free	port_free {"param": "port"}	port 11434 free on the host	blocking	every component

Uninstall

Paths removed when the data is purged: {{ paths.data }}, {{ paths.log }}.

Upstream documentation

• Homepage: https://ollama.com
• Source: https://github.com/ollama/ollama
• High availability: https://docs.ollama.com/faq
• High availability: https://docs.ollama.com/docker
• High availability: https://docs.litellm.ai/docs/routing
• Authentication: https://github.com/ollama/ollama/blob/main/docs/faq.md