promptfoo (`promptfoo`)

Runs prompt and model test suites (assertions, regressions, red teaming) and serves the result viewer.

Generated from catalog/services/promptfoo.yaml: do not edit by hand.

Summary


Version	`0.117.4`
Use	execute on demand
Layer	functional (AI catalog)
Category	Evaluation and LLM operations
Installation methods	`podman`, `binary`
External connection	no
Sandbox only	no

Licence


SPDX identifier	`MIT`
Licence policy	core licence (MIT or Apache-2.0)
Licence exception	-
Platform component	`promptfoo`

Packages and artifacts

A package the project of the service publishes (its image, its binary archive, its Python or OS package) carries the licence of the service; a third-party package carries the licence written on its row, read from the official upstream metadata.

Kind	Package	Version	Licence	Details
container image	`ghcr.io/promptfoo/promptfoo`	`0.117.4`	`MIT`	image key `promptfoo`, used by `viewer`, digest `sha256:e260878c048eb220e824f867d48ffff5ace1ccff4ca8507b9e778c17c0b46cdb`
binary archive	`promptfoo-0.117.4.tgz`	`0.117.4`	`MIT`	https://registry.npmjs.org/promptfoo/-/promptfoo-0.117.4.tgz

Images are pinned by tag in the descriptor, and by digest when the descriptor declares one (the offline bundle then refuses a pulled image that does not carry it); the digest is recorded in the mirror of the Manager when the offline bundle is imported, and every pull or load is checked against it.

Supported systems and architectures

Method	Operating systems	Architectures
`podman`	RHEL family (dnf), Debian family (apt), Arch Linux (pacman)	x86_64, aarch64 (not restricted by the descriptor)
`binary`	RHEL family (dnf), Debian family (apt), Arch Linux (pacman)	x86_64, aarch64 (not restricted by the descriptor)

Certified host systems: RHEL 8/9/10, Rocky Linux 8/9/10, AlmaLinux 8/9/10, Oracle Linux 8/9, CentOS Stream 9/10, Ubuntu 22.04/24.04, Debian 12, Arch Linux (rolling). The host preflight accepts these systems (other releases of a known family give a warning, end of life releases are refused) and the x86_64 and aarch64 architectures.

Prerequisites

Host

The descriptor declares no host preflight check.

Network and ports

Component	Port name	Port	Opened in the host firewall
`viewer`	`http`	15500/tcp	yes

Services and connections needed

litellm: litellm (optional).

Details and resolution: see Dependencies.

Installation methods and modes

Method	Default	What the method uses
`podman`	yes	the container images of the descriptor, run by Podman under systemd
`binary`	-	the binary archives of the descriptor, unpacked under the service directory

Components

Component	Kind	Installation methods	Ports
Viewer (`viewer`)	master	`podman`, `binary`	15500/tcp

Single (`single`) (default)

One viewer.

What the mode deploys:

Component	Placement
`viewer`	min 1, max 1

Install this service

The install wizard of the Navin Manager plans, validates and runs the installation; you need the services:install permission on the cluster.

Open the wizard. In Clusters, choose the row action Install services of the target cluster. The wizard opens on its first step, Service.
Choose the service. On the Service step, select promptfoo. The card shows the licence MIT and the dependencies the installation needs. Pick the Method among podman, binary (default podman).
Choose the hosts and the mode. On the Hosts step, tick the hosts, then choose the Mode: single (default). Each mode deploys the components listed in Installation methods and modes.
Bind the dependencies. On the Bindings step, choose for each dependency (litellm) Install with the platform or Use an existing connection; an optional dependency can stay on No binding.
Configure. On the Configure step, review the parameters (see Configuration parameters); the required ones are llm_api_key. Passwords left empty with a generated default are generated and sealed by the Manager.
Review and install. On the Install step the Manager validates the plan and shows the installation order; when it says "The plan is valid. Ready to install.", choose Install. A highly available mode is installed only when its HA checks pass.
Verify. The operation ends when the health checks of every instance pass (see Health checks); the service then shows as running in Services.

On demand runs

Runs a prompt and model test suite (assertions, regressions) when a test is asked and keeps the results for the viewer.

Portal screen: /quality/suites

High availability


Cluster	none (`none`)
High availability	none (`none`)
Failover	none (`none`)
Replication	none (`none`)
Quorum	none (`none`)
Discovery	none (`none`)
Entry point	none (`none`)
Official documentation	https://www.promptfoo.dev/docs/usage/web-ui/

Topology per mode

Modes	Highest catalog level	Distribution	Quorum component	Fault domains	Description
`single`	standalone (`standalone`)	parallel jobs (`parallel-jobs`)	-	-	One viewer; evaluations run as independent jobs.

Notes

A command line tool run on demand: never a highly available cluster; several scans run as independent parallel jobs.

Configuration parameters

General

Parameter	Type	Default	Allowed values	Required	Secret	Restart	Description
Viewer port (`port`)	`port`	`15500`	-	no	no	no	Port of promptfoo view.
Node.js runtime (`node`)	`path`	`/usr/bin/node`	-	no	no	no	Node.js 20 or later for the binary method (offline prerequisite).
OpenAI compatible endpoint (`llm_base_url`)	`url`	`http://127.0.0.1:4000/v1`	-	no	no	no	Base URL of the models under test when no LiteLLM binding is set (an address of the host; 127.0.0.1 is unreachable from the container of the podman method).
API key (`llm_api_key`)	`password`	-	-	yes	yes	no	Key of the endpoint.

Dependencies

For each active dependency the Manager takes the first of: the binding chosen in the wizard; a service of the same installation that provides it; a service already installed in the cluster; the only connection of the expected type. Otherwise an optional dependency stays unbound, and a required one is added to the installation in its default mode (the column below names the catalog service added).

Binding	Provided by	Connection type	Endpoint	Optional	When nothing is bound	Description
`litellm`	`litellm`	-	`api`	yes	stays unbound (optional)	LiteLLM gateway of the platform serving the models under test (OPENAI_BASE_URL); without it the endpoint parameter is used.

Commands

Command	Component	Target	Description
Run a test suite (`eval`)	`viewer`	first host	Runs promptfooconfig.yaml of a directory and stores the result for the viewer.

Command parameters

Command	Parameter	Type	Default	Required	Description
`eval`	Suite directory (`suite_dir`)	`path`	-	yes	Directory holding promptfooconfig.yaml.

Health checks

Liveness (per instance)

Component	Check	Details
`viewer`	HTTP `http://127.0.0.1:{{ config.port }}/health`	Expected 200; 30 retries; interval 2 s; timeout 5 s

Preflight checks

Check	Type	Meaning	Severity	Components
`port_free`	`port_free` `{"param": "port"}`	port 15500 free on the host	blocking	every component

Uninstall

Paths removed when the data is purged: {{ paths.data }}.

Quick links

promptfoo viewer: http://{{ host.fqdn }}:{{ config.port }}/

Upstream documentation

Homepage: https://www.promptfoo.dev
Source: https://github.com/promptfoo/promptfoo
High availability: https://www.promptfoo.dev/docs/usage/web-ui/

Summary​

Licence​

Packages and artifacts​

Supported systems and architectures​

Prerequisites​

Host​

Network and ports​

Services and connections needed​

Installation methods and modes​

Components​

Single (single) (default)​

Install this service​

On demand runs​

High availability​

Topology per mode​

Notes​

Configuration parameters​

General​

Dependencies​

Commands​

Command parameters​

Health checks​

Liveness (per instance)​

Preflight checks​

Uninstall​

Quick links​

Upstream documentation​