Ragas (ragas)
Evaluation library of retrieval augmented generation (faithfulness, answer relevancy, context precision) run on demand against the LiteLLM gateway.
Generated from catalog/services/ragas.yaml: do not edit by hand.
Summary
| Version | 0.3.2 |
| Use | execute on demand |
| Layer | functional (AI catalog) |
| Category | Evaluation and LLM operations |
| Installation methods | python |
| External connection | no |
| Sandbox only | no |
Licence
| SPDX identifier | Apache-2.0 |
| Licence policy | core licence (MIT or Apache-2.0) |
| Licence exception | - |
| Platform component | ragas |
Packages and artifacts
A package the project of the service publishes (its image, its binary archive, its Python or OS package) carries the licence of the service; a third-party package carries the licence written on its row, read from the official upstream metadata.
| Kind | Package | Version | Licence | Details |
|---|---|---|---|---|
| Python package | ragas | 0.3.2 | Apache-2.0 | interpreters, in order of preference python3.11, python3.12, python3.10 |
| Python package | langchain-openai | 0.3.28 | MIT | interpreters, in order of preference python3.11, python3.12, python3.10 |
| Python package | datasets | 4.0.0 | Apache-2.0 | interpreters, in order of preference python3.11, python3.12, python3.10 |
Supported systems and architectures
| Method | Operating systems | Architectures |
|---|---|---|
python | RHEL family (dnf), Debian family (apt), Arch Linux (pacman) | x86_64, aarch64 (not restricted by the descriptor) |
Certified host systems: RHEL 8/9/10, Rocky Linux 8/9/10, AlmaLinux 8/9/10, Oracle Linux 8/9, CentOS Stream 9/10, Ubuntu 22.04/24.04, Debian 12, Arch Linux (rolling). The host preflight accepts these systems (other releases of a known family give a warning, end of life releases are refused) and the x86_64 and aarch64 architectures.
Prerequisites
Host
The descriptor declares no host preflight check.
Network and ports
None.
Services and connections needed
litellm:litellm(optional).
Details and resolution: see Dependencies.
Installation methods and modes
| Method | Default | What the method uses |
|---|---|---|
python | yes | a Python virtual environment holding the Python packages of the descriptor |
Components
| Component | Kind | Installation methods | Ports |
|---|---|---|---|
Ragas client (ragas) | client | python | - |
Client (client) (default)
Installed on the evaluation hosts, run on demand.
What the mode deploys:
| Component | Placement |
|---|---|
ragas | min 1, recommended 1 |
Install this service
The install wizard of the Navin Manager plans, validates and runs the installation; you need the services:install permission on the cluster.
-
Open the wizard. In Clusters, choose the row action Install services of the target cluster. The wizard opens on its first step, Service.
-
Choose the service. On the Service step, select Ragas. The card shows the licence
Apache-2.0and the dependencies the installation needs. -
Choose the hosts and the mode. On the Hosts step, tick the hosts, then choose the Mode:
client(default). Each mode deploys the components listed in Installation methods and modes. -
Bind the dependencies. On the Bindings step, choose for each dependency (
litellm) Install with the platform or Use an existing connection; an optional dependency can stay on No binding. -
Configure. On the Configure step, review the parameters (see Configuration parameters); the required ones are
llm_api_key. Passwords left empty with a generated default are generated and sealed by the Manager. -
Review and install. On the Install step the Manager validates the plan and shows the installation order; when it says "The plan is valid. Ready to install.", choose Install. A highly available mode is installed only when its HA checks pass.
-
Verify. The operation ends when the health checks of every instance pass (see Health checks); the service then shows as running in Services.
On demand runs
Scores a dataset of RAG answers (faithfulness, answer relevancy, context precision) when an evaluation run is asked; nothing runs between two runs.
Portal screen: /quality/runs
High availability
| Cluster | none (none) |
| High availability | none (none) |
| Failover | none (none) |
| Replication | none (none) |
| Quorum | none (none) |
| Discovery | none (none) |
| Entry point | none (none) |
| Official documentation | https://docs.ragas.io/en/v0.3.2/getstarted/rag_eval/ |
Topology per mode
| Modes | Highest catalog level | Distribution | Quorum component | Fault domains | Description |
|---|---|---|---|---|---|
client | standalone (standalone) | parallel jobs (parallel-jobs) | - | - | Installed on the evaluation hosts; evaluation workers are orchestrated outside the tool. |
Notes
- A command line tool run on demand: never a highly available cluster; several scans run as independent parallel jobs.
Configuration parameters
General
| Parameter | Type | Default | Allowed values | Required | Secret | Restart | Description |
|---|---|---|---|---|---|---|---|
OpenAI compatible endpoint (llm_base_url) | url | http://127.0.0.1:4000/v1 | - | no | no | no | Base URL of LiteLLM used as judge and for embeddings when no LiteLLM binding is set. |
API key (llm_api_key) | password | - | - | yes | yes | no | Virtual key of the gateway. |
Judge model (judge_model) | string | navin-judge | - | no | no | no | Model alias used to score the answers. |
Embedding model (embedding_model) | string | navin-embedding | - | no | no | no | Embedding alias of the gateway. |
Dependencies
For each active dependency the Manager takes the first of: the binding chosen in the wizard; a service of the same installation that provides it; a service already installed in the cluster; the only connection of the expected type. Otherwise an optional dependency stays unbound, and a required one is added to the installation in its default mode (the column below names the catalog service added).
| Binding | Provided by | Connection type | Endpoint | Optional | When nothing is bound | Description |
|---|---|---|---|---|---|---|
litellm | litellm | - | api | yes | stays unbound (optional) | LiteLLM gateway of the platform used as judge and for the embeddings (OPENAI_BASE_URL); without it the endpoint parameter is used. |
Commands
| Command | Component | Target | Description |
|---|---|---|---|
Evaluate a dataset (evaluate) | ragas | chosen host | Scores a JSONL dataset (question, answer, contexts, ground_truth) and writes the result next to it. |
Command parameters
| Command | Parameter | Type | Default | Required | Description |
|---|---|---|---|---|---|
evaluate | Dataset file (dataset) | path | - | yes | JSONL file under the data directory. |
Health checks
Liveness (per instance)
| Component | Check | Details |
|---|---|---|
ragas | {{ paths.opt }}/venv/bin/python -c import ragas | 1 retries; interval 1 s; timeout 60 s |
Uninstall
Paths removed when the data is purged: {{ paths.data }}, {{ paths.opt }}.
Upstream documentation
- Homepage: https://docs.ragas.io
- Source: https://github.com/explodinggradients/ragas
- High availability: https://docs.ragas.io/en/v0.3.2/getstarted/rag_eval/