Testing Guide

Version française : ../../fr/guides/testing.md

This guide shows how to test an agent, a skill, a plugin or a pack before it is deployed, and how the platform checks its own API, contracts and documentation. Who: developers and release managers. Every test below fails when a tool it needs is missing: nothing is skipped silently.

Functional

Every project template holds its pytest tests; evaluation suites run through POST /v1/evaluations/runs and the quality gate through POST /v1/evaluations/gates/check.

enterprise-ai test --suite regression --target agent/agt_ap@1.4.0

Screens: Quality > Evaluation suites defines the suites and their metrics, Quality > Evaluation runs shows each run and its scores, Quality > Quality gates shows the verdict that allows or blocks a deployment.

Security

enterprise-ai test security starts a red team run and exits with code 1 above the threshold; enterprise-ai test authz runs the OPA and OpenFGA tests.

enterprise-ai test security --threshold 0
enterprise-ai test authz

The red team engine is builtin by default; --engine garak and --probe <name> select another engine and probe. Screen: Quality > Red team.

Conformance and performance

enterprise-ai test sdk-conformance checks the contracts/v1 contracts of the SDK against the endpoint. The API surface is checked by services/control-plane/tests/test_api_surface.py; the developer documentation by services/control-plane/tests/test_developer_guides.py, which also compiles the TypeScript, Go and Java snippets of the SDK guides. Performance is measured by the telemetry SLOs (GET /v1/telemetry/slo).

enterprise-ai test sdk-conformance

Procedures

Run the unit tests of your project. They prove the logic of the agent or the skill without calling the platform, so they run first and fast.
```
python -m pytest -q tests
```
Expected result: every test passes; the template tests cover the manifest and the policies it ships.

If it fails: read the assertion message, fix the code, run the failing test alone.
Run the authorization tests. They prove that the OPA policies and the OpenFGA model of the project allow what they should and refuse the rest.
```
enterprise-ai test authz
```
Expected result: exit code 0, every opa test and fga model test case passes.

If it fails: exit code 1 names the failing case, or reports a missing opa or fga binary; install the binary and run the command again.
Run the red team. It sends the attack probes of the red team engine to the agent and measures the rate of successful attacks.
```
enterprise-ai test security --agent ticket-helper --threshold 0
```
Expected result: exit code 0, the attack success rate is at or below --threshold.

If it fails: exit code 1; open the run in Quality > Red team, fix the prompt, the policy or the tool scope that let the probe through, then run it again.
Run an evaluation suite. It scores the answers of the agent against the metrics and targets of the suite.
```
enterprise-ai test --suite regression --target agent/agt_ap@1.4.0
```
regression: name of the suite; agent/agt_ap@1.4.0: the agent and version under test.

Expected result: exit code 0, each metric meets its target.

If it fails: exit code 1 with the metric below its target; 404 not_found means an unknown suite.
Check the quality gate. The gate is what a deployment and a promotion read: the latest evaluation and red team results of the target.

Screen: Quality > Quality gates. API: POST /v1/evaluations/gates/check.

Expected result: the verdict allows the target.

If it fails: 409 gate_failed lists every failing check in details.failures.
Check the SDK conformance. It proves the SDK you use and the Control Plane agree on the contracts.
```
enterprise-ai test sdk-conformance
```
Expected result: exit code 0.

If it fails: exit code 1 reports a contract mismatch; upgrade the SDK (see the Migration Guide).
Run the platform checks (contributors). They prove the published API, the request surface and these guides match the code.
```
cd services/control-plane
.venv/bin/python -m pytest -q tests/test_api_surface.py tests/test_developer_guides.py tests/test_docs_kit.py
```
Expected result: every test passes.

If it fails: the assertion names the page and the route, command, setting or path that does not exist.
Measure performance (operators). The load generator starts a throwaway PostgreSQL, a fake model upstream and the Control Plane on free ports of this host, then runs the scenarios and the quota checks; nothing is sent to a cluster or a cloud account.
```
bash tools/perf/run.sh /tmp/eai-perf
```
Expected result: the output folder holds the measurements; compare them with the SLOs of Observability > SLO and error budgets (GET /v1/telemetry/slo).

If it fails: exit code 3 means the Control Plane virtualenv or redis-server is missing.

Running the tests of one repository unit (contributors)

The repository is a monorepo: every unit (Control Plane, Manager, sector agents) has its own pyproject.toml and its own .venv virtualenv. The system python has none of the project packages: python -m pytest from the root fails at collection with ModuleNotFoundError: No module named 'navin_courtier' (or navin_rag, navin_compliance, eai). Two rules are enough.

A test file runs with the virtualenv of its unit, from the unit folder. Never mix two units in one command: pytest would take the pyproject.toml of the first one as its rootdir.
```
cd navin-enterprise-assurance-courtier-distribution/backend
.venv/bin/python -m pytest -q tests/test_kpis_hardening.py
```
Expected result: 2 passed.
The Hub pipeline tests run with the system python3 and three external tools. opa is looked up on the PATH, then in OPA_BIN; yara_x and shellcheck are Python packages pinned by the Hub CI (shellcheck-py==0.11.0.1, yara-x==1.22.1). On a workstation where those tools live in /tmp/toolchains:
```
env OPA_BIN=/tmp/toolchains/opa \
    PYTHONPATH=/tmp/toolchains/yarax-1.22.1-venv/lib/python3.12/site-packages \
    python3 -m pytest -q hub/pipeline/tests/test_pipeline_files.py -k "not shellcheck"
```
Expected result: 57 passed, 1 deselected. Without -k, test_shellcheck_clean requires shellcheck to be installed; it then passes too (58 in total). The binaries in /tmp/toolchains vanish on reboot: an opa is required or No module named 'yara_x' failure after a reboot comes from that, not from the code.

Unit	Interpreter	Run from	Example
Control Plane	`services/control-plane/.venv/bin/python`	`services/control-plane`	`.venv/bin/python -m pytest -q tests/test_api_surface.py`
Manager (server)	`navin-manager/server/.venv/bin/python`	`navin-manager/server`	`.venv/bin/python -m pytest -q tests`
RAG Agent	`navin-enterprise-rag/backend/.venv/bin/python`	`navin-enterprise-rag/backend`	`.venv/bin/python -m pytest -q -p no:cacheprovider tests/test_template_artifacts.py`
Compliance & Audit	`navin-enterprise-compliance/backend/.venv/bin/python`	`navin-enterprise-compliance/backend`	`.venv/bin/python -m pytest -q -p no:cacheprovider tests/test_migrations.py`
Banking back-office	`navin-enterprise-banking-backoffice/backend/.venv/bin/python`	`navin-enterprise-banking-backoffice/backend`	`.venv/bin/python -m pytest -q tests`
Insurance agents (5 folders `navin-enterprise-assurance-*`)	`<folder>/backend/.venv/bin/python`	`<folder>/backend`	`.venv/bin/python -m pytest -q tests`
Hub pipeline	system `python3` + `OPA_BIN`, `yara_x`, `shellcheck`	repository root	see above
Contracts, packaging, tools (`contracts/tests`, `sdks/tests`, `os/tests`)	`services/control-plane/.venv/bin/python`	repository root	`services/control-plane/.venv/bin/python -m pytest -q contracts/tests`

Run the tests closest to your change, once, after the last edit; the full suite of a unit is the job of the CI.

Test tools

Need	Command	Used by
OCI, policy and observability tools	`bash tools/oci/fetch.sh`	pack tests, `opa test`, `fga model test`, `trivy fs`
Storage emulators	`bash tools/emulators/fetch.sh`	`services/control-plane/tests/test_storage_s3_seaweedfs.py` and the other storage tests
Pack interpreter	`bash packs/bootstrap.sh`	pack test suites
Load tests	`bash tools/perf/run.sh`	performance and quota measurements

Troubleshooting

Symptom	Cause	Fix
`enterprise-ai test authz` exit code 1, binary not found	`opa` or `fga` missing	install it (`bash tools/oci/fetch.sh` fetches pinned versions)
`enterprise-ai test security` exit code 1	attack success rate above `--threshold`	fix the weakness, run again; never raise the threshold to pass
409 `gate_failed`	an evaluation or red team result below target	read `details.failures`, fix, run the suite again
403 `forbidden` on an evaluation run	role missing (`details.required_roles`)	ask for a role that may write evaluations
400 `invalid_request` with `details.engine`	unknown red team engine	use `builtin` or `garak`
`tools/perf/run.sh` exit code 3	virtualenv or `redis-server` missing	create `services/control-plane/.venv`, install `redis-server`
`ModuleNotFoundError: No module named 'navin_...'` at pytest collection	system `python` instead of the unit virtualenv	run again with `<unit>/.venv/bin/python -m pytest` from the unit folder
pytest `rootdir` points at another backend than the one under test	two units in one pytest command	one command per unit
`opa is required`, `No module named 'yara_x'`, `shellcheck is required` in `hub/pipeline/tests`	external tools missing from the `PATH` (or `/tmp/toolchains` emptied by a reboot)	`OPA_BIN=<opa path>`, `PYTHONPATH` to a venv holding `yara-x==1.22.1`, `pip install shellcheck-py==0.11.0.1`

Next steps

Getting Started: the first agent, end to end.
Migration Guide: upgrade the SDK and the contracts.
Policy Development Guide: write and test the policies.

Functional​

Security​

Conformance and performance​

Procedures​

Running the tests of one repository unit (contributors)​

Test tools​

Troubleshooting​

Next steps​