enterprise_ai.packs.reports

Source: sdks/python/src/enterprise_ai/packs/reports.py. Back to the Python SDK reference index.

note

This page is generated by tools/docs/gen_sdk_reference.py from the SDK sources and the OpenAPI text folders. Do not edit it by hand: change the source and regenerate.

Functions

run_tests

run_tests(path: str | Path, digest: str) -> dict[str, Any]

Run the tests declared in spec.tests of the pack (pytest kinds unit, integration, authz, security; opa), then the alert rule and SDK conformance checks.

Name	Type	Default	Required	Description
path	str | Path		yes	Folder of the pack.
digest	str		yes	Digest of the pack archive the report is bound to.

Returns: dict[str, Any]. Dict {digest, results}: per kind the passed, failed, errors counts, plus observability and sdk_conformance.

run_promptfoo

run_promptfoo(root: Path, config_path: Path) -> dict[str, Any]

Run a promptfoo configuration offline: each exec: provider gets every prompt of every test, scored with deterministic assertions.

Name	Type	Default	Required	Description
root	Path		yes	Folder of the pack, working folder of the provider commands.
config_path	Path		yes	YAML promptfoo configuration (providers, prompts, tests); a provider other than exec: raises ValueError.

Returns: dict[str, Any]. Dict {cases, passed_cases, score, failures}, score between 0 and 1, at most 20 failures kept.

Raises: ValueError.

run_ragas

run_ragas(root: Path, dataset_path: Path) -> dict[str, Any]

Score a JSONL dataset offline with context recall: share of the ground_truth words found in the contexts returned by rag/retrieve.py.

Name	Type	Default	Required	Description
root	Path		yes	Folder of the pack holding rag/retrieve.py.
dataset_path	Path		yes	JSONL file, one {question, ground_truth, ...} object per line.

Returns: dict[str, Any]. Dict {samples, metric, score}, metric being context_recall and score the mean between 0 and 1.

run_evaluations

run_evaluations(path: str | Path, digest: str) -> dict[str, Any]

Run offline every evaluation declared in spec.evaluations of the pack (promptfoo or ragas) and compare its score with its threshold.

Name	Type	Default	Required	Description
path	str | Path		yes	Folder of the pack.
digest	str		yes	Digest of the pack archive the report is bound to.

Returns: dict[str, Any]. Dict {digest, results}: per evaluation name its score, threshold (default 1) and passed; another type raises ValueError.

Raises: ValueError.

scan

scan(path: str | Path, digest: str) -> dict[str, Any]

trivy fs of the pack folder (vulnerabilities and secrets), offline database, bound to the digest.

Name	Type	Default	Required	Description
path	str | Path		yes	Folder of the pack; its dist folder is skipped.
digest	str		yes	Digest of the pack archive the report is bound to.

Returns: dict[str, Any]. Dict {tool: trivy, digest, report}, report being the trivy JSON without CreatedAt; a trivy failure raises RuntimeError.