enterprise_ai.evaluation

Source: sdks/python/src/enterprise_ai/evaluation/__init__.py. Back to the Python SDK reference index.

note

This page is generated by tools/docs/gen_sdk_reference.py from the SDK sources and the OpenAPI text folders. Do not edit it by hand: change the source and regenerate.

Model CaseResult

Result of one case of a local suite run: scores per metric, verdict, latency and error.

Fields:

Name	Type	Required	Description
case_id	str	yes	Case reference, 1 to 200 characters.
scores	dict[str, float]	yes	Score between 0 and 1 per metric name; empty when the target failed.
passed	bool	yes	True when the target did not fail and every score reaches the min_score of the case (default 1).
latency_ms	float	yes	Time taken by the target, in milliseconds.
error	str | None	no	Exception raised by the target as "<Type>: <message>", None when it answered.

Model SuiteResult

Aggregated local run: per metric mean and gate verdict.

Fields:

Name	Type	Required	Description
suite	str	yes	Name of the suite (suite by default).
cases	list[CaseResult]	no	Cases of the suite: objects with input, optional system, expected values for the metrics and optionally a ready output.
threshold	float	no	Value the score is compared with.

Attributes:

Name	Type	Required	Description
metrics	dict[str, float]	yes	Mean score of each metric over all cases (a missing score counts 0).
pass_rate	float	yes	Share of passed cases, between 0 and 1 (0 without cases).
passed	bool	yes	Gate verdict: true when there is at least one case and pass_rate reaches threshold.

SuiteResult.to_dict

to_dict() -> dict[str, Any]

The result as a JSON ready dict.

This element takes no parameter.

Returns: dict[str, Any]. Dict {suite, pass_rate, passed, metrics, threshold, cases}.

Class EvaluationClient

/v1/evaluations and /v1/redteam.

Resource

EvaluationClient.check_gate

check_gate(target: dict[str, Any] | str, suite_ids: list[str] | None = None, **extra: Any) -> Any

Check the quality gate of a target. Checks that the latest finished run of every blocking suite of the target (for target.version when given, and limited to suite_ids when given) has passed. With run_missing: true, a suite without any run is run now; otherwise it counts as missing.

Route: /v1/evaluations/gates/check (POST).

Name	Type	Default	Required	Description
target	dict[str, Any] | str		yes	Target whose blocking suites are checked; version restricts to the runs of that version.
suite_ids	list[str] | None	None	no	Only these blocking suites. Null: every blocking suite of the target.
**extra	Any		no	Extra fields merged into the request body.

Returns: Any. Decoded JSON answer of the operation "Check the quality gate of a target".

Raises: the typed errors of the transport (Errors).

EvaluationClient.compare

compare(comparison: dict[str, Any]) -> Any

Compare a candidate with a baseline. Runs one suite twice, on the baseline and on the candidate (each {type, ref, version}, type model by default, agent or prompt), and stores both runs, the per metric deltas and the verdict. Deltas are oriented by the comparator of each metric (gte: higher is better, lte: lower is better).

Route: /v1/evaluations/comparisons (POST).

Name	Type	Default	Required	Description
comparison	dict[str, Any]		yes	Comparison to create: the suite and its baseline and candidate runs or targets.

Returns: Any. Decoded JSON answer of the operation "Compare a candidate with a baseline".

Raises: the typed errors of the transport (Errors).

EvaluationClient.create_suite

create_suite(suite: dict[str, Any]) -> Any

Create an evaluation suite. Creates a suite. Needs tenant_admin, developer, security_officer or ai_governance_officer. Errors 400 invalid_request: target type not in model, agent, prompt, rag_pipeline, skill or tool, unknown engine or kind, comparator other than gte or lte, or, for the custom engine, a metric other than exact_match, contains, not_contains, json_valid, faithfulness, latency or refusal. A name already used answers 409 conflict. The dataset gets its size and a SHA-256 digest of the cases. Writes the audit event evaluation.suite.create.

Route: /v1/evaluations/suites (POST).

Name	Type	Default	Required	Description
suite	dict[str, Any]		yes	Suite to create, as an Evaluation Contract.

Returns: Any. Decoded JSON answer of the operation "Create an evaluation suite".

Raises: the typed errors of the transport (Errors).

EvaluationClient.get_comparison

get_comparison(comparison_id: str) -> Any

Get an A/B comparison. Returns one comparison. Needs tenant_admin, developer, security_officer, auditor, viewer or finops. Answers 404 comparison_not_found.

Route: /v1/evaluations/comparisons/{comparison_id} (GET).

Name	Type	Default	Required	Description
comparison_id	str		yes	Identifier (UUID) of the comparison.

Returns: Any. Decoded JSON answer of the operation "Get an A/B comparison".

Raises: the typed errors of the transport (Errors).

EvaluationClient.get_redteam_run

get_redteam_run(run_id: str) -> Any

Get a red team run. Returns one red team run: target, engine, probes, status, summary (total, vulnerable, attack success rate, threshold, per category) and findings. Open to any member of the tenant. Answers 404 not_found.

Route: /v1/redteam/runs/{run_id} (GET).

Name	Type	Default	Required	Description
run_id	str		yes	Identifier (UUID) of the red team run.

Returns: Any. Decoded JSON answer of the operation "Get a red team run".

Raises: the typed errors of the transport (Errors).

EvaluationClient.get_run

get_run(run_id: str) -> Any

Get an evaluation run. Returns one evaluation run with its results per metric and per case. Open to any member of the tenant. Answers 404 not_found.

Route: /v1/evaluations/runs/{run_id} (GET).

Name	Type	Default	Required	Description
run_id	str		yes	Identifier (UUID) of the evaluation run.

Returns: Any. Decoded JSON answer of the operation "Get an evaluation run".

Raises: the typed errors of the transport (Errors).

EvaluationClient.get_suite

get_suite(suite_id: str) -> Any

Get an evaluation suite. Returns one suite with its full list of cases and run_on_promotion. Open to any member of the tenant. Answers 404 not_found.

Route: /v1/evaluations/suites/{suite_id} (GET).

Name	Type	Default	Required	Description
suite_id	str		yes	Identifier (UUID) of the suite.

Returns: Any. Decoded JSON answer of the operation "Get an evaluation suite".

Raises: the typed errors of the transport (Errors).

EvaluationClient.list_comparisons

list_comparisons(suite_id: str | None = None) -> Any

List the A/B comparisons. Lists the comparisons of the tenant, newest first, at most 200, optionally for one suite. Each gives baseline, candidate, both run ids, per metric deltas, tolerance and verdict. Needs tenant_admin, developer, security_officer, auditor, viewer or finops.

Route: /v1/evaluations/comparisons (GET).

Name	Type	Default	Required	Description
suite_id	str | None	None	no	Only the comparisons of this suite (UUID).

Returns: Any. Decoded JSON answer of the operation "List the A/B comparisons".

Raises: the typed errors of the transport (Errors).

EvaluationClient.list_redteam_runs

list_redteam_runs(**page: Any) -> Any

List the red team runs. Lists the red team runs of the tenant, newest identifier first, paginated, and probes: the builtin probes (id and category), the garak probe families and whether garak is installed. Open to any member of the tenant.

Route: /v1/redteam/runs (GET).

Name	Type	Default	Required	Description
**page	Any		no	Pagination options passed to the listing: limit (maximum number of items per page) and cursor (the next_cursor of the previous page). Iterating the returned page follows every cursor.

Returns: Any. Decoded JSON answer of the operation "List the red team runs".

Raises: the typed errors of the transport (Errors).

EvaluationClient.list_runs

list_runs(suite_id: str | None = None, **page: Any) -> Any

List the evaluation runs. Lists the evaluation runs of the tenant, newest identifier first, paginated, optionally filtered by suite and status. Each item follows the Evaluation Contract (target, engine, dataset, metrics, results, status, cost) with run_id, suite_id, trigger, case_results and error. Open to any member of the tenant.

Route: /v1/evaluations/runs (GET).

Name	Type	Default	Required	Description
suite_id	str | None	None	no	Only the runs of this suite.
**page	Any		no	Pagination options passed to the listing: limit (maximum number of items per page) and cursor (the next_cursor of the previous page). Iterating the returned page follows every cursor.

Returns: Any. Decoded JSON answer of the operation "List the evaluation runs".

Raises: the typed errors of the transport (Errors).

EvaluationClient.list_suites

list_suites(**page: Any) -> Any

List the evaluation suites. Lists the suites of the tenant ordered by name, paginated, as Evaluation Contract documents; cases gives the number of cases. Open to any member of the tenant.

Route: /v1/evaluations/suites (GET).

Name	Type	Default	Required	Description
**page	Any		no	Pagination options passed to the listing: limit (maximum number of items per page) and cursor (the next_cursor of the previous page). Iterating the returned page follows every cursor.

Returns: Any. Decoded JSON answer of the operation "List the evaluation suites".

Raises: the typed errors of the transport (Errors).

EvaluationClient.redteam

redteam(target: dict[str, Any] | str, probes: list[str] | None = None, **extra: Any) -> Any

Run a red team campaign. Sends adversarial probes to the target and stores the findings. Needs tenant_admin, developer, security_officer or ai_governance_officer.

Route: /v1/redteam/runs (POST).

Name	Type	Default	Required	Description
target	dict[str, Any] | str		yes	Target attacked: model, agent, prompt, rag_pipeline, skill or tool.
probes	list[str] | None	None	no	Probes to send. Default: every builtin probe, or promptinject and dan for garak.
**extra	Any		no	Extra fields merged into the request body.

Returns: Any. Decoded JSON answer of the operation "Run a red team campaign".

Raises: the typed errors of the transport (Errors).

EvaluationClient.run

run(suite_id: str, target: dict[str, Any] | str, **extra: Any) -> Any

Run an evaluation suite. Runs a suite now and returns the finished run (201). Needs tenant_admin, developer, security_officer or ai_governance_officer or agent.

Route: /v1/evaluations/runs (POST).

Name	Type	Default	Required	Description
suite_id	str		yes	Suite to run. Required unless trigger is promotion.
target	dict[str, Any] | str		yes	Target to evaluate instead of the suite target, for example another version. Required for trigger: promotion without suite.
**extra	Any		no	Extra fields merged into the request body.

Returns: Any. Decoded JSON answer of the operation "Run an evaluation suite".

Raises: the typed errors of the transport (Errors).

EvaluationClient.run_local

run_local(suite: dict[str, Any], target: Callable[[Any], Any]) -> SuiteResult

Run a suite locally with :func:run_suite.

Name	Type	Default	Required	Description
suite	dict[str, Any]		yes	Suite {name, threshold, metrics, cases}, as for run_suite.
target	Callable[[Any], Any]		yes	Callable given the input of each case, whose return value is scored.

Returns: SuiteResult. A SuiteResult: Aggregated local run: per metric mean and gate verdict.

Functions

exact_match

exact_match(output: Any, case: dict[str, Any]) -> float

Metric: 1 when the output equals expected once trimmed, else 0.

Name	Type	Default	Required	Description
output	Any		yes	Output of the target; anything but text is compared as sorted JSON.
case	dict[str, Any]		yes	Case of the suite, read for expected.

Returns: float. Score 1.0 or 0.0.

contains

contains(output: Any, case: dict[str, Any]) -> float

Metric: share of the contains needles (or of expected) found in the output, case insensitive.

Name	Type	Default	Required	Description
output	Any		yes	Output of the target; anything but text is compared as sorted JSON.
case	dict[str, Any]		yes	Case of the suite, read for contains (list) or expected.

Returns: float. Score between 0 and 1; 0 when the case gives no needle.

regex

regex(output: Any, case: dict[str, Any]) -> float

Metric: 1 when the pattern of the case matches somewhere in the output, else 0.

Name	Type	Default	Required	Description
output	Any		yes	Output of the target; anything but text is searched as sorted JSON.
case	dict[str, Any]		yes	Case of the suite, read for pattern (a Python regular expression).

Returns: float. Score 1.0 or 0.0.

json_valid

json_valid(output: Any, case: dict[str, Any]) -> float

Metric: 1 when the output is valid JSON text (or not text at all), else 0.

Name	Type	Default	Required	Description
output	Any		yes	Output of the target.
case	dict[str, Any]		yes	Case of the suite, unused by this metric.

Returns: float. Score 1.0 or 0.0.

json_schema

json_schema(output: Any, case: dict[str, Any]) -> float

Metric: 1 when the output (parsed when it is text) is valid against the JSON Schema 2020-12 of the case.

Name	Type	Default	Required	Description
output	Any		yes	Output of the target; text that is not JSON scores 0.
case	dict[str, Any]		yes	Case of the suite, read for schema.

Returns: float. Score 1.0 or 0.0.

max_length

max_length(output: Any, case: dict[str, Any]) -> float

Metric: 1 when the output is at most max_length characters long (default 10000), else 0.

Name	Type	Default	Required	Description
output	Any		yes	Output of the target; anything but text is measured as sorted JSON.
case	dict[str, Any]		yes	Case of the suite, read for max_length.

Returns: float. Score 1.0 or 0.0.

run_suite

run_suite(suite: dict[str, Any], target: Callable[[Any], Any]) -> SuiteResult

Run suite = {name, threshold, metrics, cases: [{id, input, expected, ...}]} locally.

Name	Type	Default	Required	Description
suite	dict[str, Any]		yes	Suite dict: name, threshold (default 1), metrics (default ["contains"]) and cases, each with id, input and the fields its metrics read.
target	Callable[[Any], Any]		yes	Callable given the input of each case; an exception fails that case only.

Returns: SuiteResult. A SuiteResult: Aggregated local run: per metric mean and gate verdict.