Skip to main content

Evaluation suites

The Evaluation suites screen lets you create the test sets used to evaluate an agent, a model, a prompt or another target, with their metrics, their thresholds and whether they block a promotion. Portal path: /quality/suites. Developers, security officers and AI governance officers write the suites; everyone in the tenant can read them. The screen is drawn by portal/src/screens/quality.tsx.

What the screen shows​

  • Suites table, read with GET /v1/evaluations/suites (50 rows per call, by name, cursor paging). Columns: Name, Target type, Target, Kind, Engine, Cases (number of cases), Blocking gate.
  • When the list is empty: No evaluation suite, with the hint Create a suite with cases and metrics. and the button New suite.
  • A click on a row, or View details, opens a side panel with GET /v1/evaluations/suites/{suite_id}: the suite contract, its full list of cases and run_on_promotion.
  • New suite opens a side panel with the fields:
    • Name (required, unique in the tenant).
    • Target (required, type:ref@version, example agent:invoice-assistant@1.2.0; without a type the target is an agent). Types: model, agent, prompt, rag_pipeline, skill, tool.
    • Dataset (required, a reference such as inline://support-v1).
    • Metrics (required, comma separated, default exact_match).
    • Thresholds (JSON, default {"exact_match": 0.8}): one threshold per metric.
    • Cases (JSON list, default one case with input and expected).
    • Blocking (default on): a blocking suite counts in the quality gate.

Actions​

Button or actionWhat it doesAPI callRoles
Refresh (icon)Reloads the list.GET /v1/evaluations/suitesAny tenant member
New suiteCreates the suite with the engine custom and the kind quality.POST /v1/evaluations/suitestenant_admin, developer, security_officer, ai_governance_officer
Row click or View detailsOpens the suite with its cases.GET /v1/evaluations/suites/{suite_id}Any tenant member
Retry (error state)Sends the failed read again.Same callSame roles

The form builds this body (portal/src/lib/mappers.ts): each metric takes its threshold from Thresholds, the comparator defaults to gte (higher is better):

{
  "name": "support-faq",
  "target": {"type": "agent", "ref": "invoice-assistant", "version": "1.2.0"},
  "dataset": {"ref": "inline://support-v1"},
  "metrics": [{"name": "exact_match", "threshold": 0.8}],
  "cases": [{"input": "What is the refund policy?", "expected": "30 days"}],
  "blocking": true
}

With the engine custom, the metrics allowed are exact_match, contains, not_contains, json_valid, faithfulness, latency and refusal (services/control-plane/src/eai/domains/evaluation/engines.py). A suite created here also runs on each promotion of its target (run_on_promotion true).

Roles​

The portal hides no button. In services/control-plane/src/eai/domains/evaluation/__init__.py, creating a suite needs one of WRITE_ROLES (tenant_admin, developer, security_officer, ai_governance_officer); listing and reading need a tenant. platform_admin passes every role check.

Typical tasks​

Create a regression suite for an agent​

  1. Write the cases. Each case is an input and the expected answer. Prepare a JSON list such as [{"input": "...", "expected": "..."}]. Expected result: a valid JSON list. If it fails: Invalid JSON shows under Cases.
  2. Create the suite. The metric and its threshold decide pass or fail. Click New suite, fill Name, Target (agent:<name>@<version>), Dataset, Metrics and Thresholds, paste the Cases, keep Blocking on, then click Create. Expected result: the toast Created and the suite in the table with its number of Cases. If it fails: invalid_request (400) names the metric or the target type that is not allowed; conflict (409) means the name is taken.
  3. Run it once. A first run gives the baseline. Open Evaluation runs and click New evaluation run with the suite id. Expected result: a run with the status Passed or Failed. If it fails: see the troubleshooting of that page.

Read the cases of a suite​

  1. Open the suite. The list only counts the cases. Click the row. Expected result: the side panel shows cases, metrics and gate.blocking. If it fails: not_found (404) means the suite does not exist in the tenant.

Troubleshooting​

SymptomCauseFix
validation_error (422) on CreateA metric of Metrics has no value in Thresholds, or a required field is empty.Give a threshold to every metric, for example {"exact_match": 0.8, "contains": 1}.
invalid_request (400) with metric and allowedThe metric is not one of the custom engine.Use a metric of the allowed list.
invalid_request (400) with target.typeThe target type is not supported.Use model, agent, prompt, rag_pipeline, skill or tool.
conflict (409)A suite with this name already exists.Choose another name.
forbidden (403) with required_rolesYour roles cannot create suites.Ask for developer, security_officer or ai_governance_officer.