Skip to main content

Evaluation runs

The Evaluation runs screen lets you run an evaluation suite against its target or another one, and read the result of each metric, the cost and the status of every run. Portal path: /quality/runs. Developers and quality owners run suites by hand; agents and the promotion workflow run them automatically. The screen is drawn by portal/src/screens/quality.tsx.

What the screen shows​

  • Runs table, read with GET /v1/evaluations/runs (50 rows per call, newest first, cursor paging). Columns: Run id, Name (suite name), Target, Status, Results (one entry per metric: metric, value, passed, samples), Cost, Started.
  • Filters Suite (suite id) and Status (Passed, Failed, Error, Running, or all), then Filter.
  • When the list is empty: No evaluation run, with the hint Run a suite against an agent, model or prompt. and the button New evaluation run.
  • A click on a row, or View details, opens a side panel with GET /v1/evaluations/runs/{run_id}: the run contract, trigger, case_results and error.
  • New evaluation run opens a side panel with Suite (required, the suite id), Target (optional, type:ref@version; empty means the target of the suite) and Outputs (JSON list, default []).

Actions​

Button or actionWhat it doesAPI callRoles
Refresh (icon)Reloads the list.GET /v1/evaluations/runsAny tenant member
FilterApplies the Suite and Status filters.GET /v1/evaluations/runs with suite_id and statusAny tenant member
New evaluation runRuns the suite at once with the trigger manual, then stores the results and the cost.POST /v1/evaluations/runstenant_admin, developer, security_officer, ai_governance_officer, agent
Row click or View detailsOpens the run with the result of each case.GET /v1/evaluations/runs/{run_id}Any tenant member
Retry (error state)Sends the failed read again.Same callSame roles

How a run works (services/control-plane/src/eai/domains/evaluation/__init__.py): for each case, the output comes from Outputs (same order as the cases), from the output of the case, or from a call to the target through the platform. Each metric is scored and compared with its threshold. The status is passed when every metric passes, failed when one fails, and error when the target or the engine is unavailable or a metric has no sample. The cost is the sum of the usage recorded for the run.

Roles​

The portal hides no button. Starting a run needs one of the evaluation WRITE_ROLES (tenant_admin, developer, security_officer, ai_governance_officer) or agent; listing and reading need a tenant. platform_admin passes every role check.

Typical tasks​

Run a suite against a new version​

  1. Copy the suite id. The run form asks for the id, not the name. On Evaluation suites, open the suite and copy evaluation_id. Expected result: you have the id. If it fails: create the suite first.
  2. Start the run. A target given here overrides the target of the suite. Click New evaluation run, paste the Suite, type the Target with the new version (for example agent:invoice-assistant@1.3.0), then click Create. Expected result: the toast Created and a new row with its Status. If it fails: not_found (404) means the suite id is wrong.
  3. Read the results. Each metric says its value and whether it passed. Click the row and read results and case_results. Expected result: every metric has passed: true. If it fails: the status Error comes with error, for example metrics without samples: ....

Score outputs produced elsewhere​

  1. Paste the outputs. The suite can score answers you already have, without calling the target. Click New evaluation run, fill Suite and write Outputs as a JSON list of strings, one per case in order. Expected result: the run uses your outputs. If it fails: Invalid JSON under Outputs means the list is malformed.

Troubleshooting​

SymptomCauseFix
not_found (404) with eval_suiteThe suite id does not exist.Copy evaluation_id from the suite.
invalid_request (400) with required: suite_idSuite is empty.Fill the suite id.
Status Error with an error messageThe target or the engine is unavailable, or a metric got no sample.Check that the target is reachable, or provide Outputs.
Status FailedAt least one metric is below (or above, for lte) its threshold.Read case_results to find the failing cases.
forbidden (403) with required_rolesYour roles cannot run suites.Ask for one of the listed roles.