A/B comparisons
The A/B comparisons screen lets you run one evaluation suite on a baseline and on a candidate (two models, two model versions, two agents or two prompts) and read the per metric deltas and the verdict. Portal path: /quality/comparisons. Developers, security officers and tenant administrators compare before switching a route or a version; auditors and FinOps read the results. The screen is drawn by portal/src/screens/kit-data.tsx.
What the screen shows
- Comparisons table, read with
GET /v1/evaluations/comparisons(200 most recent, in one call). Columns: Id, Suite, Baseline, Candidate, Verdict, Created. - Filter Suite (suite id), then Filter.
- When the list is empty: No comparison, with the hint Compare a candidate with a baseline on a suite., and the button New comparison for the roles allowed to create.
- A click on a row, or View details, opens a side panel with
GET /v1/evaluations/comparisons/{comparison_id}: baseline, candidate, the two run ids, the metric rows (metric,baseline,candidate,delta,comparator,threshold), the tolerance and the verdict. - New comparison opens a side panel with Suite (required), Baseline (JSON, required, default
{"type": "model", "ref": "", "version": ""}), Candidate (JSON, required, same default), Tolerance (number from 0 to 1) and Outputs (JSON, default{"baseline": [], "candidate": []}).
The verdicts (services/control-plane/src/eai/domains/evaluation/ab.py):
| Verdict | Meaning |
|---|---|
regression | At least one metric is worse than the baseline by more than the tolerance. |
candidate_better | No metric is worse, and at least one is better by more than the tolerance. |
equivalent | Every delta stays within the tolerance. |
inconclusive | One of the two runs ended in error, or a metric has no score on one side. |
Actions
| Button or action | What it does | API call | Roles |
|---|---|---|---|
| Refresh (icon) | Reloads the list. | GET /v1/evaluations/comparisons | tenant_admin, developer, security_officer, auditor, viewer, finops |
| Filter | Keeps the comparisons of one suite. | GET /v1/evaluations/comparisons with suite_id | Same as Refresh |
| New comparison (hidden for other roles) | Runs the suite on the baseline, then on the candidate, computes the deltas oriented by each comparator and stores the verdict. | POST /v1/evaluations/comparisons | tenant_admin, developer, security_officer |
| Row click or View details | Opens the comparison. | GET /v1/evaluations/comparisons/{comparison_id} | Same as Refresh |
| Retry (error state) | Sends the failed read again. | Same call | Same roles |
Example of the fields Baseline and Candidate for two versions of one model:
{"type": "model", "ref": "llama-3-1-70b-local", "version": "2026-01"}
The type is model, agent or prompt; the baseline and the candidate must differ. With Outputs, each side is scored on the answers you give (one per case, in order) instead of calling the target.
Roles
This is the only screen of this group where the portal itself hides a button: New comparison shows only for ROLES.comparisonWriters of portal/src/lib/kit.ts (tenant_admin, developer, security_officer). The server checks the same WRITE_ROLES on creation and the READ_ROLES (tenant_admin, developer, security_officer, auditor, viewer, finops) on the list and the detail (services/control-plane/src/eai/domains/evaluation/ab.py). platform_admin passes every role check.
Typical tasks
Compare a new model version with the current one
- Pick the suite. The comparison reuses the cases and the metrics of a suite. Copy the
evaluation_idof a suite from Evaluation suites. Expected result: you have the id. If it fails: create a suite first. - Describe both sides. The baseline is what runs today. Click New comparison, paste Suite, write Baseline and Candidate in JSON with
type,refandversion, set a Tolerance such as 0.02, then click Create. Expected result: the toast Created and a row with its Verdict. If it fails:comparison_invalid(422) names the field at fault (baseline,candidate,tolerance,outputs). - Read the deltas. The verdict summarizes, the deltas explain. Click the row and read
metrics. Expected result: no row with a negative gain larger than the tolerance. If it fails: aregressionverdict means the candidate must not replace the baseline yet.
Review past comparisons of a suite
- Filter by suite. Each suite keeps its history. Type the suite id in Suite and click Filter. Expected result: the comparisons of the suite, newest first. If it fails:
comparison_not_found(404) means the id is not a valid suite id.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
comparison_invalid (422): "The evaluation comparison is invalid: ..." | A side has no ref, a type is not model, agent or prompt, both sides are equal, the tolerance is outside 0 to 1, or Outputs is not an object. | Fix the field named after the colon. |
not_found (404) on Create | The suite id does not exist. | Copy evaluation_id from the suite. |
Verdict inconclusive | One run ended in error, or a metric got no score. | Open the two runs on Evaluation runs and read their error. |
| New comparison missing | Your roles are not tenant_admin, developer or security_officer. | Ask for one of these roles. |
forbidden (403) on the list | Your roles are not among the readers. | Ask for viewer or auditor. |