G-09 Evaluation

Brick G of the Enterprise AI Platform. Technology: Ragas (RAG metrics) + Promptfoo (prompt and agent regression, assertions). French version: ../../fr/bricks/G-09-evaluation.md.

Role and responsibilities

• Define suites (dataset + metrics + thresholds), run them against an agent, prompt, model or RAG version, store results.
• Metrics: faithfulness, answer relevancy, context precision/recall (Ragas), assertions, latency and cost budgets (Promptfoo).

Position and interactions

• Execution path: batch workflow.
• Runs are workflows (Temporal) calling the gateway like any agent, so they are traced and costed.
• Results feed G-13 quality gates and Opik feedback scores.

Open source and developed parts

• Open source reused: Ragas, Promptfoo.
• Developed in the platform: Evaluation Contract, engine adapters, dataset versioning, result normalization.

Data model

Control Plane tables (schema eai): eai.eval_suites, eai.eval_runs, eai.usage_events.

Every tenant table has tenant_id uuid NOT NULL and a row level security policy on current_setting('app.tenant_id', true)::uuid. ClickHouse usage_events is the analytics replica of eai.usage_events (ReplacingMergeTree keyed by event_id).

Contract: contracts/v1/evaluation-contract.schema.json (schema_version 1.0). Usage is always reported with contracts/v1/usage-event.schema.json, audit with contracts/v1/audit-event.schema.json.

API endpoints

Method	Path
POST	/v1/evaluations/suites
POST	/v1/evaluations/runs
GET	/v1/evaluations/runs/{id}
GET	/v1/evaluations/runs

POST /v1/evaluations/suites

Request:

{
  "name": "invoice-agent-regression",
  "target": {
    "type": "agent",
    "ref": "agt_ap"
  },
  "engine": "ragas",
  "kind": "rag",
  "dataset": {
    "ref": "eai-datasets/invoice-qa-v3.jsonl",
    "version": "3"
  },
  "metrics": [
    {
      "name": "faithfulness",
      "threshold": 0.85
    },
    {
      "name": "answer_relevancy",
      "threshold": 0.8
    }
  ]
}

Response:

{
  "id": "evs_12",
  "name": "invoice-agent-regression",
  "status": "active"
}

POST /v1/evaluations/runs

Request:

{
  "suite_id": "evs_12",
  "target": {
    "type": "agent",
    "ref": "agt_ap",
    "version": "1.4.0"
  }
}

Response:

{
  "evaluation_id": "evr_77",
  "status": "running"
}

GET /v1/evaluations/runs/{id}

Response:

{
  "evaluation_id": "evr_77",
  "status": "succeeded",
  "results": [
    {
      "metric": "faithfulness",
      "value": 0.91,
      "passed": true
    }
  ],
  "gate": {
    "passed": true
  },
  "cost": {
    "measured": {
      "amount": "3.12",
      "currency": "EUR"
    }
  }
}

GET /v1/evaluations/runs

Response:

{
  "items": [
    {
      "evaluation_id": "evr_77",
      "status": "succeeded"
    }
  ],
  "next_cursor": null
}

SDK examples

Python

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_live_xxx", tenant="acme", locale="en")
client.evaluation.create_suite({"name": "invoice-agent-regression", "engine": "ragas", "target": {"type": "agent", "ref": "agt_ap"}, "dataset": {"ref": "ds_invoices"}, "metrics": [{"name": "faithfulness", "threshold": 0.85}]})
client.evaluation.run("evs_12", {"type": "agent", "ref": "agt_ap", "version": "1.4.0"})
client.evaluation.get_run("evr_77")
client.evaluation.list_runs(suite_id="evs_12")

TypeScript

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY, tenant: "acme", locale: "en" });
await client.evaluations.createSuite({ name: "invoice-agent-regression", engine: "ragas", target: { type: "agent", ref: "agt_ap" }, dataset: { ref: "ds_invoices" }, metrics: [{ name: "faithfulness", threshold: 0.85 }] });
await client.evaluations.createRun({ suite_id: "evs_12", target: { type: "agent", ref: "agt_ap", version: "1.4.0" } });
await client.evaluations.getRun("evr_77");
await client.evaluations.listRuns({ suite_id: "evs_12" });

Java

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .tenant("acme")
    .locale(Locale.ENGLISH)
    .build();
client.evaluations().createSuite(Map.of("name", "invoice-agent-regression", "engine", "ragas", "target", Map.of("type", "agent", "ref", "agt_ap"), "dataset", Map.of("ref", "ds_invoices"), "metrics", List.of(Map.of("name", "faithfulness", "threshold", 0.85))));
client.evaluations().createRun(Map.of("suite_id", "evs_12", "target", Map.of("type", "agent", "ref", "agt_ap", "version", "1.4.0")));
client.evaluations().getRun("evr_77");
client.evaluations().listRuns(Map.of("suite_id", "evs_12"));

Go

client, err := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")), eai.WithTenant("acme"), eai.WithLocale("en"))
if err != nil { log.Fatal(err) }
ctx := context.Background()
_, err = client.Evaluations.CreateSuite(ctx, map[string]any{"name": "invoice-agent-regression", "engine": "ragas", "target": map[string]any{"type": "agent", "ref": "agt_ap"}, "dataset": map[string]any{"ref": "ds_invoices"}, "metrics": []map[string]any{{"name": "faithfulness", "threshold": 0.85}}})
_, err = client.Evaluations.CreateRun(ctx, map[string]any{"suite_id": "evs_12", "target": map[string]any{"type": "agent", "ref": "agt_ap", "version": "1.4.0"}})
_, err = client.Evaluations.GetRun(ctx, "evr_77")
_, err = client.Evaluations.ListRuns(ctx, eai.WithQuery(map[string]string{"suite_id": "evs_12"}))

Configuration

Environment variables: EAI_PROMPTFOO_NPX.

Helm values keys (deploy/helm/enterprise-ai/values.yaml): workers.items[name=evaluation].

Security

• Datasets are classified media; production data used for evaluation requires an approval.
• Tenant resolved from the credential, never from the body; PostgreSQL row level security on every tenant table.
• Every mutation and policy decision writes a hash chained audit event (G-11).
• Errors use the standard envelope with message_i18n en and fr.

Observability

• Metric trends per target version.
• W3C traceparent propagated, X-Request-Id returned, spans carry eai.tenant_id, eai.project, eai.agent_id, eai.run_id.

FinOps units

• Evaluation cost stored in the Evaluation Contract cost field and as Usage Events with labels.purpose=evaluation.

Failure modes and resilience

• Judge model unavailable -> run status failed, gate cannot pass (fail closed).

Standalone adoption

Teams can evaluate their own RAG with only the evaluation API.

Acceptance tests

1. A suite with a 0.85 faithfulness threshold fails on a deliberately broken retriever.
2. The run document validates against evaluation-contract.schema.json.

Implementation roadmap