Skip to main content

G-13 Quality orchestrator

Brick G of the Enterprise AI Platform. Technology: Developed component orchestrating G-09, G-10, policy checks and approvals into blocking gates. French version: ../../fr/bricks/G-13-quality-orchestrator.md.

Role and responsibilities​

  • Decide if an agent, prompt, model or skill version can be promoted to an environment.
  • Collect evidence ids and write a gate decision in the audit chain.

Position and interactions​

  • Execution path: workflow (promotion pipeline).
  • Called by CI (H-15), by the CLI (enterprise-ai test, publish) and by manifest apply for prod.

Open source and developed parts​

  • Open source reused: None: orchestration code on top of Ragas, Promptfoo, garak.
  • Developed in the platform: Whole component: gate definitions per environment, freshness rules, waiver through approvals.

Data model​

Control Plane tables (schema eai): eai.eval_suites, eai.eval_runs, eai.redteam_runs, eai.audit_events.

Every tenant table has tenant_id uuid NOT NULL and a row level security policy on current_setting('app.tenant_id', true)::uuid. ClickHouse usage_events is the analytics replica of eai.usage_events (ReplacingMergeTree keyed by event_id).

Contract: contracts/v1/evaluation-contract.schema.json (schema_version 1.0). Usage is always reported with contracts/v1/usage-event.schema.json, audit with contracts/v1/audit-event.schema.json.

API endpoints​

MethodPath
POST/v1/evaluations/gates/check
POST/v1/evaluations/runs
POST/v1/redteam/runs

POST /v1/evaluations/gates/check​

Request:

{
  "target": {
    "type": "agent",
    "ref": "agt_ap",
    "version": "1.4.0"
  },
  "suite_ids": [
    "evs_12"
  ],
  "run_missing": true
}

Response:

{
  "passed": false,
  "reasons": [
    "redteam run missing for version 1.4.0"
  ],
  "evidence": [
    "evr_77"
  ]
}

POST /v1/evaluations/runs​

Request:

{
  "suite_id": "evs_12",
  "target": {
    "type": "agent",
    "ref": "agt_ap",
    "version": "1.4.0"
  }
}

Response:

{
  "evaluation_id": "evr_77",
  "status": "running"
}

POST /v1/redteam/runs​

Request:

{
  "target": {
    "type": "agent",
    "ref": "agt_ap",
    "version": "1.4.0"
  },
  "engine": "garak",
  "probes": [
    "promptinject",
    "dan",
    "leakreplay"
  ],
  "max_attack_success_rate": 0.05
}

Response:

{
  "id": "rtr_5",
  "status": "running"
}

SDK examples​

Python​

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_live_xxx", tenant="acme", locale="en")
client.evaluation.check_gate(target={"type": "agent", "ref": "agt_ap", "version": "1.4.0"})
client.evaluation.run(suite_id="evs_12", target={"type": "agent", "ref": "agt_ap", "version": "1.4.0"})
client.evaluation.redteam(target={"type": "agent", "ref": "agt_ap"}, engine="garak", probes=["promptinject", "dan"])

TypeScript​

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY, tenant: "acme", locale: "en" });
await client.evaluations.checkGate({ target: { type: "agent", ref: "agt_ap", version: "1.4.0" }, suite_ids: ["evs_12"] });
await client.evaluations.createRun({ suite_id: "evs_12", target: { type: "agent", ref: "agt_ap", version: "1.4.0" } });
await client.redteam.createRun({ target: { type: "agent", ref: "agt_ap" }, engine: "garak", probes: ["promptinject", "dan"] });

Java​

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .tenant("acme")
    .locale(Locale.ENGLISH)
    .build();
client.evaluations().checkGate(Map.of("target", Map.of("type", "agent", "ref", "agt_ap", "version", "1.4.0"), "suite_ids", List.of("evs_12")));
client.evaluations().createRun(Map.of("suite_id", "evs_12", "target", Map.of("type", "agent", "ref", "agt_ap", "version", "1.4.0")));
client.redteam().createRun(Map.of("target", Map.of("type", "agent", "ref", "agt_ap"), "engine", "garak", "probes", List.of("promptinject", "dan")));

Go​

client, err := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")), eai.WithTenant("acme"), eai.WithLocale("en"))
if err != nil { log.Fatal(err) }
ctx := context.Background()
_, err = client.Evaluations.CheckGate(ctx, map[string]any{"target": map[string]any{"type": "agent", "ref": "agt_ap", "version": "1.4.0"}})
_, err = client.Evaluations.CreateRun(ctx, map[string]any{"suite_id": "evs_12", "target": map[string]any{"type": "agent", "ref": "agt_ap", "version": "1.4.0"}})
_, err = client.RedTeam.CreateRun(ctx, map[string]any{"target": map[string]any{"type": "agent", "ref": "agt_ap"}, "engine": "garak", "probes": []string{"promptinject", "dan"}})

Configuration​

Environment variables: EAI_PROMPTFOO_NPX.

Helm values keys (deploy/helm/enterprise-ai/values.yaml): controlPlane.config, controlPlane.secrets.

Security​

  • A waiver needs an approval by someone other than the requester.
  • Tenant resolved from the credential, never from the body; PostgreSQL row level security on every tenant table.
  • Every mutation and policy decision writes a hash chained audit event (G-11).
  • Errors use the standard envelope with message_i18n en and fr.

Observability​

  • Gate pass rate and blocking reasons.
  • W3C traceparent propagated, X-Request-Id returned, spans carry eai.tenant_id, eai.project, eai.agent_id, eai.run_id.

FinOps units​

  • Gate checks are cheap; costs come from the evaluation runs they trigger.

Failure modes and resilience​

  • Missing or stale evidence -> passed=false with reasons (fail closed).

Standalone adoption​

CI pipelines of a client can call only the gate API.

Acceptance tests​

  1. Promotion to prod without red team run returns passed=false.
  2. An approved waiver makes the gate pass and is visible in audit.

Implementation roadmap