Skip to main content

G-08 Alerting

Brick G of the Enterprise AI Platform. Technology: Alertmanager (routing, inhibition, silences) + FinOps alerts from /v1/finops/alerts. French version: ../../fr/bricks/G-08-alerting.md.

Role and responsibilities​

  • Route alerts (SLO burn, provider outage, GPU saturation, budget thresholds, audit chain failure) to on-call and tenant owners.

Position and interactions​

  • Execution path: event.
  • Critical security alerts create incidents through POST /v1/incidents.

Open source and developed parts​

  • Open source reused: Alertmanager, PrometheusRule.
  • Developed in the platform: Alert catalog with runbook links, tenant routing.

Data model​

Control Plane tables (schema eai): eai.quotas.

Every tenant table has tenant_id uuid NOT NULL and a row level security policy on current_setting('app.tenant_id', true)::uuid. ClickHouse usage_events is the analytics replica of eai.usage_events (ReplacingMergeTree keyed by event_id).

Contract: contracts/v1/usage-event.schema.json (schema_version 1.0). Usage is always reported with contracts/v1/usage-event.schema.json, audit with contracts/v1/audit-event.schema.json.

API endpoints​

MethodPath
GET/v1/finops/alerts
POST/v1/incidents

GET /v1/finops/alerts​

Response:

{
  "items": [
    {
      "quota_id": "quo_4",
      "threshold": 0.8,
      "triggered_at": "2026-10-27T08:00:00Z"
    }
  ]
}

POST /v1/incidents​

Request:

{
  "title": "Prompt injection burst on agt_support",
  "severity": "high",
  "category": "security",
  "description": "Wazuh rule 100210: prompt injection burst on agt_support.",
  "resources": [
    {
      "type": "agent",
      "id": "agt_support"
    }
  ]
}

Response:

{
  "id": "inc_8",
  "status": "open"
}

SLO budget burn alerts​

EaiSloAvailabilityFastBurn, EaiSloAvailabilitySlowBurn, EaiSloLatencyFastBurn and EaiSloWorkerFastBurn (file deploy/observability/slo/eai-slo-rules.yaml, tested with promtool test rules) follow the multi window method. See docs/operations/en/slo.md.

SDK examples​

Python​

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_live_xxx", tenant="acme", locale="en")
client.finops.alerts()
client.incidents.create(title="Prompt injection burst", severity="high")

TypeScript​

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY, tenant: "acme", locale: "en" });
await client.finops.alerts.list();
await client.incidents.create({ title: "Prompt injection burst", severity: "high" });

Java​

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .tenant("acme")
    .locale(Locale.ENGLISH)
    .build();
client.finops().alerts();
client.incidents().create(Map.of("title", "Prompt injection burst", "severity", "high"));

Go​

client, err := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")), eai.WithTenant("acme"), eai.WithLocale("en"))
if err != nil { log.Fatal(err) }
ctx := context.Background()
_, err = client.FinOps.Alerts(ctx)
_, err = client.Incidents.Create(ctx, map[string]any{"title": "Prompt injection burst", "severity": "high"})

Configuration​

Environment variables: EAI_SIEM_WEBHOOK_URL, EAI_SIEM_WEBHOOK_TOKEN.

Helm values keys (deploy/helm/enterprise-ai/values.yaml): global.monitoring.alertmanager.

Security​

  • Webhook receivers signed (HMAC).
  • Tenant resolved from the credential, never from the body; PostgreSQL row level security on every tenant table.
  • Every mutation and policy decision writes a hash chained audit event (G-11).
  • Errors use the standard envelope with message_i18n en and fr.

Observability​

  • Alert delivery failures and time to acknowledge.
  • W3C traceparent propagated, X-Request-Id returned, spans carry eai.tenant_id, eai.project, eai.agent_id, eai.run_id.

FinOps units​

  • Budget and quota alerts at 50%, 80% and 100% of the limit (GET /v1/finops/alerts?kind=budget|quota, severity info, warning, critical).

Failure modes and resilience​

  • Alertmanager cluster of 3, dead man's switch alert.

Standalone adoption​

Rules can be shipped to the client's own Alertmanager.

Acceptance tests​

  1. Crossing 80% of a quota produces one alert in /v1/finops/alerts and one notification.

Implementation roadmap