SLO and error budgets
The SLO and error budgets screen lets you check, for each platform service, how much of its availability and
latency error budget is left over the selected window, and lets a platform administrator change the objectives.
Portal path: /observability/slo (menu Observability > SLO and error budgets). Platform and tenant
administrators, support engineers, auditors and viewers use it to know whether a service spends its budget too fast.
What the screen shows
| Area | Content |
|---|---|
| Window (minutes) | 15, 60 (default), 360 or 1440. Sent as window_minutes. |
| SLO table | One row per service (control_plane_api, model_gateway, tool_gateway, realtime): Service, Status (burning when a burn rate is above 1, else healthy), Requests, Errors (5xx), Over latency target, Availability target, Availability budget left, Availability burn rate, Latency budget left, Latency burn rate. |
| Detail panel | A click on a row shows the row as JSON, with the Latency target (s) of the service. |
| Operator consoles (platform administrators only) | Links Open the console to Opik (AI traces), Grafana (when the client runs one, EAI_GRAFANA_URL; never deployed by the platform) and OpenSearch Dashboards (logs), or the reason a console is not configured. Machines and services are followed in the Navin Manager. |
| Empty state | No measurement: measurements appear once the API serves requests. |
A burn rate above 1 spends the budget faster than the objective allows. The measurements are kept in memory by each
API replica for EAI_SLO_RETENTION_MINUTES (1440 by default): the screen shows the replica that answered. A
platform administrator sees the whole platform; the other readers see their own tenant.
Actions
| Button or action | What it does | API call | Roles |
|---|---|---|---|
| Window (minutes) | Reloads the budgets for the new window. | GET /v1/telemetry/slo (window_minutes) | platform_admin, tenant_admin, auditor, viewer, support_engineer |
| Refresh (round arrow icon) | Reloads the budgets. | GET /v1/telemetry/slo | Same roles |
| Click on a row | Opens the detail panel. | None | Same roles |
| Edit the objectives (row menu) | Opens the form Availability objective (default 0.995), Latency target (s) (default 2), Latency objective (default 0.95), Reason, and saves the objectives of the service. | PUT /v1/telemetry/slo/targets/{service} | platform_admin |
| Open the console (Operator consoles) | Opens the console in a new tab. | GET /v1/platform-manager/console (links) | platform_admin |
Roles
| Role | Access |
|---|---|
platform_admin | Reads the budgets of the whole platform, edits the objectives and sees the Operator consoles. |
tenant_admin, auditor, viewer, support_engineer | Read the budgets of their tenant (ROLES of services/control-plane/src/eai/domains/observability/slo.py, Observability section of portal/src/lib/nav-access.ts). |
| Other roles | The Observability section is hidden; the page shows Your roles do not open this page. |
warning
An objective changed here applies at once on the replica that wrote it and within 30 seconds on the others (cache of the stored objectives); it is audited with the values before and after. Fill Reason so the change can be explained later.
Typical tasks
Check that the services hold their objectives
- Choose the window. A short window shows an incident in progress, a long one the trend. Choose
60or1440in Window (minutes). Expected result: one row per service with its budgets. If it fails:slo_window_invalidmeans the window is longer than the retention of the replicas. - Read the status. A
burningrow has a burn rate above 1. Expected result: every service ishealthy. If it fails: for aburningservice, open Live metrics and Traces to find the failing requests.
Change the latency objective of a service
- Read the current objective. Note the Availability target of the row and open the detail panel for the Latency target (s): the edit form opens with the default values, not the current ones. Expected result: you know the values to keep. If it fails: click Refresh.
- Edit. Open the row menu, choose Edit the objectives, fill Availability objective (0.5 to less than 1),
Latency target (s) (above 0, up to 300), Latency objective (0.5 to less than 1) and Reason, then click
Edit the objectives.
Expected result: the toast Done appears and the budgets are recomputed with the new objectives.
If it fails:
forbiddenmeans you are not a platform administrator;validation_errormeans a value is out of bounds.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
Error slo_window_invalid (HTTP 422): "The SLO window must be between 1 and ... minutes." | The window exceeds EAI_SLO_RETENTION_MINUTES. | Choose a shorter window, or raise the setting. |
Error slo_targets_invalid (HTTP 500) | The EAI_SLO_* settings of the platform are malformed. | Fix EAI_SLO_AVAILABILITY_TARGETS or EAI_SLO_LATENCY_TARGETS_SECONDS (one service=value per service). |
Error slo_service_unknown (HTTP 404) | The service is not one of the four platform services. | Edit one of the rows listed. |
Error tenant_required or slo_scope_forbidden (HTTP 403) | A tenant reader without a tenant, or reading another tenant. | Sign in to your tenant. |
| Numbers change from one refresh to the next | Each replica keeps its own measurements. | Use the Grafana dashboards for the fleet view. |
| Operator consoles missing | You are not a platform administrator. | Expected: the section is shown to platform_admin only. |