Monitoring and alerts

What navin-manager watches

Scope	Definition	Condition (default thresholds warning / critical)
Host	host_heartbeat_lost	No heartbeat for 60 / 120 seconds. Resolved as soon as the agent sends a heartbeat again.
Host	host_disk_usage	Fullest filesystem among the root and the Navin directories (/etc/navin, /opt/navin, /var/lib/navin, /var/log/navin) at 80 / 90 percent. Other mounts stay visible in the host metrics.
Host	host_memory_usage	Used memory at 85 / 95 percent.
Host	host_load	Load over 5 minutes divided by the CPU count at 1.5 / 3.
Host	host_preflight_failed	A preflight check of the host failed.
Host	agent_version_mismatch	The agent version differs from the server version.
Component	component_down	A component that must run is not running.
Component	component_health	The health checks of a running component fail.
Component	config_stale	The configuration changed and the component still runs the previous one.
Cluster	certificate_expiry	A certificate issued by the manager expires within 30 / 7 days.
Cluster	operation_failed	An operation of the cluster failed during the last 24 hours.
Manager	manager_engine_lag	Last pass of the operations engine 30 / 120 seconds ago.
Manager	manager_database	Round trip to the manager database of 200 / 1000 milliseconds.

Every definition can be disabled, and its thresholds and interval changed, in Alerts, Definitions. An alert keeps its history (state changes with the measured value), can be acknowledged with a note, and resolves by itself when the condition clears. Hosts in maintenance raise no alert.

Notifications

Alerts, Notification targets sends the alerts that reach the minimum severity of a target:

Kind	Details
Email	SMTP of Administration, Settings (STARTTLS and authentication supported).
Webhook	HTTPS POST with the header X-NM-Signature: t=<unix>,v1=<hex>, where v1 is the HMAC-SHA256 of <t>.<body> with the secret of the target. Reject a timestamp older than 5 minutes.
Syslog	RFC 5424 messages over UDP, TCP or TLS to the syslog of Administration, Settings (the SIEM of the bank).

Failed deliveries are retried, then marked failed after 5 attempts. Test sends a test notification.

Self monitoring of the manager

System shows the health of the manager: database latency, delay of the operations engine, queued tasks, agents online and lost, last run and last error of each periodic job. The same data is served by GET /api/v1/system/health (permission system:read).

GET /metrics serves Prometheus metrics. It requires Authorization: Bearer <NM_METRICS_TOKEN>; in production the token is mandatory.

Metric	Labels	Meaning
nm_build_info	version	Version of the manager.
nm_hosts	cluster, state	Hosts per cluster and state.
nm_host_components	state	Component instances per state.
nm_operations	status	Operations per status.
nm_tasks_queued		Tasks waiting for an agent.
nm_alerts	severity	Open alerts per severity.
nm_engine_lag_seconds		Seconds since the last pass of the operations engine.
nm_database_latency_ms		Round trip to the manager database.
nm_scheduler_job_errors	job	1 when the last run of a periodic job failed.
nm_agents_online		Agents that sent a heartbeat during the last minute.

The readiness of the process is GET /readyz (database reachable) and its liveness GET /healthz.

Monitoring of the managed platform

The catalog holds a complete observability stack that navin-manager installs like any other service: Prometheus, Alertmanager, OpenSearch, OpenSearch Dashboards, Jaeger, Opik, Alloy, the OpenTelemetry Collector and the exporters (node, Redis, PostgreSQL, blackbox). The platform ships no dashboard server (ADR-0169): machines and services are followed in the manager itself (service monitoring tab with its history in the database, host metrics, telemetry export to ClickHouse), logs in OpenSearch Dashboards, traces in Jaeger and Opik, metrics and alerts in Prometheus and Alertmanager, whose alert and recording rules stay. A Grafana already run by the client can still be connected to these sources; the platform never deploys it (AGPL licence, ADR-0101).

Prometheus discovers the components to scrape from the manager: GET /api/v1/clusters/<cluster id>/sd/prometheus returns one target group per component instance that exposes metrics, with the labels nm_cluster, nm_service, nm_component and nm_host. The Prometheus service of the catalog configures this discovery with an API token stored in a file of mode 0640: create a token restricted to cluster:read in Profile, API tokens and give it to the parameter of the Prometheus service.

Alloy ships the journal of the navin-* units and the files of /var/log/navin as OTLP to the OpenTelemetry Collector, which writes them to OpenSearch, where OpenSearch Dashboards reads them.