End to end acceptance: from the machine to a verified restore

This journey shows that the client does everything in one console, without switching tools: add a machine, validate it, bootstrap it over SSH, enrol it, create a cluster, install a service on it, choose its number of instances, enable high availability, configure a backup and test a restore.

The automated proof is the test server/tests/test_e2e_infrastructure_journey.py. It starts a real server process (console API and agent API), two real unprivileged OpenSSH sshd (tests/infra_sshd.py) on free ports and two static Go agents built from agent-go, each with its own state directory. Every step goes through the public /api/v1 routes the console uses and asserts the observed state, not only the HTTP status. The real OPA binary and the real etcd archive come from NM_E2E_ARTIFACTS (by default the sandbox mirror, filled by sandbox.sh bundle opa,etcd) and are checked against the upstream sha256 of the manifest. When a tool or an artefact is missing (go, sshd, ssh, OPA, etcd), the test fails with a clear message. It is never skipped.

Steps in the console

Step	Screen	Route	What the step proves
1. Create the cluster	Clusters	POST /api/v1/clusters	The cluster appears in the list
2. Preflight of both machines	Hosts, SSH bootstrap wizard	POST /api/v1/hosts/preflight	SSH connection with the verified host key, DNS, disks; report with a reproducible digest
3. Bootstrap plan	SSH bootstrap wizard	POST /api/v1/hosts/bootstrap/plan	StrictHostKeyChecking=yes, exact sudo allowlist
4. Bootstrap and enrolment	SSH bootstrap wizard	POST /api/v1/clusters/{id}/bootstrap-tokens	The signed agent package is copied over SSH to each machine, the temporary access is removed, each single use token enrols one host and does not work a second time
5. Agent heartbeats	Node agents	GET /api/v1/agents/nodes/{id}, PATCH /api/v1/hosts/{id}	Both Go agents heartbeat, both hosts are trusted in the cluster, zones zone-a and zone-b
6. Install a service and choose 2 instances	Services, install wizard	POST /api/v1/clusters/{id}/services/validate, POST /api/v1/clusters/{id}/services	Catalogue OPA in replicated mode with 2 instances: the operation succeeds, host_components holds 2 started instances with a PID, one per host, and /health answers 200 on each
7. High availability validated	Availability	POST /api/v1/ha-groups, POST /api/v1/ha-groups/{id}/test-failover, GET /api/v1/ha-groups/{id}/operations	The group covers the 2 instances (members bound to their host) and is HA_CONFIGURED. The failover test really stops the instance of host 1 through the agent stop operation, runs the catalogue functional smoke check on the instance of host 2, starts the stopped instance again, then records the result: HA_VALIDATED only after that run, with the operations, the observed stopped state and the measured interruption
8. Backup and restore of a service	Cluster backups	POST /api/v1/clusters/{id}/backups, POST /api/v1/clusters/{id}/backups/{bid}/restore	Catalogue etcd: a known key is written, backed up, deleted, restored, then read back by the test with etcdctl and compared with the original; restore_status is passed
9. Backup and recovery centre	Backup centre	POST /api/v1/backup-plans, /backups, /verify, POST /api/v1/restores/plan, /approve, /execute	Encrypted artefact, sha256, verification passed; the requester can neither approve nor execute alone; two distinct approvers; verified execution

A second test (test_unbound_failover_request_is_not_a_validation) checks that a group not bound to an installed service keeps a requested test pending without validation, that an unknown host is refused (host_not_found) and that two members on the same host are refused (ha_members_same_host).

Measured interruption of the failover test

The interruption is the time between the end of the stop of the instance and the end of the first successful functional check on the other instance. Values observed over three runs of the test: 0.804 s, 0.922 s and 0.127 s. It depends on the pace of the engine and of the agents (one second polling), not on a traffic switch: OPA is stateless and both instances answer in parallel.

Live run on the sandbox

The sandbox was restarted with today's code (sandbox.sh down, then up), then the script /tmp/nm_live_journey.py drove the journey through the console API, logging in with password and TOTP like the console. Browser check not requested by the user.

Step	Result on the sandbox
Login with password and TOTP	Passed
Cluster	sandbox, mode sandbox
Enrolled host and heartbeats	One host gadhgadhi.localdomain, state healthy, last heartbeat received
Single use enrolment token	Created by the console
Installed services	etcd, nats, node-exporter, opa and openfga started
OPA functional check	Operation succeeded
etcd backup and restore with data read back	Key written (OK), backup succeeded, key deleted then absent, restore succeeded, value read back equal to the original, restore_status passed
2 instances, HA group and failover test	Not run: the sandbox has one host and one agent. The group with two members on that host is refused with ha_members_same_host (422). This journey is proven by the automated test with two machines
Preflight and SSH bootstrap	Not run: bootstrapping the local machine would install the agent in /usr/local/bin and /etc/systemd. Proven by the automated test

Before the ha_members_same_host fix, the first live run created on the sandbox a group opa-live-1793391747 with both members on the same host, wrongly shown HA_CONFIGURED. That defect is the one fixed; the group stays in the sandbox database (there is no delete route) and goes away with sandbox.sh down --purge.

Limits on one machine

• Both hosts run on the same machine: host 1 answers on 127.0.0.1 with a random FQDN, host 2 on a random 127.x.y.z address that is also its FQDN, because etcd advertises the FQDN of its member and a random name would not resolve. The fault domains zone-a and zone-b are logical.
• The agents run in process mode under a file prefix in /tmp, without systemd or sudo.
• The unprivileged sshd only accepts the account that runs it: the bootstrap is proven with use_sudo=false.
• etcd is restored as a single member (its restore script refuses a cluster of several members).
• The target of the backup centre is a local encrypted file, declared as SeaweedFS; its restore execution checks and decrypts the artefact. The data restore of a service is proven by step 8.

Run the acceptance

cd navin-manager/server
.venv/bin/python -m pytest -q tests/test_e2e_infrastructure_journey.py