Skip to main content

Node security: SSH bootstrap, enrollment, break glass and agent update

This guide covers the hosts the Platform Manager administers through its node agent. The contract is in CONTRACT.md section 7.15.

SSH bootstrap​

  1. Set the SSH policy of the cluster (PUT /api/v1/clusters/{id}/ssh-policy): bootstrap account (default navin-bootstrap), approved user certificate authorities, RSA allowed or not (never under 3072 bits), SPIFFE trust domain, rotation lead time.
  2. On the host, create the bootstrap account and install the sudoers drop-in shown by the console (Node agents, SSH bootstrap, Preview the plan) or by POST /api/v1/hosts/bootstrap/plan. It lists exact commands only (install of the agent binary and unit, systemctl daemon-reload, systemctl enable --now, removal for the rollback, nm-agent register --options-file); the manager validates it with visudo -cf.
  3. Record the host key (or the host certificate authority) in the request: the manager never learns a host key on the fly and always runs StrictHostKeyChecking=yes. An unknown or changed host key stops the run (ssh_host_key_refused). A bastion chain uses the same inventory keys for every hop.
  4. Run the preflight (POST /api/v1/hosts/preflight): SSH, operating system, DNS forward and reverse, clock offset, free disk, required ports and reachability, each pass, warning or blocking. The digest of the report is stable for the same observations.
  5. Start the bootstrap (POST /api/v1/hosts/bootstrap). The credential is an Ed25519 key or an OpenSSH user certificate of an approved CA; passwords and root are refused. The run verifies the signature of the agent binary first, adds a temporary key to authorized_keys, installs the binary and the hardened unit (NoNewPrivileges, ProtectSystem=strict, ProtectHome, PrivateTmp, explicit CapabilityBoundingSet, ReadWritePaths limited to the state directory), registers the agent, then removes the temporary key and the staging directory. A second run changes nothing already in place; a failure rolls back the files it created.

The private key and the bootstrap token stay in the memory of the manager for the run only: PostgreSQL holds a reference (key_ref: memory), the logs are redacted by the secret scanner.

Limitation: the hardened unit only lets the agent write its state directory. Hosts where the agent must write the Navin directories (/etc/navin, /opt/navin, /var/lib/navin) need a reviewed drop-in that adds them to ReadWritePaths.

Enrollment​

POST /api/v1/hosts/enroll returns a token nme_... valid at most 15 minutes, for one FQDN and one agent key (fingerprint: sha256 of the DER SubjectPublicKeyInfo). The agent presents it once to POST /agent/v1/enroll with a CSR of that key; reuse, expiry, another host or another key are refused and audited. The answer carries the mTLS certificate (with the SPIFFE ID in the SAN when a trust domain is set), the manager CA, the command key and the package key. The heartbeat answer says cert_renew_due when the certificate enters the rotation window; the agent renews through POST /agent/v1/renew, the previous certificate is revoked and listed in GET /agent/v1/crl.pem.

Quarantine (POST /api/v1/hosts/{id}/quarantine) blocks tasks, commands and renewals; revocation (/revoke) also revokes every certificate and refuses every agent call; /recover lifts a quarantine. A revoked host comes back only through a new enrollment.

Break glass​

  1. A user with hosts:manage requests access for a host with a reason, an SSH public key, the account and a TTL (60 s to 4 h).
  2. Two other users approve. The requester cannot approve, nobody approves twice.
  3. The second approval issues an OpenSSH user certificate signed by the break glass CA of the manager, valid until the end of the TTL. Configure the hosts with TrustedUserCAKeys (CA from GET /agent/v1/breakglass/ca.pub) and RevokedKeys (KRL from GET /agent/v1/breakglass/revoked-keys).
  4. Record the session evidence (POST /api/v1/breakglass/{id}/evidence: start, command digest, end). The job breakglass.expire marks the request expired at the end of the TTL; /revoke ends it earlier.

Agent update​

Publish the new binary as mirror/agent/nm-agent with nm-agent.sig (signed with the agent_packages key), then POST /api/v1/hosts/{id}/agent-update. The agent receives a signed agent.update command, verifies the digest and the signature, runs nm-agent version on the new binary, keeps the previous one as nm-agent.prev, swaps and restarts. The first heartbeat confirms the update; without a heartbeat before the deadline the previous binary is restored (by the watchdog of the new agent, or by nm-agent.prev update-check in ExecStartPre). The test selfupdate_binary_test.go compiles the real nm-agent, swaps it, and checks that the previous real binary restores itself without a heartbeat while keeping the state directory, or confirms after a heartbeat.