Lifecycle of the services

This page covers what the Platform Manager does around an installation: installation profiles, preflight, liveness readiness and smoke checks, upgrades and rollbacks, backups and restores, drift detection, the support bundle, the signed acceptance report and the component sheet. The routes are listed in navin-manager/CONTRACT.md sections 7.8 to 7.12.

Installation profiles

The profile is chosen in the cluster wizard (Clusters, New cluster) and can be changed on the Profile tab of the cluster. Every rule is evaluated against the real state of the manager and of the cluster, never declared.

Profile	Rules
Development	Valid catalog. DNS problems only warn.
Standard (enterprise)	TLS on the manager, agent client certificates, TLS on every service that supports it, MFA required for the administrator roles.
Banking secure	Standard plus TLS on every endpoint, MFA enrolled by every local administrator, mirror only, remote audit (syslog), no sandbox service, approval of a second person for every disruptive operation.
Air gapped	TLS everywhere, mirror only, no bound connection or webhook outside the private networks.
Hybrid	Standard plus TLS on every connection to the client infrastructure.

A profile whose rules do not all hold is shown blocked with the failing rules and their reason. While it is blocked, installations, upgrades, commands, starts and restarts are refused (profile_blocked); stopping, uninstalling, configuring, rolling back and backing up stay possible so that the cluster can be brought back in line.

Two person rule (banking secure): stop, restart, reinstall, uninstall, upgrade, rollback, restore and configuration apply need an approval. Request it on the Approvals tab (action, service, reason); another administrator approves or rejects it; the requester then runs the operation with that approval, once, within 24 hours.

The profiles never replace the security of the client infrastructure (BYO): the manager only reads connections, never runs a destructive command on them, and the egress and TLS rules report what is bound.

Preflight

The preflight runs at the start of every installation and upgrade, and on demand (Profile tab, Run the preflight):

• host facts: operating system, architecture, memory, disks and systemd block; the other host checks warn;
• DNS forward (the FQDN resolves to the registered address) and reverse (the address resolves back to the FQDN); a failure blocks under every profile except development;
• catalog preflight entries of the services (ports free, kernel settings, tools);
• capacity: memory and CPU minimums of every managed service placed on each host;
• versions: agent version of each host, artifacts of the installed release present in the mirror;
• TLS: validity of the manager CA, real TLS handshake with the manager agent endpoint and with every TLS endpoint the services are bound to (expired, untrusted or refused fails, less than 14 days warns);
• the rules of the cluster profile.

One critical failure fails the operation before any file is written.

Liveness, readiness and smoke tests

A port that answers does not prove that the service works. The Checks block of the component sheet shows three distinct results:

• liveness: the health checks of every instance, refreshed by the agents;
• readiness: the service check of the catalog;
• smoke test: a real operation of the tool with a fresh nonce, run by the agent task smoke.run: an OPA policy loaded then a decision that must allow and one that must deny, an OpenFGA store, model, tuple and two checks, an etcd put then get then delete, a NATS subscribe then publish then delivery, a node exporter scrape of the CPU, memory and file system collectors. Cleanup steps run even after a failure.

Every installation and configuration plan ends with the service check then the smoke test; the Run the smoke test button of the sheet runs it alone.

Upgrades and rollbacks

The upgrade block of a descriptor declares the earlier releases an upgrade is supported from (from), whether the service is stateful, the migration hooks (each may be declared irreversible) and whether a rollback is possible (with its reason when it is not). The mirror keeps several releases of an artifact (manifest keys <service>:<name>@<version>), which also allows an installation at an earlier release.

The Upgrade tab shows the compatibility (from, to, problems), then runs: preflight, backup of a stateful service, the canary host first (new release, stop, migrations, files and units, start, health, smoke test), the other hosts one at a time, the service check, the smoke test and the new version. Before anything runs, the tasks that bring the previous release and its files back are sealed with the upgrade.

Rollback: previous release, files and units, and previous configuration version (stored as a new version restored from the old one), then health, service check and smoke test. When an irreversible migration ran or the catalog declares the rollback impossible, the rollback is refused with the reason and the restore is offered: previous release plus the data of the backup taken before the upgrade. Without such a backup the impossibility is stated.

Tested on the sandbox host: OPA 1.21.3 to 1.21.4 (stateless) and etcd 3.5.20 to 3.5.21 (stateful, backup before, data kept), each followed by a rollback.

Backups and restores

The backup block of a stateful service declares the steps, the files produced, the restore steps and an optional verification. Declared today: etcd (consistent snapshot, single member restore with etcdutl) and NATS (JetStream store archived while the server is stopped for a few seconds).

Backups tab of the cluster: Back up now runs a coordinated backup of the selected services (all by default) in dependency order, under one set identifier; the sha256 of every file is recorded. Restore checks the digests again (a modified file stops the restore before anything is stopped), stops the instances, runs the restore steps, starts them, then runs the health checks, the verification, the service check and the smoke test. A backup is restored only into the release it was taken with; to go back across an upgrade use the restore of the upgrade.

Limits stated by the scripts: the etcd restore handles a single member cluster and the binary method; a multi member cluster is restored member by member with a new cluster token.

Drift detection

Every file and unit written by the manager is recorded with its sha256. A drift scan (Drift tab, Scan now, and every hour) asks each agent for the digests of the managed paths: a file modified, removed or replaced outside the manager opens a finding and the config_drift alert. A finding is resolved when the file is put back by hand (restored), rewritten by the manager (Reconcile on the finding, or any configuration apply) or removed with the service.

Support bundle

Support bundles tab, Create a support bundle: tar.gz archive with the cluster and profile, the host facts, the services and versions, the rendered configuration files, the log tails of the instances, the operations, the open alerts and the drift findings. Every secret known to the manager, credential pairs, tokens, private keys, e-mail addresses, the names of the registered users, phone and card numbers and the IP addresses of the logs are masked; operators become stable pseudonyms. MANIFEST.json lists the files and the number of replacements by kind.

Signed acceptance report

Acceptance tab, Run the acceptance: preflight, liveness of every instance, readiness and smoke test of every managed service, then a report with one line per check: PASS (ran and met its expectation), FAIL (ran and did not) or BLOCKED (could not run, for example no smoke test declared), each with its evidence. The installation lines include the offline source: every artifact of the installation came from the internal mirror. The verdict is FAIL when one line failed, BLOCKED when one could not run, else PASS.

The JSON report is signed with the Ed25519 key of the manager and readable as HTML in English and in French. Verify a downloaded report offline:

navin-manager verify-report --file report.json --public-key manager.pem

(exit 0 when the signature is valid and made with that key). The public key is shown on the Acceptance tab.

Component sheet

Every service has the same sheet with twelve tabs, filled from the observed state and never from the requested action: Overview (state, health, versions, checks, open drift), Instances, Config, Dependencies, Logs, Metrics, Security (TLS, secrets, ports, certificates, profile), Operations, Upgrade, Backups, Audit and Support.