Operations

Clusters and hosts

• Create a cluster in Clusters, New cluster: a name and a mode (production, preprod, test, development or sandbox).
• Add hosts with a bootstrap token or the SSH bootstrap (see Offline installation). A token belongs to one cluster, expires and has a maximum number of uses; revoke it once the hosts are registered.
• Host page: facts, preflight results (run them again with Run checks), metrics (CPU, memory, load, disks, network), components and their actions.
• Machine parameters: the page lists the kernel parameters (sysctl) and process limits the services placed on the host need (for example vm.max_map_count for OpenSearch), with the current and the recommended values. Apply writes /etc/sysctl.d/90-navin-*.conf and /etc/security/limits.d/90-navin-*.conf on the host.
• Maintenance: a host in maintenance raises no alert; use it during hardware work.
• Decommission: remove its components first, then Decommission: the agent forgets its credentials and the certificates of the host are revoked.

Services

• Install: the wizard of Clusters, Install services chooses services, placements per mode, bindings and configuration. Validation refuses a placement that breaks a rule of the mode (for example an even number of OpenSearch manager nodes) and a production install of a service reserved for sandboxes.
• Start, stop, restart: on the whole service, a component, or one host component.
• Rolling restart: hosts are restarted by batches (batch_size), with a health check after each batch; the operation stops at the first failed host (tolerate_failures, 0 by default). The console offers the batch size among the hosts that run the service and never asks for the tolerated failures; the API keeps the field for automation.
• Approvals (banking-secure): a disruptive action without approval_id uses the approval granted to the caller for the same action and service (approved, unexpired, unused); nobody types an approval id.
• Service check: the end to end check of the catalog (for example an OPA decision or a Redis PING).
• Commands: actions of the catalog such as cluster_create for Redis, snapshot for etcd, setup_schema for Temporal, bundle_status for OPA. Commands that change data ask for a confirmation.
• Logs: the Logs button fetches the last lines of the journal (journalctl) or of the log files of a component on a host.
• Add or remove hosts of a component: the configuration of every instance is rendered again (peer lists change) and a rolling restart applies it.
• Uninstall: refused while another service uses the service; purge_data also removes the data directories.
• Client configuration: the files a consumer needs to reach the service (for example the EAI_* variables of the Navin Control Plane).
• Cluster actions (Cluster actions button of the Services page, POST /api/v1/clusters/{id}/actions): start, stop, restart or rolling restart every service, restart only the instances whose configuration changed (restart_stale, after their files and units are written again) and check every service. One operation follows the dependency order: a provider starts before its consumers and stops after them; a restart stops everything in reverse order, then starts everything in order with the health checks. The plan is shown before the run, with the services and instances left out and the reason: external connection, installation or removal in progress, administrative state, instance or host in maintenance, decommissioned host.
• Host actions (menu of the host in the Hosts and services view and on the host page, POST /api/v1/hosts/{id}/actions): start, stop, restart every component of the host, or only those whose configuration changed, as one operation in dependency order. Under the banking-secure profile, the stop and the restarts need the approval of a second person for the service * (cluster) or host:<id> (host).

Configurations

• The Configuration tab of a service groups the parameters by section, with their description, default, recommended value and unit; secrets show as set or not set.
• Save creates a version with a note. Versions compares two versions (secrets show only as changed or not) and Restore creates a new version with the content of an older one.
• Config groups override parameters for some hosts (for example more memory on larger hosts). A host belongs to at most one group per service.
• Preview shows the files rendered for a host and a component, secrets masked.
• Apply writes the changed files and restarts the components (rolling, all or none). Hosts that still run an older configuration are marked Restart required.

Administrative states and operating modes

A service carries an administrative state (enabled, disabled, maintenance, quarantined, suspended) and a run authorisation security_state (authorized or revoked). POST /api/v1/clusters/{cluster_id}/services/ {service}/mode applies a mode; each mode has its own effect, returned in effect:

Mode	New requests	Transactions in flight	Processes	State left
drain	refused	completed, up to drain_timeout_s (300 s by default)	stopped	maintenance
scale_to_zero	refused until the next start	completed (agent stop timeout)	stopped	enabled
security_suspend	refused	aborted at once (SIGKILL)	stopped	suspended, revoked
emergency_stop	refused	aborted at once (SIGKILL)	stopped	disabled
disable	refused after a stop	unchanged	unchanged	disabled
quarantine	unchanged	continue	frozen (only a stop is allowed)	quarantined
maintenance	unchanged	continue	unchanged	maintenance
enable	lifts the state, starts nothing			enabled

The agent receives the stop timeout in the service.control task (stop_timeout_s): SIGTERM then SIGKILL after that delay, SIGKILL at once for 0. The planner refuses (409 service_admin_state) what a state forbids: a disabled service refuses whatever would run it, a quarantined or suspended service refuses everything except a stop and an uninstall. The security suspension and lifting it (with a reason) need pki:manage. Drain and scale-to-zero follow the two person rule of the profile like a stop.

Security lock: POST /api/v1/clusters/{cluster_id}/security-lock {reason} (pki:manage) refuses every new operation of the cluster (423 cluster_security_locked) except a security suspension or an emergency stop, until DELETE .../security-lock. Every mode, lock and refusal is audited.

Operations

Every action is an operation. The Operations page lists them by cluster; an operation shows its stages, the tasks of each stage per host, their status, duration and live logs. Abort stops the queued tasks (running tasks finish). Retry creates a new operation from the failed stage. A task without result after its timeout fails as timed_out.

Blueprints

Clusters, Blueprint exports the services of a cluster (modes, methods, placements by host group, non secret configuration). Importing a blueprint in another cluster with a host mapping reproduces the installation; secrets are generated again or asked for.

Upgrades

1. Build and load the offline bundle of the new version (the mirror can hold several versions).
2. Upgrade the server package and run navin-manager migrate, then restart navin-manager.
3. Upgrade the agent packages (dnf upgrade or apt install): the agents keep their identity.
4. Upgrade the services one at a time from their page: Reinstall with the version of the new catalog, then a rolling restart. The alert agent_version_mismatch lists the agents left on an older version.

Service dependencies at install time

Before every install, the Manager resolves what each service needs (its requires in the chosen mode), in this order:

1. the binding chosen in the wizard;
2. a service of the same install;
3. a service already installed on the cluster: its connection parameters (address, port, database, user) are read back, its secrets stay sealed;
4. the only declared connection of that type;
5. otherwise the provider joins the install (default mode, or the first HA mode the hosts allow when the service asks for it), placed on the hosts of the service that needs it, then installed and configured before it; its own dependencies are resolved the same way.

The wizard shows the result in the Bindings and Install steps: status of each dependency, address, parameters, names of the sealed keys, then the numbered list of every stage of the operation. During the install, each service starts with its "dependencies" stage: it checks the providers again and writes one log line per binding, without secret; the operation page shows this report and the message of any failed task.

The install is refused when an installed provider is stopped, failed or still installing (dependency_unavailable), when several connections of the same type fit (dependency_choice_required), or when the provider is reserved to sandboxes while the cluster is production or pre-production (dependency_connection_required: bind an existing connection of the same type instead, for example the PostgreSQL cluster of the bank). The API keeps the validation rule only with auto_dependencies: false.

Backups

What	How
Manager database	Backups of the PostgreSQL of the bank (it holds clusters, configurations, operations, audit).
Master key	/etc/navin-manager/master.key, copied once in the bank vault. Without it the stored secrets are lost.
Server settings	/etc/navin-manager/ (server.yaml, server.env, certificates).
Mirror	Can be rebuilt from the bundles; keep the bundles of the versions in service.
Service data	With the tools of each service (for example the snapshot command of etcd, pg_basebackup for PostgreSQL).

PostgreSQL

The catalog installs PostgreSQL 16 with pgvector as a single server (mode standalone, methods podman, package and python) or as a highly available cluster of 2 to 7 members, 3 recommended (mode ha, methods package and python): Patroni keeps the leader key in the etcd service of the cluster, the other members are streaming replicas and the most advanced healthy replica is promoted when the leader is lost; the old leader rejoins as a replica (pg_rewind, or a new clone). Install etcd in mode cluster first: Patroni demotes the leader when etcd is unreachable.

• Consumers: the binding lists every member; clients use the libpq list with target_session_attrs=read-write and always reach the leader. For a client that only takes one address, put HAProxy in front of the members with an HTTP check GET /primary on the Patroni REST API port (8008): only the leader answers 200.
• TLS (tls_enabled, methods podman and package): SQL, replication and the Patroni REST API use the certificate of the Manager CA, connections without TLS are rejected (hostnossl reject), the members check each other with verify-full. Turn it on when etcd uses TLS: etcd then asks Patroni for a client certificate. The bindings give sslmode=require because the consumers do not receive the CA; a client given the CA can use verify-full.
• pgvector (pgvector, on by default): provision_consumer creates the vector extension in template1 and in the database of each consumer; enable_extension adds vector, pg_trgm or another contrib extension to one database.
• Commands of the mode ha: cluster_status (patronictl list), switchover to a named replica, reinit_member of a replica, push_settings (writes max_connections, WAL, ttl, loop_wait, retry_timeout and synchronous_mode into etcd, then restarts the members one by one, the leader last).
• Backup: pg_basebackup on the leader (tar, WAL streamed, SHA256 manifest), checked with pg_verifybackup. A restore stops every member, replaces the data directory of the member of the backup and resets the cluster state in etcd; that member starts as the leader and every other member is cloned again from it when it starts.