Hardened installation for banking production
How the Manager installs services in banking production. Architecture decision: ADR-0171.
Principles
- The four methods stay:
binary,podman,pythonandpackage(split at decision time in ADR-0171). Services are not all containerized: each method is hardened in its own way. - Installation stays offline: everything comes from the mirror / Trusted Hub.
- systemd stays the single driver:
navin-<service>-<component>.service, whatever the method. - The customer's existing services (PostgreSQL, Redis, Keycloak) stay connected, never reinstalled.
1. Container network
Every Podman component declares its network policy in the catalog (run.podman.network):
| Policy | Effect | Condition |
|---|---|---|
isolated (default) | Bridge network of the service, navin-<cluster>-<service>, created on the host (podman network create --ignore, or a Quadlet .network file): two services never share a network, a flow between services always goes through a declared port | None |
host | Host network (--network host) | network_justification required: the technical need of this component, otherwise the catalog is refused at load time |
none | No network | None |
Ports of an isolated component (ports of the catalog, host port = container port):
| Declaration | Published | Reached by |
|---|---|---|
public: true | --publish p:p/proto, every address of the host | the hosts the firewall allows (firewall.open) |
public: false | --publish 127.0.0.1:p:p/proto | the agent and the processes of the host, no other host |
publish: false | never published | only the containers of the same service, by name |
list_param and field | one port per entry of the list (HAProxy frontends), by public | by public |
modes | published in those modes only (ringpop ports of the Temporal cluster mode) | by public |
Rules checked when the catalog loads:
--network,--net,--publish,-p,--exposeare forbidden inextra: the network only goes throughnetworkandports;- published ports stay filtered by the firewall: only
public: trueports are opened (firewall.open).
The catalog templates write their addresses with net of the render context, which changes nothing for the binary,
python and package methods nor for the host network:
| Helper | Isolated network | Other cases |
|---|---|---|
net.listen(address) | unspecified address (every interface of the container) | address |
net.local(address) | unspecified address, port published on 127.0.0.1 only | address (127.0.0.1 by default) |
net.peer(component, address) | container name navin-<service>-<component> on the service network | address (127.0.0.1 by default) |
net.check(address) | 127.0.0.1 (check by the agent of a non public port) | address |
The address advertised to peers and clients stays the one of the host (host.ip, host.fqdn): the other hosts reach
the published port.
Network policy per service
| Policy | Services | Reason |
|---|---|---|
isolated | Every other Podman component of the catalog: data stores (postgresql, mysql, redis, etcd, clickhouse, opensearch, qdrant, seaweedfs, nats), gateways (apisix, haproxy with one published port per frontend), observability (jaeger, otel-collector), keycloak, temporal (every role, in both modes), presidio, opik, navin-enterprise, ollama, clamav, docling, litellm, marquez, opa, openfga, opensearch-dashboards, promptfoo, temporal-ui, label-studio (application and interface), llama-factory (GPUs through CDI) and vllm outside the distributed mode | Listens on the unspecified address of the container, the host address advertised to peers, components of the same host reached by their container name |
host (kernel probe) | falco, tetragon | eBPF, --privileged, --pid=host: justified specific profile |
host (demonstrated need) | livekit (WebRTC media, official LiveKit recommendation), vllm in the distributed mode only (host_network_modes: Ray and NCCL on dynamic ports) | Justification of its own, checked when the catalog loads |
Migrating a service to isolated: write its listen addresses with net.listen or net.local, its access to another
component of the same host with net.peer (and declare same_host_as on the component: the installation then refuses a
placement where the other component is missing on the host), its checks of non public ports with net.check, advertise
host.ip or host.fqdn to peers, declare every port used, then set network: isolated and drop the justification.
tests/test_catalog_podman_policy.py checks that no component uses the host network without a justification.
The Security tab of the service sheet shows, for a service installed with Podman, the isolation of each container as the catalog declares it: network policy and justification, network of the service, published ports and their scope (on the host or on 127.0.0.1 only), privileges (no new privileges, capabilities dropped, read only file system, added capabilities) and quotas. The view is read only: the isolation changes in the catalog, where every weakening carries its written justification.
Flows between hosts
A public port that only the cluster uses is opened to the hosts of the cluster and to the Manager, never to every
source ("internal communications explicitly allowed"). The Manager derives the list from the flow matrix
(network/flows.py, page Network): a port is restricted when every flow that reaches it on its host comes from a
host, a component or the Manager of the cluster (transport, peers, replication, gossip, bindings between services,
Prometheus of the cluster), or when no flow reaches it yet and the matrix classifies it between the nodes of the
service (single node service). A port reached by clients or by an external source (endpoints of provides, https,
media, bundle, ports of a quick link, UDP) stays open to every source.
| Firewall | Restricted port | Client port |
|---|---|---|
| firewalld | permanent ipset navin-<12 hex of the cluster id> (hash:ip, family inet, plus <set>-v6 in inet6 for IPv6 members) and a rich rule source ipset=<set> port port=<p> protocol=<proto> accept per port | --add-port=<p>/<proto> |
| ufw | allow from <member> to any port <p> proto <proto> comment <set> per member and port | allow <p>/<proto> |
The members are the addresses of the hosts of the cluster that are not decommissioned and the address of the Manager
when NM_AGENT_URL or NM_PUBLIC_URL holds an IP (a Manager known by name only reaches no restricted port, which it
does not need). Each firewall.open carries every restricted port of the host: the agent removes the
--add-port or ufw allow <p>/<proto> rule of a restricted port, the rules of the set that no longer match a port
or a member, and adds nothing when the host already matches (running it twice changes nothing). The periodic job
network.firewall_members (every 60 s) starts an operation network.firewall_members when a host joins, leaves or
moves to another cluster, so that every host of the cluster gets the new set; a failed refresh is tried again after
15 minutes, a cluster under a security lock is left alone. Rootless agents and hosts without firewalld or ufw keep
their behavior (no task, or backend: none). The cloud security groups are not changed: the rules of the page Network
cover them.
2. Quadlet
catalog.render.quadlet(service, component, ctx) renders, from the same specification as podman run:
navin-<service>-<component>.container:Image=by digest,Pull=never,Network=,PublishPort=,Volume=,EnvironmentFile=,NoNewPrivileges=,DropCapability=,ReadOnly=,AddDevice=(CDI),SecurityLabelDisable=(GPU),PidsLimit=, quotas,LogDriver=journald;navin-<cluster>-<service>.networkfor isolated components (one network per service).
An option that renders empty (a CPU only option on a GPU host) is dropped: an empty Quadlet key would reset its whole
list (an empty PodmanArgs= would erase the security options and quotas already set). The entrypoint goes through
PodmanArgs=--entrypoint=: Quadlet has no Entrypoint= key before Podman 5. The file sets no Notify=true:
readiness comes from conmon, as with --sdnotify conmon of the podman run unit; Notify=true would wait for an
sd_notify signal the images never send.
The files go to /etc/containers/systemd/; the Quadlet generator produces the same unit name
navin-<service>-<component>.service: supervision, logs and service.control do not change.
Switch
The Podman components move to Quadlet by default, host by host (NM_PODMAN_QUADLET, on by default; false keeps
podman run everywhere):
- a systemd host whose Podman is 4.9 or later and whose agent announces the
quadlet.unittask in its facts (task_types, Python and Go agents) receivesquadlet.unitinstead ofsystemd.unit; it writes the.containerand the.networkof the service, disables then removes thepodman rununit it replaces (a unit of/etc/systemd/systemwould hide the generated one), then reloads systemd; - the other hosts (older Podman or not reported yet, agent that does not know the task) keep the
podman rununit, with the same isolation options: the Manager never plans a task the agent would refuse; - a generated unit cannot be enabled with
systemctl enable: the Manager starts it and the[Install]section of the file starts it at boot; - the Quadlet files are tracked by the drift detection like the systemd units; the uninstall removes the
.containerand the unit, the.networkof the service stays.
A host already installed moves to Quadlet at the next configuration apply of the service, once its agent is up to date; the container runs under its new unit at its next restart.
3. Security profile per service
run.podman field | Podman option | Default |
|---|---|---|
| (always, unless privileged) | --security-opt no-new-privileges | on |
cap_drop_all: true | --cap-drop ALL (then cap_add) | off, enabled once the image is validated |
cap_add | --cap-add | requires privileged_justification |
read_only: true | --read-only (tmpfs /tmp, /var/tmp, /run) | off |
security_opt | --security-opt (seccomp, AppArmor); label=disable requires a justification | empty |
user | --user | image |
devices | --device nvidia.com/gpu=<index> (CDI) | none |
extra: [--privileged, --pid=host, ...] | host namespaces | requires privileged_justification |
Health checks: every permanent component (kind: master or worker) declares its health; the catalog loader
refuses a permanent component without a health check, whatever its method. The Manager runs them after every start,
at every upgrade (canary host then each host) and continuously for the monitoring.
Quotas (component limits), applied to every method:
| Field | Podman | systemd (binary, python) |
|---|---|---|
cpus | --cpus | CPUQuota= |
memory | --memory | MemoryMax= |
pids | --pids-limit | TasksMax= |
Secrets stay off the command line: 0640 environment file rendered by the Manager (values from OpenBao); with
Quadlet, Secret= will replace that file.
MCP servers of the tenants
The stdio MCP servers of the tenant projects run on the hosts in the mcp-host service (ADR-0171; on Kubernetes the
MCP Runtime isolates them per user and project). Its parameter isolation decides how:
isolation | Each server runs as | Servers hosted |
|---|---|---|
process (default) | a child process of the host user (0700 folders, clean environment, rlimits) | validated servers of the company catalogue only |
container | a rootless Podman container: one network per tenant project, read-only root, /tmp tmpfs, --cap-drop=ALL, no-new-privileges, --userns=keep-id, memory and pids quotas, --pull=never | every server, untrusted ones included |
A server is trusted when its preset is listed in the Manager setting mcp_validated_presets
(NM_MCP_VALIDATED_PRESETS, the presets the company published in its MCP catalogue) with its command and arguments
unchanged. Any other server (another preset of the Navin library, a server written by a user) is refused on a
process mcp-host with 409 mcp_untrusted_needs_container, and the runtime refuses it as well. The container
isolation needs podman and subordinate uid and gid ranges for the service user (preflight podman_present and
subordinate_ids; the agent adds the ranges with usermod on user.ensure). The runtime images node and python
are pinned by digest in the catalogue, delivered by the offline mirror and loaded by the agent into the read-only
image store /opt/navin/mcp-host/images, listed by the containers-storage.conf of the service user.
4. Trusted Hub
Before installation, every artifact (binary, OCI image, wheel, package) must be: listed in the signed Hub manifest, verified by sha256, signed (Cosign, verifiable offline), shipped with an SBOM, scanned (CVE), compliant with the licence policy (ADR-0101) and approved for that version. The Hub is built separately; the Manager will refuse an artifact that does not meet these conditions.
With the hub.enforce setting on, the Manager checks every artifact against the last verified signed document
(Ed25519, pinned keys, expiry, no rollback) before any task runs; a refusal stops the plan with 409:
| Kind | Checked value | Refusal |
|---|---|---|
Image (image.pull, image.load) | manifest digest of the mirror entry | hub_package_not_approved |
Binary, archive (artifact.fetch) | sha256:<hex> of the file | hub_artifact_not_approved |
Wheel (python.venv) | sha256:<hex> of every wheel of the wheelhouse | hub_artifact_not_approved |
A digest is approved when it is the digest of a non revoked target or the digest of one of its files, and no
revocation names it. What the Manager expects from the Hub document for the kinds other than images (contract for
the Hub): a target {project, name, version, kind: bundle, ref, digest, service, image, files: [{name, digest: "sha256:<hex>"}]} whose files list the sha256 of each binary, archive and wheel the bundle holds (the digest
of an OCI bundle is its manifest digest, not the file digest); a revocation may repeat files so that the revoked
files stay refused. A malformed file digest makes the whole document invalid (hub_document_invalid). The cluster
profile banking-secure is blocked (rule hub_enforced) while hub.enforce is off or no valid document is stored.
5. Upgrades and rollback
| Method | Hardening |
|---|---|
binary | Immutable directory per version, current link switched; rollback = switch back; Cosign signature of the file |
python | Hub wheels listed with their sha256 in the mirror manifest (wheels), pinned versions, pip install --no-index --require-hashes --only-binary=:all: -r navin-requirements.txt, one venv per version {opt}/venvs/<version>; {opt}/venv is a link switched by link.ensure, so the {opt}/venv/bin/... paths of the catalog stay valid and a rollback switches the link back. The first switch over a real {opt}/venv directory renames it {opt}/venv.nm-legacy (never deleted) |
package | Signed Hub repositories, versions pinned in the catalog with name==version (dnf install name-version, apt-get install name=version); the signature checks of the OS repositories are never disabled |
podman | Signed image by digest, Quadlet, isolated network |
Lifecycle of an upgrade
Every step of the target cycle is carried by the Manager and proven by a test:
| Step | What the Manager does | Code and proof |
|---|---|---|
| 1. Prerequisites and rights | services:install permission, security lock, modes and profile of the cluster, approval under banking-secure, preflight checks on every host | navin-manager/server/src/navin_manager/services/planner.py, navin-manager/server/src/navin_manager/services/profiles.py; navin-manager/server/tests/test_lifecycle_profiles.py |
| 2. Signed artifact | Binary, archive, wheel or image: an approved, non revoked target of the signed Trusted Hub document when hub.enforce is on | navin-manager/server/src/navin_manager/hub.py, navin-manager/server/src/navin_manager/mirror.py; navin-manager/server/tests/test_hub_client.py |
| 3. Dependencies | Bindings the target release requires are bound, the provider is installed and available, the connection still exists; refused before any task | navin-manager/server/src/navin_manager/services/upgrades.py; test test_upgrade_checks_the_dependencies_of_the_target_release |
| 4. Backup of configuration and state | Backup of a stateful service before the upgrade (upgrade.backup_before); the previous release (artifacts, files, units, checks, configuration version) sealed before any task | navin-manager/server/src/navin_manager/services/backups.py; test test_stateful_upgrade_backup_migration_refused_rollback_and_restore |
| 5. Install of the requested release | Canary host first, then one host at a time: fetch, stop, migrations, files and units, start; one venv per release for Python | tests test_stateless_upgrade_with_canary_then_rollback, test_python_upgrade_installs_a_venv_per_release_and_rollback_switches_the_link_back |
| 6. Start and health | Health checks of every instance after its start, mandatory for every permanent component | navin-manager/server/src/navin_manager/catalog/loader.py; test test_a_permanent_component_without_health_check_is_refused |
| 7. Application validation | Smoke test on the canary host, service check, final smoke test | navin-manager/server/src/navin_manager/services/checks.py |
| 8. Confirm or roll back | Version recorded when everything passes; on a failure with auto_rollback, the Manager replays the previous release at once (system actor, audited); after an irreversible migration, the way back is the restore of the backup | test test_failed_upgrade_with_auto_rollback_brings_the_previous_release_back |
The tests of steps 3 to 8 are in navin-manager/server/tests/test_lifecycle_upgrades.py (step 6:
navin-manager/server/tests/test_catalog_podman_policy.py). When an upgrade migrates a database schema, the rollback
restores the backup before going back to the previous binary.
6. Privileged actions
Tasks signed by the Manager and authorized by role, audit log of every installation, least-privilege service
account on every host. Restricting control to navin-* units is a guard, not a barrier: the task allow list, the
agent path guard and task signatures remain essential.
High availability
systemd and Podman manage one host. Quorums, replicas and rolling upgrades across hosts stay orchestrated by the Manager (availability).