Skip to main content

ADR-0166: High availability declared per service in the Navin Manager catalog, mandatory ClamAV and YARA-X file scanning before any ingestion

Context​

Banking customers ask for high availability or cluster deployments that really keep their promises. Adding instances is not enough: every service has its own mechanism (Raft of etcd, OpenBao and Qdrant, ClickHouse Keeper, Alertmanager gossip, Sentinel or Redis Cluster, MySQL Group Replication, a load balancer in front of stateless instances, an agent per host), its dependencies must be highly available themselves, and some tools (garak, promptfoo, Ragas) have no native cluster at all. The catalog described modes and cardinalities without saying what each mode guarantees and under which conditions.

In addition, the files uploaded by users and connectors (PDF, Word, Excel, PowerPoint, archives, images, audio) reached Docling, OCR, media storage and transcription without any antivirus scan. A bank isolated from the Internet requires a local scan before any ingestion, a block when in doubt and full traceability.

Decision​

  1. A mandatory high_availability block in every catalog descriptor (navin-manager/catalog/schema/service.schema.json): full link to the official documentation, nature of the cluster, of high availability, failover, replication and quorum, discovery, entry point, peer security, prerequisites (dependencies that must be HA), one topology per mode (level standalone, ha or cluster, distribution active-active, active-passive, per host or parallel jobs, quorum component, fault domains), cluster status command and written caveats. The loader refuses a block inconsistent with the modes, components, bindings, parameters and commands of the service; liveness and readiness are derived from the health checks already declared. GET /api/v1/catalog/high-availability publishes the matrix; the Manager console shows the HA capability and state of each service (Services page, "High availability (HA)" row action); the generated service pages and navin-manager/docs/{fr,en}/high-availability.md document the matrix and the target architecture.
  2. HA install checks (navin_manager/services/ha_checks.py) on install and mode change: distinct hosts, fault domains (label zone, else rack), odd quorum, HA prerequisites, HAProxy and Keepalived load balancer or the bank load balancer, peer TLS, readiness checks. A failure refuses the operation (409 ha_requirements_not_met), a warning needs an explicit acknowledgement (ha_acknowledged). The service dashboard gives the HA state (healthy, degraded, quorum lost).
  3. Two security services in the catalog: ClamAV 1.4.6 LTS (GPL-2.0-only) and YARA-X 1.22.1 (BSD-3-Clause), installable standalone or replicated. clamd never listens on the network: an authenticated HTTPS gateway only forwards INSTREAM, PING and VERSION. freshclam only reads the internal mirror of the bank; YARA-X rules come from the internal repository, checked by sha256, compiled in a staging area then switched atomically, the previous version staying active on failure.
  4. Mandatory platform pipeline: Upload, Quarantine, ClamAV + YARA-X, Validation, Docling, RAG / AI. Every upload starts in quarantine; only a clean verdict of both engines yields a scan clearance bound to the sha256 of the bytes, structurally required by Docling, OCR, media storage and transcription. Malware, suspicious file, uninspectable file (encrypted, limits exceeded), tenant rule violated, engine unavailable or timeout: blocked. The enforce mode is imposed in production by the startup guard; the locked rules (scan before ingestion, both engines, fail closed) cannot be changed by a tenant.
  5. Logging and alerts: one record per scan, one audit event, one security event and one SOC detection per block, eai_file_scan_* metrics and Prometheus alert rules shipped by the Manager; File security screen of the portal; /v1/file-security routes in the four SDKs.

Alternatives considered​

  • KicomAV and PRX-SD (MIT, Apache-2.0): operational maturity and detection efficiency not demonstrated for a banking production.
  • Assemblyline 4 and Strelka: file analysis platforms, not standalone antivirus engines; the documented Assemblyline stack depends on Elasticsearch, outside the Apache licence. They remain possible as a complement.
  • Exposing clamd over TCP: its protocol has neither TLS nor authentication and accepts SHUTDOWN.
  • Linking libclamav into the Control Plane: mixes licences and the failure domain; talking over a socket and HTTPS keeps ClamAV in a separate process.
  • Calling the scan by convention before Docling: a new route would forget it; the clearance bound to the sha256 makes forgetting impossible.
  • Scanner failover through a load balancer only: the platform client fails over by itself across every instance, a load balancer address is still accepted.

Consequences​

A simultaneous outage of every instance of an engine blocks ingestion (503 file_scan_unavailable): this is intended. Every ClamAV host needs about 3 GB of memory for the full database. Signature freshness depends on the internal mirror of the bank; their age is monitored and can block depending on the policy. The antivirus does not protect against prompt injection: the existing screening of the extracted text (injection markers, hidden text, RAG quarantine) still applies after Docling.

Evidence​

navin-manager/server/tests/test_catalog_high_availability.py, navin-manager/server/tests/test_ha_install_checks.py, navin-manager/server/tests/test_catalog_scanners.py, services/control-plane/tests/test_file_security.py, services/control-plane/tests/test_file_security_api.py, portal/tests/file-security.test.ts, the conformance tests of the four SDKs.