High availability of the catalog services
Generated from the high_availability block of every catalog descriptor (navin-manager/catalog/services): do not edit by hand. The same data is served by GET /api/v1/catalog/high-availability; the installation checks it before a highly available mode is installed (see the platform availability).
Target HA architecture of Navin Enterprise
Users reach a pair of HAProxy load balancers sharing a Keepalived virtual IP (VRRP), which spread the traffic over replicated services backed by a highly available data layer. In the Manager, the HA state of each installed service (HA groups, quorum, failover tests and their evidence, catalog capability) is under Services, row action "High availability (HA)".
[ Users and applications ]
|
v
[ HAProxy pair + Keepalived VIP ] haproxy, keepalived
|
v
[ Replicated services ] apisix, litellm, keycloak, opik, openfga, temporal, navin-enterprise, clamav, yara-x, label-studio
|
v
[ HA data layer ] postgresql, redis, etcd, clickhouse, nats, qdrant, seaweedfs
| Layer | Services (Highest catalog level) |
|---|---|
| HAProxy pair + Keepalived VIP | HAProxy: ha keepalived: ha |
| Replicated services | Apache APISIX: ha LiteLLM gateway: ha Keycloak: cluster Opik: ha OpenFGA: ha Temporal: cluster Navin Enterprise: ha ClamAV: ha YARA-X: ha Label Studio: ha |
| HA data layer | PostgreSQL: ha Redis Open Source: cluster etcd: cluster ClickHouse: cluster NATS: cluster Qdrant: cluster SeaweedFS object storage: cluster |
Services with a native or distributed cluster
| Service | Prerequisites | Official documentation | High availability | Replicas / configuration |
|---|---|---|---|---|
| LiveKit server | redis: Highly available Redis holding the node registry and the room routing. | https://docs.livekit.io/transport/self-hosting/distributed/ | external (external); failover client side (client); quorum none (none) | single: standalone, single (livekit-server min 1, max 1)distributed: cluster, active-active (livekit-server min 2, recommended 3) |
| Voice agent workers | - | https://docs.livekit.io/agents/server/ https://github.com/livekit/agents | redundancy (redundancy); failover native (native); quorum none (none) | single: standalone, single (worker min 1, max 1)pool: ha, parallel jobs (worker min 2, recommended 2) |
| LiveKit Egress | redis: Highly available Redis shared with LiveKit Server. | https://docs.livekit.io/transport/self-hosting/egress/ https://github.com/livekit/egress | redundancy (redundancy); failover native (native); quorum none (none) | single: standalone, single (egress min 1, max 1)pool: ha, parallel jobs (egress min 2, recommended 2) |
| LiveKit SIP | redis: Highly available Redis shared with LiveKit Server. | https://docs.livekit.io/transport/self-hosting/sip-server/ https://github.com/livekit/sip | redundancy (redundancy); failover client side (client); quorum none (none) | single: standalone, single (livekit-sip min 1, max 1)redundant: ha, active-active (livekit-sip min 2, recommended 2) |
| ClickHouse | - | https://clickhouse.com/docs/guides/oss/deployment-and-scaling/keeper | native (native); failover client side (client); quorum ClickHouse Keeper (keeper) | standalone: standalone, single (server min 1, max 1)cluster: cluster, active-active (keeper min 3, max 5, odd number, recommended 3, server min 2, recommended 4) |
| etcd | - | https://etcd.io/docs/v3.5/op-guide/clustering/ | native (native); failover native (native); quorum Raft (raft) | standalone: standalone, single (etcd min 1, max 1)cluster: cluster, active-active (etcd min 3, max 7, odd number, recommended 3) |
| NATS | - | https://docs.nats.io/learn/clustering/ | native (native); failover client side (client); quorum Raft (raft) | standalone: standalone, single (server min 1, max 1)cluster: cluster, active-active (server min 3, odd number, recommended 3) |
| OpenSearch | - | https://docs.opensearch.org/latest/tuning-your-cluster/discovery-cluster-formation/settings/ | native (native); failover native (native); quorum Raft (raft) | single-node: standalone, single (node min 1, max 1)cluster: cluster, active-active (manager min 3, odd number, recommended 3, node min 2, recommended 3) |
| Qdrant | - | https://qdrant.tech/documentation/distributed_deployment/ | native (native); failover native (native); quorum Raft (raft) | standalone: standalone, single (node min 1, max 1)distributed: cluster, active-active (node min 3, odd number, recommended 3) |
| Redis Open Source | - | https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/ https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/ | native (native); failover native (native); quorum Sentinel (sentinel) | standalone: standalone, single (redis min 1, max 1)replication: ha, active-passive (redis min 2, recommended 3)sentinel: ha, active-passive (redis min 2, recommended 3, sentinel min 3, odd number, recommended 3)cluster: cluster, active-active (redis min 6, multiple of 2, recommended 6)cluster_ha: cluster, active-active (redis min 9, multiple of 3, recommended 9) |
| Temporal | db: Highly available PostgreSQL (persistence).visibility: Highly available PostgreSQL (visibility). | https://docs.temporal.io/self-hosted-guide/deployment | native (native); failover native (native); quorum gossip (gossip) | standalone: standalone, single (schema min 1, max 1, frontend min 1, max 1)cluster: cluster, active-active (schema min 1, max 1, frontend min 2, recommended 2, history min 2, recommended 3, matching min 2, recommended 2, worker min 1, recommended 2) |
| Keycloak | database: Highly available PostgreSQL holding realms, users and offline sessions. | https://www.keycloak.org/high-availability/introduction | native (native); failover load balancer (load-balancer); quorum none (none) | standalone: standalone, single (keycloak-server min 1, max 1)cluster: cluster, active-active (keycloak-server min 2, recommended 3) |
| OpenBao | - | https://openbao.org/docs/configuration/storage/raft/ | native (native); failover native (native); quorum Raft (raft) | standalone: standalone, single (openbao-server min 1, max 1)ha: cluster, active-passive (openbao-server min 3, max 5, odd number, recommended 3) |
| Kamailio SIP edge | asterisk: A pool of two Asterisk nodes or more, so the dispatcher has a next active node for every tenant. | https://www.kamailio.org/docs/modules/5.7.x/modules/dmq.html https://www.kamailio.org/docs/modules/5.7.x/modules/dispatcher.html https://www.kamailio.org/docs/modules/5.7.x/modules/htable.html https://www.kamailio.org/docs/modules/5.7.x/modules/dialog.html | native (native); failover client side (client); quorum none (none) | single: standalone, single (kamailio min 1, max 1)ha: ha, active-active (kamailio min 2, recommended 2) |
| keepalived | - | https://keepalived.readthedocs.io/en/latest/ | native (native); failover virtual IP (vip); quorum VRRP (vrrp) | pair: ha, active-passive (keepalived min 2, max 2) |
| Alertmanager | - | https://prometheus.io/docs/alerting/latest/high_availability/ | native (native); failover client side (client); quorum gossip (gossip) | single: standalone, single (alertmanager min 1, max 1)cluster: cluster, active-active (alertmanager min 3, odd number, recommended 3) |
| SeaweedFS object storage | - | https://github.com/seaweedfs/seaweedfs/blob/master/note/slides/seaweedfs-production-setup.md https://github.com/seaweedfs/seaweedfs/wiki/Production-Setup | native (native); failover load balancer (load-balancer); quorum Raft (raft) | single: standalone, single (master min 1, max 1, volume min 1, max 1, filer min 1, max 1, s3 min 1, max 1, admin min 1, max 1)distributed: cluster, active-active (master min 3, odd number, recommended 3, volume min 3, recommended 3, filer min 2, recommended 2, s3 min 2, recommended 2, admin min 1, max 1) |
High availability by replication
| Service | Prerequisites | Official documentation | High availability | Replicas / configuration |
|---|---|---|---|---|
| Docling Serve | - | https://github.com/docling-project/docling-serve/blob/main/docs/deployment.md | external (external); failover load balancer (load-balancer); quorum none (none) | standalone: standalone, single (docling-serve min 1, max 1)replicated: ha, active-active (docling-serve min 2, recommended 2) |
| Ollama | litellm: Highly available LiteLLM (two proxies or more): it balances and fails over between the instances; agents and applications never call Ollama directly. | https://docs.ollama.com/faq https://docs.ollama.com/docker https://docs.litellm.ai/docs/routing | redundancy (redundancy); failover client side (client); quorum none (none) | single: standalone, single (server min 1, max 1, tls-front min 1)replicated: ha, active-active (server min 2, recommended 2, tls-front min 1) |
| vLLM | litellm: Highly available LiteLLM (two proxies or more): it balances, retries and fails over between the replicas; agents and applications never call vLLM directly. | https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html https://docs.vllm.ai/en/latest/deployment/k8s.html https://docs.litellm.ai/docs/routing | redundancy (redundancy); failover client side (client); quorum none (none) | single: standalone, single (server min 1, max 1)replicated: ha, active-active (server min 2, recommended 2)distributed: standalone, single (server min 1, max 1, ray-worker min 1, recommended 1) |
| Label Studio | database: Highly available PostgreSQL holding the projects, the annotations and the users.s3: Highly available S3 storage (SeaweedFS in distributed mode) shared by every application. | https://labelstud.io/guide/persistent_storage https://labelstud.io/guide/storedata https://labelstud.io/guide/install_k8s https://labelstud.io/guide/security | external (external); failover load balancer (load-balancer); quorum none (none) | standalone: standalone, single (app min 1, max 1, nginx min 1, max 1)replicated: ha, active-active (app min 2, recommended 2, nginx min 1, recommended 2) |
| Marquez | database: Highly available PostgreSQL holding the lineage metadata. | https://marquezproject.ai/docs/deployment/ | external (external); failover load balancer (load-balancer); quorum none (none) | standalone: standalone, single (api min 1, max 1, web min 1, max 1)replicated: ha, active-active (api min 2, recommended 2, web min 1, recommended 1) |
| MySQL Community Server | - | https://dev.mysql.com/doc/mysql-shell/8.4/en/deploying-production-innodb-cluster.html | redundancy (redundancy); failover client side (client); quorum none (none) | standalone: standalone, single (mysqld min 1, max 1)replication: ha, active-passive (mysqld min 2, recommended 3) |
| OpenSearch Dashboards | opensearch: OpenSearch with its security plugin and TLS, itself highly available (cluster mode); it stores the saved objects. | https://opensearch.org/docs/latest/install-and-configure/install-dashboards/index/ | external (external); failover load balancer (load-balancer); quorum none (none) | single: standalone, single (account min 1, max 1, dashboards min 1, max 1)replicated: ha, active-active (account min 1, max 1, dashboards min 2, recommended 2) |
| PostgreSQL | dcs: etcd in mode cluster (3 or 5 members, Raft quorum): Patroni demotes the leader when etcd is unreachable, so a single etcd member would make it the single point of failure. | https://patroni.readthedocs.io/en/latest/ha_loop.html https://www.postgresql.org/docs/16/warm-standby.html https://patroni.readthedocs.io/en/latest/yaml_configuration.html https://patroni.readthedocs.io/en/latest/rest_api.html https://www.postgresql.org/docs/16/libpq-connect.html https://github.com/pgvector/pgvector | native (native); failover orchestrated (orchestrated); quorum Raft (raft) | standalone: standalone, single (postgres min 1, max 1)ha: ha, active-passive (postgres min 2, max 7, recommended 3) |
| Temporal Web UI | temporal: Temporal Server reachable in gRPC, itself highly available. | https://docs.temporal.io/references/web-ui-configuration | external (external); failover load balancer (load-balancer); quorum none (none) | single: standalone, single (ui min 1, max 1)replicated: ha, active-active (ui min 2, recommended 2) |
| Opik | mysql: Highly available MySQL (state database).clickhouse: Highly available ClickHouse (traces and spans).redis: Highly available Redis.s3: Highly available object storage for the attachments. | https://www.comet.com/docs/opik/self-host/scaling | external (external); failover load balancer (load-balancer); quorum none (none) | single: standalone, single (python-backend min 1, max 1, backend min 1, max 1, frontend min 1, max 1)replicated: ha, active-active (python-backend min 2, recommended 2, backend min 2, recommended 2, frontend min 1, recommended 2) |
| Apache APISIX | etcd: etcd holds the routes of every node: it needs its own Raft cluster (3 or 5 members). | https://apisix.apache.org/docs/apisix/installation-guide/ | external (external); failover load balancer (load-balancer); quorum none (none) | standalone: standalone, single (apisix-gateway min 1, max 1)replicated: ha, active-active (apisix-gateway min 2, recommended 2) |
| HAProxy | keepalived: keepalived on the same two hosts moves the virtual IP (VRRP). | https://docs.haproxy.org/3.2/configuration.html | external (external); failover virtual IP (vip); quorum VRRP (vrrp) | single: standalone, single (haproxy min 1, max 1)pair: ha, active-passive (haproxy min 2, max 2) |
| LiteLLM gateway | database: Highly available PostgreSQL (keys, budgets, spend).cache: Highly available Redis, mandatory with several instances (shared limits and cache). | https://docs.litellm.ai/docs/proxy/prod | external (external); failover load balancer (load-balancer); quorum none (none) | single: standalone, single (litellm-proxy min 1, max 1)replicated: ha, active-active (litellm-proxy min 2, recommended 3) |
| LemonLDAP::NG | database: Highly available PostgreSQL holding the sessions of every portal node. | https://lemonldap-ng.org/documentation/latest/highavailability.html | external (external); failover load balancer (load-balancer); quorum none (none) | standalone: standalone, single (schema min 1, max 1, portal min 1, max 1)cluster: ha, active-active (schema min 1, max 1, portal min 2, recommended 2) |
| SPIRE | datastore: Highly available PostgreSQL shared by the servers (MySQL is also supported upstream). | https://spiffe.io/docs/latest/planning/scaling_spire/ | external (external); failover load balancer (load-balancer); quorum none (none) | standalone: standalone, single (spire-server min 1, max 1, spire-agent min 0)ha: ha, active-active (spire-server min 3, max 3, spire-agent min 0) |
| Navin Enterprise | database: The replicas share their state in PostgreSQL, which must itself be highly available (Patroni cluster).redis: The rate limit counters of the replicas live in Redis, which must itself be highly available (Sentinel).clamav: Every upload is scanned before ingestion (enforce mode); at least two ClamAV gateways keep uploads flowing when one fails.yarax: Every upload is matched against the YARA-X rules before ingestion; at least two workers keep uploads flowing when one fails.haproxy: The API and portal replicas are reached through HAProxy behind a Keepalived virtual IP. | https://docs.haproxy.org/3.2/configuration.html https://www.postgresql.org/docs/16/high-availability.html | external (external); failover load balancer (load-balancer); quorum none (none) | standard: standalone, single (migrate min 1, max 1, control-plane min 1, max 1, worker min 1, max 1, media-worker min 0, portal min 1, max 1)ha: ha, active-active (migrate min 1, max 1, control-plane min 2, recommended 3, worker min 1, max 2, recommended 2, media-worker min 0, portal min 2, recommended 3) |
| Jaeger | opensearch: Highly available OpenSearch shared by every instance; the memory storage cannot be shared. | https://www.jaegertracing.io/docs/2.22/storage/ | external (external); failover load balancer (load-balancer); quorum none (none) | single: standalone, single (jaeger min 1, max 1)replicated: ha, active-active (jaeger min 2, recommended 2) |
| OpenTelemetry Collector | - | https://opentelemetry.io/docs/collector/deploy/gateway/ | external (external); failover load balancer (load-balancer); quorum none (none) | agent: ha, per host (otel-collector min 1, recommended 3)gateway: ha, active-active (otel-collector min 2, recommended 2) |
| Prometheus | alertmanager: Alertmanager cluster deduplicating the alerts of both replicas. | https://prometheus.io/docs/introduction/faq/ | redundancy (redundancy); failover client side (client); quorum none (none) | single: standalone, single (prometheus min 1, max 1)ha: ha, active-active (prometheus min 2, max 2, recommended 2) |
| ClamAV | - | https://docs.clamav.net/ https://docs.clamav.net/manual/Usage/Scanning.html https://docs.clamav.net/manual/Usage/Configuration.html https://docs.clamav.net/appendix/CvdPrivateMirror.html | external (external); failover client side (client); quorum none (none) | standalone: standalone, single (freshclam min 1, max 1, clamd min 1, max 1, gateway min 1, max 1)replicated: ha, active-active (freshclam min 2, recommended 2, clamd min 2, recommended 2, gateway min 2, recommended 2) |
| Open Policy Agent | - | https://www.openpolicyagent.org/docs/deploy | external (external); failover load balancer (load-balancer); quorum none (none) | standalone: standalone, single (opa-server min 1, max 1)replicated: ha, active-active (opa-server min 2, recommended 3) |
| OpenFGA | database: Highly available PostgreSQL holding the authorization model and the tuples. | https://openfga.dev/docs/best-practices/running-in-production | external (external); failover load balancer (load-balancer); quorum none (none) | memory: standalone, single (openfga-server min 1, max 1)standalone: standalone, single (migrate min 1, max 1, openfga-server min 1, max 1)replicated: ha, active-active (migrate min 1, max 1, openfga-server min 2, recommended 3) |
| Presidio | - | https://microsoft.github.io/presidio/installation/ | external (external); failover load balancer (load-balancer); quorum none (none) | standalone: standalone, single (analyzer min 1, max 1, anonymizer min 1, max 1)replicated: ha, active-active (analyzer min 2, recommended 2, anonymizer min 2, recommended 2) |
| YARA-X | - | https://virustotal.github.io/yara-x/ https://virustotal.github.io/yara-x/docs/ | external (external); failover client side (client); quorum none (none) | standalone: standalone, single (worker min 1, max 1)replicated: ha, active-active (worker min 2, recommended 2) |
Per host agents
| Service | Prerequisites | Official documentation | High availability | Replicas / configuration |
|---|---|---|---|---|
| Grafana Alloy | otel: OpenTelemetry Collector receiving the data of the agents; replicate it (gateway mode) so that an agent always has a destination. | https://github.com/grafana/alloy/blob/main/docs/sources/configure/clustering/configure-alloy.md | per host (per-host); failover none (none); quorum none (none) | every-host: ha, per host (alloy min 0, recommended 3) |
| Node exporter | - | https://github.com/prometheus/node_exporter | per host (per-host); failover none (none); quorum none (none) | every-host: ha, per host (node-exporter min 0, recommended 3) |
| Falco | - | https://falco.org/docs/setup/operator/ | per host (per-host); failover none (none); quorum none (none) | per-host: ha, per host (falco min 1, recommended 3)standalone: standalone, single (falco min 1, max 1) |
| Tetragon | - | https://tetragon.io/docs/installation/kubernetes/ | per host (per-host); failover none (none); quorum none (none) | per-host: ha, per host (tetragon min 1, recommended 3)standalone: standalone, single (tetragon min 1, max 1) |
Tools without a native cluster
| Service | Prerequisites | Official documentation | High availability | Replicas / configuration |
|---|---|---|---|---|
| LiveKit Ingress | redis: Highly available Redis shared with LiveKit Server. | https://docs.livekit.io/transport/self-hosting/ingress/ https://github.com/livekit/ingress | redundancy (redundancy); failover load balancer (load-balancer); quorum none (none) | single: standalone, single (ingress min 1, max 1)pool: ha, active-active (ingress min 2, recommended 2) |
| garak | - | https://docs.garak.ai/garak | none (none); failover none (none); quorum none (none) | client: standalone, parallel jobs (garak min 1, recommended 1) |
| LLaMA-Factory | - | https://llamafactory.readthedocs.io/en/latest/getting_started/webui.html https://github.com/hiyouga/LLaMA-Factory/tree/v0.9.5/docker/docker-cuda | none (none); failover none (none); quorum none (none) | single: standalone, single (board min 1, max 1)multi-host: standalone, per host (board min 2, recommended 2) |
| promptfoo | - | https://www.promptfoo.dev/docs/usage/web-ui/ | none (none); failover none (none); quorum none (none) | single: standalone, parallel jobs (viewer min 1, max 1) |
| Ragas | - | https://docs.ragas.io/en/v0.3.2/getstarted/rag_eval/ | none (none); failover none (none); quorum none (none) | client: standalone, parallel jobs (ragas min 1, recommended 1) |
| MCP hosting | - | https://modelcontextprotocol.io/specification/2025-03-26/basic/transports | redundancy (redundancy); failover none (none); quorum none (none) | single: standalone, single (mcp-host min 1, max 1)replicated: standalone, active-active (mcp-host min 2, recommended 2) |
| Cognee | - | https://docs.cognee.ai/setup-configuration/graph-stores https://github.com/topoteretes/cognee/blob/v1.6.6/entrypoint.sh https://github.com/topoteretes/cognee/blob/v1.6.6/cognee/infrastructure/databases/graph/postgres_demo/adapter.py https://github.com/topoteretes/cognee/blob/v1.6.6/cognee/infrastructure/databases/dataset_database_handler/supported_dataset_database_handlers.py | none (none); failover none (none); quorum none (none) | single: standalone, single (api min 1, max 1, ui min 1, max 1, mcp min 1, max 1) |
| Mem0 | - | https://docs.mem0.ai/open-source/features/rest-api https://github.com/mem0ai/mem0/blob/v2.2.1/server/main.py https://github.com/mem0ai/mem0/blob/v2.2.1/mem0/memory/storage.py | none (none); failover none (none); quorum none (none) | single: standalone, single (mem0-server min 1, max 1, request-log-pruner min 1, max 1) |
| Asterisk PBX | - | https://docs.asterisk.org/Fundamentals/Asterisk-Architecture/ https://www.kamailio.org/docs/modules/stable/modules/dispatcher.html https://docs.asterisk.org/About-the-Project/Asterisk-Versions/ | redundancy (redundancy); failover load balancer (load-balancer); quorum none (none) | single: standalone, single (asterisk min 1, max 1)pool: ha, active-active (asterisk min 2, recommended 2) |
| RTPengine media relay | - | https://rtpengine.readthedocs.io/en/latest/overview.html https://www.kamailio.org/docs/modules/5.7.x/modules/rtpengine.html | redundancy (redundancy); failover client side (client); quorum none (none) | single: standalone, single (rtpengine min 1, max 1)redundant: ha, active-active (rtpengine min 2, recommended 2) |
| Blackbox exporter | - | https://github.com/prometheus/blackbox_exporter/blob/master/README.md | redundancy (redundancy); failover none (none); quorum none (none) | single: standalone, single (blackbox-exporter min 1, recommended 1) |
| PostgreSQL exporter | postgresql: Monitoring account on the watched server. | https://github.com/prometheus-community/postgres_exporter/blob/master/docs/README.md | external (external); failover none (none); quorum none (none) | single: standalone, single (postgres-exporter min 1, max 1) |
| Redis exporter | redis: Monitoring account on the watched server. | https://github.com/oliver006/redis_exporter/blob/master/README.md | external (external); failover none (none); quorum none (none) | single: standalone, single (redis-exporter min 1, max 1) |
Libraries and workstation software
| Service | Prerequisites | Official documentation | High availability | Replicas / configuration |
|---|---|---|---|---|
| Haystack | - | https://docs.haystack.deepset.ai/docs/intro | none (none); failover none (none); quorum none (none) | - |
| OpenLineage SDK | - | https://openlineage.io/docs/ | none (none); failover none (none); quorum none (none) | - |
| Navin SM | - | https://github.com/Navinspire-ia/navin-sm | none (none); failover none (none); quorum none (none) | - |
| Tink | - | https://developers.google.com/tink | none (none); failover none (none); quorum none (none) | - |