Harden a production installation

This page is the security checklist of a production installation of the Navin Platform: each step says what it does, why, the command or screen, the expected result and what to do if it fails. It is meant for the platform operator (platform_admin role) and the security officer (security_officer). Indicative duration: half a day, backups included.

Version française : ../hardening.md

The settings quoted are fields of services/control-plane/src/eai/config.py (EAI_ prefix); their full reference is in configuration.md and ../../operations/en/configuration.md. Infrastructure hardening (SPIFFE mTLS, Cilium, pods, supply chain) is described in ../../operations/en/security-hardening.md and is not repeated here.

Before you begin

• Access: a platform_admin account and a distinct security_officer account (several steps need two people).
• OpenBao initialised and unsealed, entries written as in ../../operations/en/secrets.md.
• Keycloak with the realm deploy/keycloak/realm-enterprise-ai.json imported.
• Production Helm values: deploy/helm/enterprise-ai/values-prod.yaml.

danger

Never disable the startup guard to "get the Control Plane to start". Every message it prints is a real weakness: fix the setting.

Steps

1. Turn the startup guard on. EAI_ENVIRONMENT=prod makes the Control Plane refuse to start with any unsafe setting (eai.security.startup.enforce, which lists every problem). In staging the same problems are only logged.

   controlPlane:
     config:
       EAI_ENVIRONMENT: prod

   Expected result: the pod starts; no must message in the Control Plane log.

   If it fails: the pod stops with a RuntimeError listing every faulty setting; fix them one by one with the next steps.

2. Encrypt every link (TLS). The guard refuses EAI_DATABASE_URL and EAI_RUNTIME_DATABASE_URL without sslmode=require, verify-ca or verify-full, EAI_REDIS_URL without rediss://, and a Redis without EAI_REDIS_PASSWORD_FILE (ACL user). Turn HSTS on (EAI_HSTS_ENABLED=true, EAI_HSTS_MAX_AGE_SECONDS=31536000). The file scanning engines must be https://.

   psql "<EAI_DATABASE_URL>" -c "SHOW ssl"

   <EAI_DATABASE_URL>: the connection string read from OpenBao, with sslmode=verify-full and the CA.

   Expected result: on. API responses carry Strict-Transport-Security: max-age=31536000; includeSubDomains.

   If it fails: check controlPlane.secretFormats in Helm and the database certificate (../../operations/en/data-services-hardening.md).

3. Close development authentication. EAI_AUTH_DEV=false (the X-Dev-* headers authenticate anyone; eai.auth.dev_auth_active ignores them in prod anyway). Configure OIDC: EAI_OIDC_ISSUER, EAI_OIDC_AUDIENCE (the audience then becomes mandatory in the token), optionally EAI_OIDC_JWKS_URL. Keep EAI_JWT_ALGORITHMS asymmetric (HS* and none are removed in every environment). EAI_AUTH_SPIFFE=true only behind the mesh, with EAI_TRUSTED_PROXIES limited to the gateway pods.

   curl -s <control-plane-url>/v1/identity/whoami -H "Authorization: Bearer <token>"

   Expected result: auth_method is oidc and tenant_id is the user's tenant.

   If it fails: 401 invalid_token with a reason; check the issuer, the audience and the clock (EAI_JWT_LEEWAY_SECONDS).

4. Secure the installer account. The local account created from EAI_BOOTSTRAP_ADMIN_USERNAME and EAI_BOOTSTRAP_ADMIN_PASSWORD holds platform_admin and tenant_admin and must change its password at the first sign in (POST /v1/auth/password). In production the guard refuses a weak password (fewer than EAI_LOCAL_PASSWORD_MIN_LENGTH characters, no digit or no letter, equal to the login or known). Local accounts lock after EAI_LOCAL_AUTH_MAX_FAILURES failures for EAI_LOCAL_AUTH_LOCKOUT_SECONDS. If all authentication goes through Keycloak, set EAI_LOCAL_AUTH_ENABLED=false.

   Expected result: after the first sign in, remove EAI_BOOTSTRAP_ADMIN_PASSWORD from the environment; a restart never resets the account.

   If it fails: 403 password_change_required on every other route until the initial password is changed.

5. Restrict hosts, origins and documentation. EAI_ALLOWED_HOSTS lists the public API host names, EAI_CORS_ORIGINS the exact portal origins (never *), EAI_DOCS_ENABLED=false closes /docs, /redoc and /openapi.json. EAI_METRICS_TOKEN (32 characters minimum) protects GET /metrics.

   Expected result: a request with an unknown Host header is refused; /docs answers 404.

   If it fails: the guard names the faulty setting.

6. Close network egress. EAI_EGRESS_ALLOW_HTTP=false (mandatory), EAI_EGRESS_ALLOWED_PORTS=443, EAI_EGRESS_PRIVATE_ALLOWLIST limited to exact internal services. Per tenant, egress_allowed_hosts of PUT /v1/security/settings restricts egress to a list. Internal MCP servers go in EAI_MCP_INTERNAL_HOSTS.

   Expected result: a tool calling 169.254.169.254 is refused and the security.egress.denied event is written to the audit trail.

   If it fails: see runbooks.md and ../../operations/en/network-dns-hardening.md.

7. Wire OPA and OpenFGA fail closed. With EAI_OPA_URL, every engine=opa policy is evaluated by OPA; an unreachable OPA means deny (403 policy_denied, reason policy engine unavailable). In production prefer EAI_OPA_DELIVERY=bundle with EAI_POLICY_BUNDLE_SIGNING_KEY_FILE (RS256 signed bundles, mandatory in bundle mode). With EAI_OPENFGA_URL, EAI_OPENFGA_STORE_ID, EAI_OPENFGA_MODEL_ID and EAI_OPENFGA_API_TOKEN, an OpenFGA outage answers 502 authz_backend_error and the denial is audited. EAI_POLICY_DEFAULT=deny refuses an action no rule covers (code default: allow, also settable per tenant with policy_default).

   curl -s <control-plane-url>/v1/policies/engines -H "Authorization: Bearer <token>"

   Expected result: OPA reachable and in_sync is true in bundle mode.

   If it fails: in_sync: false means OPA keeps the previous revision (bundle refused); see identity-policy-plane.md.

8. Keep rate and size limits. EAI_RATE_LIMIT_ENABLED=true, EAI_RATE_LIMIT_PER_MINUTE (per principal), EAI_RATE_LIMIT_GATEWAY_PER_MINUTE (on /v1/gateway/*), EAI_RATE_LIMIT_AUTH_FAILURES_PER_MINUTE (per IP, then 429), EAI_MAX_BODY_BYTES, EAI_MAX_UPLOAD_BYTES, EAI_MAX_JSON_DEPTH. With several replicas, EAI_REDIS_URL shares the counters.

   Expected result: past the budget, a 429 answer and a security.rate_limited event.

   If it fails: tuning in runbooks.md.

9. Check headers and CSP. EAI_SECURITY_HEADERS_ENABLED=true adds to every API response (errors included, eai.security.headers): X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Content-Security-Policy: default-src 'none'; frame-ancestors 'none'; base-uri 'none'; form-action 'none', Referrer-Policy: no-referrer, Permissions-Policy, Cross-Origin-Opener-Policy: same-origin, Cross-Origin-Resource-Policy: same-site. The portal sets its own CSP with a nonce (portal/src/middleware.ts, buildCsp in portal/src/lib/security.ts).

   curl -sI <control-plane-url>/v1/identity/whoami | grep -i -E "content-security|x-frame|strict-transport"

   Expected result: the three headers are present.

   If it fails: a CSP violation is handled as in runbooks.md, never with unsafe-inline.

10. Keep keys outside the database. The guard requires EAI_OPENBAO_URL and EAI_OPENBAO_TOKEN (and EAI_AUDIT_SIGNING other than hmac), EAI_AUDIT_SEAL_SIGNING_KEY_REF (Ed25519 seal key in OpenBao), EAI_AUDIT_PAYLOAD_KEK=openbao-transit, EAI_CONTEXT_SIGNING_KEY_REF and EAI_CONTEXT_REQUIRED=true. For an HSM: EAI_KMS_PROVIDER=pkcs11 and its three settings. Details: secrets-vault.md.

    Expected result: GET /v1/context/jwks publishes a key; GET /v1/evidence/verify answers without a break.

    If it fails: 503 context_authority_unavailable means the Context Authority key cannot be read.

11. Enforce file scanning. EAI_FILE_SCAN_MODE=enforce, EAI_CLAMAV_URLS, EAI_CLAMAV_API_KEY, EAI_YARAX_URLS, EAI_YARAX_API_KEY and EAI_FILE_SCAN_CA_FILE. Details: file-security-pipeline.md.

    Expected result: GET /v1/file-security/status shows both engines ready.

    If it fails: with no reachable engine every file upload is refused (503), never accepted unscanned.

12. Prepare the kill switch. EAI_SOC_WEBHOOK_SECRET (32 characters minimum) authenticates the SOC webhooks and the cluster receipts; empty, every delivery is refused. Check that the registry answers and that two leads know the four eyes recovery procedure.

    curl -s <control-plane-url>/v1/soc/kill-switches -H "Authorization: Bearer <token>"

    Expected result: a list (empty on a fresh installation).

    If it fails: see soc-runbook.md.

13. Bound break glass. EAI_BREAK_GLASS_MAX_SECONDS (default 3600) caps the duration of a break glass grant; approval needs two distinct approvers, never the requester nor the beneficiary. Run a drill: POST /v1/break-glass, two approvals, then POST /v1/break-glass/{grant_id}/revoke.

    Expected result: the grant expires on its own or is revoked, and every step shows in GET /v1/audit/events.

    If it fails: 403 separation_of_duties when the requester tries to approve.

14. Back up and prove the restore. services/control-plane/scripts/pg_backup.sh and services/control-plane/scripts/pg_restore.sh keep a verifiable audit chain, seals and legal holds (test_restored_backup_keeps_verifiable_chain_seals_and_hold_state). The Navin Manager backup and DR plans are read with GET /v1/platform-manager/backup/plans; a restore needs two approvers other than the requester (POST /v1/platform-manager/backup/restores/{plan_id}/approve).

    Expected result: after a restore on a test environment, GET /v1/audit/verify and GET /v1/evidence/verify are green.

    If it fails: see ../../operations/en/ha-dr.md.

Verify

GET /v1/security/posture (security_officer or tenant_admin roles) returns every posture check and a summary. A hardened installation leaves no failing check among auth_dev_disabled, cors_restricted, allowed_hosts_restricted, hsts_enabled, docs_disabled, secrets_backend, oidc_configured, egress_http_disabled, egress_private_allowlist_narrow, rate_limit_enabled, trusted_proxies_restricted, spiffe_behind_proxy, database_tls, jwt_algorithms_asymmetric, prompt_injection_enabled.

curl -s <control-plane-url>/v1/security/posture -H "Authorization: Bearer <token>"

Next steps

• Roles and rights: authentication-authorization.md.
• Governance: governance.md.
• Incident procedures: runbooks.md.

Troubleshooting

Symptom	Cause	Fix
The Control Plane refuses to start	Unsafe setting with EAI_ENVIRONMENT=prod	Fix every line listed by the guard
Every request seems to come from the gateway	Wrong EAI_TRUSTED_PROXIES	Set the CIDR of the gateway pods
403 policy_denied on every tool call	OPA unreachable	Restore OPA; the denial is intended (fail closed)
502 authz_backend_error	OpenFGA unreachable	Restore OpenFGA; the denial is audited
503 on every file upload	No ClamAV or YARA-X engine reachable	Restore the engines (file-security-pipeline.md)