Skip to main content

Security hardening

Identity and mTLS​

  • Every workload gets spiffe://enterprise-ai.internal/ns/<ns>/sa/<sa> (ServiceAccount annotation spiffe.io/spiffe-id, ClusterSPIFFEID in deploy/spire/registration-entries.yaml).
  • Cilium mutual auth (authentication.mode: required) between the control plane and LiteLLM, and between APISIX and the control plane; WireGuard node to node encryption.
  • Cilium mutual authentication wiring with SPIRE (hardened charts spire 0.30.4 and spire-crds 0.6.1, Cilium 1.16.5):
    • authentication.mutual.spire.serverAddress: spire-server.spire-server.svc:443 (Service spire-server in namespace spire-server, port 443 to container port 8081), trustDomain: enterprise-ai.internal, install.enabled: false (deploy/terraform/modules/platform-bootstrap/values/cilium.yaml).
    • SPIRE agent Workload API socket: /run/spire/agent-sockets/spire-agent.sock on the host (directory published to pods by the SPIFFE CSI driver), used by the Cilium operator.
    • Admin socket (Delegated Identity API): /run/spire/agent/sockets/csi.spiffe.io/admin/admin.sock on the host (/tmp/spire-agent/private/admin.sock in the container, sockets.admin.mountOnHost: true). This directory is not published by the CSI driver: only the Cilium agent mounts it.
    • authorized_delegates = ["spiffe://enterprise-ai.internal/cilium-agent"] in the SPIRE agent config (deploy/spire/values.yaml).
    • Entries (deploy/spire/registration-entries.yaml): ClusterSPIFFEID cilium-agent (kube-system, SA cilium) to spiffe://enterprise-ai.internal/cilium-agent and cilium-operator (kube-system, SA cilium-operator, admin: true) to spiffe://enterprise-ai.internal/cilium-operator; the operator then creates the /identity/<id> entries through the server API. controllerManager.ignoreNamespaces no longer lists kube-system so both objects are reconciled.
    • Network: SPIRE agents and Cilium are hostNetwork (entities host and remote-node); no default deny on namespace spire-server, port 8081 stays reachable.
  • cert-manager issues 30 day ECDSA certificates with key rotation.
  • EAI_AUTH_SPIFFE=1 only behind the mesh; APISIX strips external X-SPIFFE-ID and X-Dev-* headers. EAI_AUTH_DEV must be 0 outside development.

Network​

  • Default deny per namespace (chart CiliumNetworkPolicy) plus explicit allow lists.
  • Internet egress restricted to allowed provider FQDNs, one policy per provider, enabled by label. No egress from the sandbox namespace.
  • Never add an egressDeny to world on LiteLLM pods: deny rules override FQDN allows.

Secrets​

  • No secret in Git or values: OpenBao (KV v2) with External Secrets or the agent injector.
  • Per tenant envelope encryption with transit eai-tenant-*; audit checkpoints signed by eai-audit (ed25519, non exportable).
  • Least privilege HCL policies per service (deploy/openbao/policies). Rotation: bao write -f transit/keys/<key>/rotate.

Supply chain​

  • CI: Trivy fs + image, SPDX and CycloneDX SBOM (Syft), cosign signature and SBOM attestation.
  • Kyverno verifies cosign key signatures (the same public key as the CI for our harbor.*/eai/* images and for mirrors; the CI refuses to sign without COSIGN_PRIVATE_KEY), enforces non-root, read-only rootfs, resource limits, no latest, gVisor/Kata in the sandbox, tenant/owner labels.

Pods​

  • runAsNonRoot, UID 10001, readOnlyRootFilesystem, capabilities.drop: [ALL], seccomp RuntimeDefault, Pod Security restricted.

Audit​

  • Hash chained audit events, WORM export to the object lock bucket, forwarding to Wazuh (deploy/wazuh/rules).
  • Telemetry: the OTel collector drops prompts, completions and auth headers and masks emails, IBANs, card numbers and eai_/sk- keys.