Skip to main content

Network and DNS hardening

French version: ../network-dns-hardening.md.

Principle​

Every namespace starts from a full deny, ingress and egress, DNS included. Each workload then opens only the flows it needs. Chart rules are rendered by deploy/helm/enterprise-ai/templates/control-plane/networkpolicy.yaml (Cilium flavour by default, plain Kubernetes NetworkPolicy with global.networkPolicy=kubernetes). Data and security namespaces are covered by deploy/cilium/data-services.yaml, cluster-wide policies by deploy/cilium/baseline.yaml.

PolicyScopeEffect
eai-default-denyenterprise-ai namespaceingress: [{}] and egress: [{}]: no flow, not even DNS
eai-deny-metadataevery pod of the namespaceegressDeny to 169.254.169.254/32, 169.254.0.0/16, fd00:ec2::254/128
eai-deny-cloud-metadatawhole cluster (CiliumClusterwideNetworkPolicy)same deny for every workload in any namespace
eai-deny-kube-apiserverevery pod except control-planeegressDeny to the kube-apiserver entity
default-denyeai-data, openbao, eai-securityfull deny, then one policy per service

egressDeny rules always win over allow rules: an SSRF bug in an application cannot read instance credentials (AWS IMDS, GCP, Azure, OpenStack).

Chart flow matrix​

WorkloadAllowed ingressAllowed egress
control planeAPISIX, portal, observability on 8080PostgreSQL, NATS, Redis, Qdrant, Temporal, LiveKit (eai-data), external S3 storage 443, OpenFGA, OPA, Keycloak (eai-security), OpenBao 8200, OTLP, Opik, ClickHouse, LiteLLM 4000, vLLM 8000, Ollama 11434, Kubernetes API, provider FQDNs on 443
background workernonePostgreSQL, NATS, Redis, OpenBao, OTLP
migrationnonePostgreSQL, OpenBao
portalingress controller, APISIX on 3000control plane, Keycloak, OTLP
LiteLLMcontrol plane, observability on 4000vLLM, Ollama, PostgreSQL, Redis, OpenBao, OTLP, provider FQDNs on 443
vLLMLiteLLM, control plane, observability on 8000none (weights preloaded, HF_HUB_OFFLINE=1)
multimodal workersnoneNATS, external S3 storage 443, OTLP

Only the control plane and LiteLLM reach the Internet, through an FQDN list (networkSecurity.providerFQDNs in values.yaml, same as deploy/cilium/egress-providers.yaml). Set networkSecurity.providerEgress.controlPlane or .litellm to false to close one side.

DNS: per workload L7 allowlist​

The Cilium DNS proxy enforces a matchName / matchPattern list per workload:

  • internal names limited to the namespaces actually called, for example *.eai-data.svc.cluster.local;
  • external names limited to the provider FQDNs, for the control plane and LiteLLM only;
  • vLLM has no DNS access at all.

Any other query gets REFUSED: a compromised pod can neither resolve an arbitrary domain nor exfiltrate data encoded in subdomains (DNS tunneling). Cilium unions the DNS rules of every policy selecting a pod, so no policy may contain matchPattern: "*". The eai-dns-visibility policy in baseline.yaml is now limited to *.svc.cluster.local, and test_platform_security_manifests.py rejects any * pattern.

Application side: use fully qualified names (nats.eai-data.svc.cluster.local), as values-prod.yaml does. With short names the resolver walks the search domains, those intermediate queries are refused and resolution slows down.

CoreDNS and NodeLocal DNSCache​

  • No open resolver: the kube-dns Service stays ClusterIP, never exposed through LoadBalancer or NodePort (the Kyverno policy eai-restrict-service-exposure blocks NodePorts outside ingress and apisix). CoreDNS listens on the pod network only.
  • Restricted upstream forwarding: in the Corefile, forward . <corporate resolvers> with explicit addresses, never the node /etc/resolv.conf when it points to a public resolver. Add policy sequential and health_check 5s.
  • DNSSEC or DoT to the resolver: either the upstream resolver validates DNSSEC (Unbound auto-trust-anchor-file, BIND dnssec-validation auto), or CoreDNS speaks DoT to it: forward . tls://10.0.0.53 { tls_servername <dns-host> }. Do not enable the CoreDNS dnssec plugin (it signs, it does not validate).
  • No external recursion for pod records: the cluster.local block uses the kubernetes plugin with pods verified (not pods insecure), so a-b-c-d.ns.pod.cluster.local only answers for an existing pod.
  • Logging: log plugin filtered on denial and error, shipped to OpenSearch (index eai-logs-*) by the OpenTelemetry Collector; the Cilium DNS proxy also reports refusals in Hubble (hubble observe --verdict DROPPED --protocol dns).
  • NodeLocal DNSCache: the eai-dns-visibility policy and the workload rules select k8s-app: kube-dns. With NodeLocal DNSCache, add the k8s-app: node-local-dns selector to the same rules (same port 53, same L7 rules); the local cache forwards to CoreDNS over TCP and has no other upstream.
  • Rate limiting: cache 30 plus the ratelimit external plugin or the NodeLocal DNSCache query limit protect CoreDNS from a pod flooding DNS.

Offline verification​

helm template eai deploy/helm/enterprise-ai -n enterprise-ai -f deploy/helm/enterprise-ai/values-prod.yaml | kubeconform -strict -summary
services/control-plane/.venv/bin/python -m pytest services/control-plane/tests -q -k platform

On a test cluster (never in production without a change window): hubble observe --namespace enterprise-ai --verdict DROPPED must show the expected drops, and a kubectl exec from a portal pod to 169.254.169.254 must fail.