Skip to main content

ADR-0167: MCP Runtime isolated per tenant, project and user on the ToolHive Kubernetes Operator

Context​

The MCP Gateway (eai.domains.mcp) decides which users and agents reach which MCP servers. A runtime was missing that really creates isolated instances, starts them on demand and destroys them after use, with the credentials and authorisations of each user. The hosting of the Navin Manager (ADR-0165) runs one process per server and per project, not per user. The ToolHive Kubernetes Operator (Stacklok, Apache-2.0, self-hosted) deploys MCP servers in Kubernetes namespaces and links them to an OIDC source, but authentication is not enabled by default, network policies are not set by default and an idle timeout is not a verified ToolHive capability.

Decision​

  1. ToolHive Operator 0.59.0 service: pinned OCI charts, images pinned by digest, 2 replicas with leader election, PodDisruptionBudget, CRDs kept, deployed by Argo CD in toolhive-system and also installable from the MCP runtime page of the Navin Manager.
  2. Lifecycle controller in the Control Plane (services/control-plane/src/eai/domains/mcp_runtime, migration V200): creation, execution, idle time, destruction. One namespace per project mcp-<12 hex> (Pod Security restricted, ResourceQuota, LimitRange, NetworkPolicy default-deny, MCPOIDCConfig). Per instance: an MCPServer inst-<10 hex> that always has oidcConfigRef (audience of its own), a Cedar policy limited to the owner, its agents and the allowed tools, audit on, an image pinned by digest and a container without privilege (read only root, no capability, non root, RuntimeDefault seccomp, no service account token); an MCPToolConfig; a Secret of its own; a CiliumNetworkPolicy (Control Plane to the proxy of the instance only, MCP container to the hosts of its template only).
  3. Temporary credentials of the instance only: dynamic OpenBao lease (database engine) for PostgreSQL, the personal secret of the user given by reference for GitHub. They are written in the Secret of the instance only; the lease id is sealed, then cleared once revoked.
  4. Policy per project set by tenant_admin: templates, hosts, database roles, idle time (900 seconds by default), maximum lifetime, quotas per user and per project. The idle time is a Navin policy: the worker job mcp_runtime_reaper destroys the instances that are idle, expired, orphaned or stuck while starting, and retries the revocations and deletions that failed.
  5. Gateway: the entry of an instance is callable by its owner and its agents only, with a short JWT signed by the key of the Context Authority (public keys at GET /v1/mcp/runtime/jwks). The access list per user and agent (PUT /v1/mcp/servers/{server_id}/access) applies to regular servers too.
  6. Destruction: lease revoked, MCPServer, MCPToolConfig, CiliumNetworkPolicy and Secret deleted, gateway entry disabled. The journal eai.mcp_runtime_events is append only, without foreign key to the instance, and outlives it; the logs of the pods of the MCP Runtime namespaces go to a separate OpenSearch index (eai-mcp-runtime-audit, 400 days retention).
  7. Cluster guardrails: Kyverno ClusterPolicy eai-mcp-runtime in Enforce (only the service account of the Control Plane writes in these namespaces; an MCPServer without OIDC, Cedar, audit, pinned image or hardened container is refused; NetworkPolicy default-deny generated) and ClusterRole eai-mcp-runtime-controller of the Helm chart (value mcpRuntime.enabled).
  8. Interface: ToolHive UI is a local desktop application, with no web interface on the cluster side. Operation happens in the Navin Manager (MCP runtime page: operator state, instances, destruction) and in the portal (MCP instances tab of the Navin library).

Alternatives considered​

  • mcp-host hosting alone (ADR-0165): one process per server and per project, no identity per user and no isolated container. It stays for installations without Kubernetes.
  • ToolHive without a Navin controller: no idle time, no revocation of credentials, and authentication and network to set by hand.
  • One MCP proxy shared per tenant: the credentials of a user would be usable by the other members of the project.

Consequences​

Every user runs their MCP servers with their own credentials and authorisations, without exposing their resources to other projects nor to the other members of their project. The MCP Runtime needs a Kubernetes cluster with ToolHive, Cilium and Kyverno, and EAI_MCP_RUNTIME_ENABLED; without it, creating an instance answers 409 mcp_runtime_disabled.

Evidence​

services/control-plane/tests/test_mcp_runtime.py, services/control-plane/tests/test_platform_mcp_runtime.py, navin-manager/server/tests/test_mcp_runtime_k8s.py, navin-manager/ui/src/test/mcp-runtime.test.tsx, portal/tests/navin-library.test.ts, deploy/policies/kyverno/mcp-runtime.yaml, deploy/cilium/mcp-runtime.yaml.