MCP Runtime (ToolHive Operator)

French version: ../mcp-runtime.md.

The MCP Runtime runs the MCP (Model Context Protocol) servers of the agents and of the users inside the cluster. It relies on the ToolHive Kubernetes Operator by Stacklok (Apache-2.0, self-hosted, no external service), version 0.59.0. Every MCP instance is ephemeral and isolated per tenant, project and user: the lifecycle controller of the control plane creates it on demand and deletes it as soon as it is no longer used.

1. Architecture: Gateway and Runtime

Part	Role	Where
MCP Gateway	Single entry point of the MCP calls of agents and users: authentication (Keycloak), authorisation per user and per agent (OpenFGA, OPA), control plane audit trail, routing to the instance.	Control plane, namespace enterprise-ai
MCP Runtime	Runs the MCP servers: one namespace per project, one instance per user and server template, an authenticating ToolHive proxy in front of each server.	ToolHive Operator (toolhive-system) and namespaces mcp-<12 hex>

Flow of a call:

1. The agent or the user calls the Gateway of the control plane.
2. The lifecycle controller creates the project namespace if needed, then the instance (MCPServer and related objects).
3. The ToolHive operator reconciles the MCPServer: a proxy (eai.io/mcp-role: proxy, port 8080) and the MCP server (eai.io/mcp-role: server).
4. The Gateway calls the proxy with a short JWT signed by the control plane. The proxy checks the signature against the public JWKS of the control plane (route /v1/mcp/runtime/jwks, EAI_MCP_RUNTIME_JWKS_URL), the issuer (EAI_MCP_RUNTIME_ISSUER) and the audience, then applies its Cedar policy.
5. When the instance is no longer used, the controller deletes it and revokes its temporary credentials. When an MCPServer disappears on the cluster side, the controller closes the instance the same way.

Objects created by the controller (contract shared with services/control-plane):

Scope	Objects
Project namespace	Namespace mcp-<12 hex of sha256(tenant_id/project_id)> labelled eai.io/mcp-runtime: "true", eai.io/tenant, eai.io/project, app.kubernetes.io/managed-by: eai-mcp-runtime and pod-security.kubernetes.io/enforce, audit, warn set to restricted; NetworkPolicy default-deny (ingress and egress), ResourceQuota mcp-quota, LimitRange mcp-limits, MCPOIDCConfig eai-mcp-runtime
Instance	MCPServer inst-<10 hex>, MCPToolConfig of the same name, Secret inst-<10 hex>-env, CiliumNetworkPolicy inst-<10 hex>; labels eai.io/mcp-instance (UUID), eai.io/mcp-template, eai.io/owner (16 hex of sha256 of the user id, never the identity), eai.io/tenant, eai.io/project, app.kubernetes.io/managed-by: eai-mcp-runtime
Pods	eai.io/mcp-instance and eai.io/mcp-role (proxy or server)

2. Installation

Files:

File	Content
deploy/toolhive/crds-values.yaml	Values of the toolhive-operator-crds chart 0.59.0 (crds.keep: true)
deploy/toolhive/operator-values.yaml	Values of the toolhive-operator chart 0.59.0: 2 replicas with leader election, operator.rbac.scope: cluster, resources, security contexts, images pinned by digest
deploy/toolhive/k8s/namespace.yaml	Namespace toolhive-system (Pod Security restricted, labels tenant: platform, owner: platform-team)
deploy/toolhive/k8s/operator.yaml	PodDisruptionBudget, metrics Service and ServiceMonitor of the operator (the chart does not ship them)
deploy/argocd/repository-toolhive.yaml	Argo CD declaration of the OCI registry ghcr.io/stacklok/toolhive (enableOCI)
deploy/policies/kyverno/mcp-runtime.yaml	ClusterPolicy eai-mcp-runtime (Enforce)
deploy/cilium/mcp-runtime.yaml	Network policies of the operator, of the proxies and of the control plane
deploy/helm/enterprise-ai/templates/control-plane/mcp-runtime-rbac.yaml	ClusterRole and ClusterRoleBinding eai-mcp-runtime-controller

Argo CD components (deploy/argocd/applicationset.yaml):

Component	Layer	Source
toolhive-operator-crds	infra, before kyverno-policies (the policy matches the MCPServer kind)	OCI chart ghcr.io/stacklok/toolhive/toolhive-operator-crds 0.59.0
toolhive-operator	platform	OCI chart ghcr.io/stacklok/toolhive/toolhive-operator 0.59.0
toolhive-runtime	platform	deploy/toolhive/k8s

Argo CD only pulls a Helm chart from an OCI registry declared with enableOCI, and the address has no oci:// prefix: apply deploy/argocd/repository-toolhive.yaml with the AppProject and the ApplicationSet (installation.md). The AppProject allows ghcr.io/stacklok/toolhive. The toolhive-system namespace is created by the deploy/terraform/modules/platform-bootstrap module, like the other namespaces of the platform.

Images (Harbor mirror, digest of the OCI index read from ghcr.io for the v0.59.0 tag; copy with skopeo copy --all to keep the same digest, then sign with the internal key like the other mirror images):

Upstream image	Digest
ghcr.io/stacklok/toolhive/operator:v0.59.0	sha256:b094d9220a687ef975a62f9c073fa5faf433e88c3f355f8ad29f50cd082e430c
ghcr.io/stacklok/toolhive/proxyrunner:v0.59.0	sha256:f0e970f749ef10004d150721311fa8512ba4c349768ca7b1915ff77452750250
ghcr.io/stacklok/toolhive/vmcp:v0.59.0	sha256:e9077bc25803740423a04333d8400a02b6f5f44d5ba7c6b5edf65437ce1fe0dd
ghcr.io/stacklok/thv-registry-api:v1.5.3	sha256:323652f7bd195a0a6a84815e749f52a809633a07d911580abb2d58945abc78f3

Chart artefacts: toolhive-operator 0.59.0 sha256:0920e45a4d371a6a9096c0019afea688e3aecbd1167182f5dacfe86b37605449, toolhive-operator-crds 0.59.0 sha256:b7694eb741a04043818978c34b0136cf8cef30329a95ae16de518bd02daaf620.

Control plane side, in the values of the deploy/helm/enterprise-ai chart:

mcpRuntime:
  enabled: true
  issuer: ""            # empty: http://<release>-control-plane.<namespace>.svc.cluster.local:8080/v1/mcp/runtime
  jwksUrl: ""           # empty: <issuer>/jwks
  gatewayNamespace: ""  # empty: namespace of the release

The value adds the ClusterRole eai-mcp-runtime-controller (namespaces: get, list, create, patch, delete; resourcequotas, limitranges, secrets, networkpolicies, ciliumnetworkpolicies, mcpservers, mcpoidcconfigs, mcptoolconfigs: get, list, watch, create, patch, update, delete), its binding to the control plane service account and the variables EAI_MCP_RUNTIME_ENABLED, EAI_MCP_RUNTIME_ISSUER, EAI_MCP_RUNTIME_JWKS_URL, EAI_MCP_RUNTIME_GATEWAY_NAMESPACE of the control plane Deployment. The default is false: nothing is rendered.

3. Hardening

Mandatory authentication. ToolHive does not enable authentication by default: an MCPServer without an OIDC configuration exposes an anonymous proxy. Every MCPServer therefore references the MCPOIDCConfig eai-mcp-runtime of its namespace (spec.oidcConfigRef with the name and an audience of its own). The proxy only accepts JWTs issued by the control plane and checked against its public JWKS. The Kyverno rule mcpserver-oidc refuses any MCPServer without oidcConfigRef.

Cedar authorisation. Every instance carries its inline Cedar policy (spec.authzConfig) or a reference (spec.authzConfigRef): only the owner of the instance (fingerprint eai.io/owner) calls its tools, and the MCPToolConfig narrows the list of exposed tools. Kyverno rule mcpserver-authz.

Audit. spec.audit.enabled: true on every MCPServer (rule mcpserver-audit): the proxy writes one audit event per call, shipped to the separate index described below.

Network isolation.

• Project namespace: NetworkPolicy default-deny created by the controller and kept by the Kyverno rule generate-default-deny (synchronize: true, recreated when changed or deleted).
• Proxies (deploy/cilium/mcp-runtime.yaml, eai-mcp-runtime-proxy): ingress from the control plane only, on 8080; egress to the control plane for the JWKS path only (L7 HTTP rule), to the Kubernetes API (attach to the stdio server) and to the cluster DNS.
• MCP servers: no egress outside the CiliumNetworkPolicy inst-<10 hex> of the instance (exact domains of the server template).
• Operator: default-deny in toolhive-system, egress to the Kubernetes API and the DNS only, metrics ingress from Prometheus (observability).
• Control plane: eai-mcp-runtime-control-plane opens the egress to the proxies and the JWKS ingress from the proxies.

Pod Security. The project namespaces are restricted (enforce, audit, warn), required by the rule namespace-name-and-labels; so is toolhive-system. The mcp container of the podTemplateSpec: readOnlyRootFilesystem: true, allowPrivilegeEscalation: false, capabilities.drop: [ALL], runAsNonRoot: true; at pod level: seccompProfile: RuntimeDefault, runAsNonRoot: true, automountServiceAccountToken: false (rules mcpserver-mcp-container-*). The general policies of deploy/policies/kyverno/pod-network-security.yaml and cluster-policies.yaml also apply to every pod created.

Quotas. ResourceQuota mcp-quota and LimitRange mcp-limits in every project namespace, created by the controller: a project cannot exhaust the cluster.

Pinned images. Operator, proxy and side images by digest (table above); the image of every MCPServer must carry @sha256: (rule mcpserver-image-digest).

Kyverno, policy eai-mcp-runtime (Enforce).

Rule	Check
namespace-writers	Only system:serviceaccount:enterprise-ai:enterprise-ai-control-plane creates, changes or deletes an eai.io/mcp-runtime=true namespace
runtime-object-writers	Same restriction for the MCPServer, MCPToolConfig, MCPOIDCConfig, Secret and CiliumNetworkPolicy objects of these namespaces; the operator may only update its own resources (finalizers), the namespace controller and the garbage collector may only delete
namespace-name-and-labels, namespace-name-format, mcp-name-requires-label	Name mcp-<12 hex>, Pod Security, tenant and project labels; a namespace with that name always carries the label
mcpserver-oidc, mcpserver-authz, mcpserver-audit, mcpserver-image-digest	OIDC, Cedar, audit, digest
mcpserver-mcp-container-present, mcpserver-mcp-container-hardened, mcpserver-mcp-container-drops-all	mcp container present and hardened, no service account token
generate-default-deny	Synchronised default-deny NetworkPolicy

Test cases: deploy/policies/kyverno/tests/mcp-runtime (conformant and non conformant resources, requests of the control plane and of a cluster administrator).

RBAC scope. The operator runs with operator.rbac.scope: cluster because the project namespaces are created dynamically; the guard rails are Kyverno and Cilium. The control plane ClusterRole reads the Secrets of the whole cluster: Kyverno bounds the writes, not the reads. Keep the Kubernetes API audit log on for Secrets and watch the reads of the enterprise-ai-control-plane account outside the mcp-* namespaces.

4. Separate audit logs

• Alloy (deploy/observability/alloy/values.yaml) removes the pods of the mcp-<12 hex> namespaces from the platform log flow and reads them in a flow of their own (pods labelled eai.io/mcp-instance), labelled eai_stream=mcp-runtime-audit with the instance, the role, the tenant and the project. Kyverno reserves that name to the namespaces labelled eai.io/mcp-runtime=true, so selecting by name equals selecting by label.
• The OpenTelemetry Collector (deploy/observability/otel-collector) keeps these records out of the logs pipeline (index eai-logs-* and ClickHouse) and writes them through the logs/mcp-runtime-audit pipeline to the daily index eai-mcp-runtime-audit-yyyy.MM.dd, after the same personal data filters.
• OpenSearch (deploy/observability/opensearch): index template eai-mcp-runtime-audit, ISM policy eai-mcp-runtime-audit-retention (read only after 2 days, deleted after 400 days), installed by bootstrap.py. Read access for the role eai_mcp_runtime_audit_reader only (OpenSearch backend role eai-security-officer, as for Security Analytics); OpenSearch Dashboards and the other log readers have no access to it.
• The control plane audit trail (PostgreSQL hash chain, WORM exports) stays the reference evidence; this index serves the investigations on the instances.

5. Operations

• Management: ToolHive UI is a local desktop application for a developer workstation; there is no ToolHive web interface on the cluster side and the former Cloud UI is no longer offered. Management happens in the Navin Manager (operator state, instances per tenant and project, destruction) and in the portal (the instances of the user).
• Operator state: 2 replicas, one active (lease toolhive-operator-leader-election of the toolhive-system namespace), probes /healthz and /readyz on 8081, metrics on 8080 (ServiceMonitor toolhive-operator).
• Instances: MCPServer objects labelled app.kubernetes.io/managed-by=eai-mcp-runtime in every namespace, phase Pending, Ready, Failed, Terminating or Stopped.
• Destruction: from the Navin Manager or the portal. Deleting an MCPServer is safe: the controller revokes the temporary credentials and puts the instance to sleep; the next call of its owner recreates it. The Navin Manager also restarts an instance (annotation mcpserver.toolhive.stacklok.dev/restarted-at); its identities are listed in the ConfigMap kyverno/eai-mcp-runtime-operators (deploy/policies/kyverno/mcp-runtime-operators.yaml), the only ones allowed to delete and restart, never to change the spec.
• Emergency change: Kyverno refuses the writes of an administrator in the mcp-* namespaces. Go through the control plane; otherwise, a reviewed change of the policy in Git (Argo CD), never a manual switch off.
• Upgrade: CRDs first (toolhive-operator-crds), then the operator, with the new digests; the CRDs are kept on uninstall (crds.keep).
• Operator outage: running instances keep serving; no new instance is reconciled while no replica holds the lease.

6. No online MCP server, company catalogue in SeaweedFS

• Users never use an online MCP server. The gateway reaches internal addresses only: servers hosted by the Navin Manager (EAI_EGRESS_PRIVATE_ALLOWLIST), MCP Runtime instances, internal hosts where the company hosts the servers it develops (EAI_MCP_INTERNAL_HOSTS: exact host, host:port, *.suffix or CIDR). A public host is refused when it is registered and at every call (403 mcp_online_forbidden, audit security.egress.denied, metric eai_mcp_online_refused_total); a declared name that resolves to a public address is refused at connect time. An online preset of the Navin library (hosting: online) is installed only with the internal address where the company hosts it. The MCP registry of the Navin Manager applies the same rule and the mcp-host service starts its servers offline (npm, uv and pip without download, without proxy).
• The templates of the MCP Runtime come from the company catalogue in SeaweedFS: object EAI_MCP_CATALOG_KEY (default mcp-runtime/templates.yaml) of the platform storage profile EAI_MCP_CATALOG_STORAGE_PROFILE, read again every EAI_MCP_CATALOG_REFRESH_SECONDS (300). An image must come from an internal registry (EAI_MCP_RUNTIME_IMAGE_REGISTRIES, default <harbor-host>) and be pinned by digest. A version that cannot be read or is invalid never removes the templates in use: the last valid version keeps serving, marked stale (eai_mcp_catalog_stale, alert EaiMcpCatalogStale). In production, without a profile, creation answers 503 mcp_catalog_not_configured. Publication by the platform team:

cd services/control-plane
.venv/bin/python -m eai.domains.mcp_runtime.catalog_cli validate templates.yaml
.venv/bin/python -m eai.domains.mcp_runtime.catalog_cli publish templates.yaml

7. Lifecycle: destroyed on failure, woken on call

The controller loop runs in the API pods of the control plane, the only pods allowed to call the Kubernetes API, one replica at a time (PostgreSQL advisory lock), every EAI_MCP_RUNTIME_REAPER_SECONDS (60).

Situation	Effect
Idle past the policy idle time (900 s by default), end of the lifetime of a generation, stopped by the owner (POST .../stop)	Sleep: credentials revoked, ToolHive resources deleted, gateway entry kept
Call of a sleeping instance by its owner or one of its agents	Recreated: template read again from the catalogue, policy checked again, new generation and new credentials; the call waits up to EAI_MCP_RUNTIME_WAKE_TIMEOUT_SECONDS (90)
MCPServer Failed, a *Validated condition False, pods in CrashLoopBackOff, image not found, container that does not start, MCPServer never picked up by the operator, start timeout, two failed /health probes of the proxy, proxy not answering a call	Destroyed at once, then asleep; the next call recreates it. A call whose request never reached the server, or without side effect, is retried once on the recreated instance
EAI_MCP_RUNTIME_FAILURE_LIMIT failures (3) within EAI_MCP_RUNTIME_FAILURE_WINDOW_SECONDS (900), configuration refused by ToolHive, template removed from the catalogue, project policy	Failed: calls refused (mcp_instance_failed) until the owner starts it again (POST .../start)

8. Traceability by default

• Every project namespace gets the MCPTelemetryConfig eai-mcp-telemetry: traces and metrics of the ToolHive proxies over OTLP/HTTP with TLS to the platform collector (EAI_MCP_RUNTIME_OTEL_ENDPOINT, sampling EAI_MCP_RUNTIME_OTEL_SAMPLING_RATE, 1.0 by default), the internal CA copied from EAI_MCP_RUNTIME_OTEL_CA_FILE (Secret eai-internal-ca of the release namespace, mounted by the chart). Kyverno refuses an MCPServer without telemetryConfigRef.
• The gateway propagates the W3C context (traceparent) to the proxy: the trace of a call continues in ToolHive. Every call is counted (eai_mcp_calls_total by target, method and outcome) and carried by the span eai.mcp.proxy.
• The lifecycle journal eai.mcp_runtime_events carries the trace_id of the request and the generation of the instance; the metrics eai_mcp_runtime_instances, eai_mcp_runtime_lifecycle_total, eai_mcp_runtime_failures_total and eai_mcp_runtime_wake_seconds follow the instances; the pod logs go to the separate index eai-mcp-runtime-audit.
• Alerts (group eai-mcp-runtime): EaiMcpRuntimeFailures, EaiMcpRuntimeInstancesFailed, EaiMcpCatalogStale, EaiMcpOnlineServerRefused, EaiMcpCallErrorRatio.

9. What is taken from the ToolHive guide

Section of the run-mcp-k8s guide	In the platform
Create an MCP server, stdio and Streamable HTTP transports	MCPServer written by the controller from the catalogue template (transport, mcpPort)
Automatic RBAC management, existing ServiceAccount	Service account of each proxy, never a token mounted in the MCP container
Customise the pod, server with secrets, volume	Hardened podTemplateSpec (non root, read only root, no capability, seccomp), Secret of the instance, bounded /tmp
Server status, troubleshooting, status conditions reference	Phase, *Validated conditions, pods and probes read by the controller; reason and message shown in the Manager and the portal
Restart a server	Restart action of the Navin Manager (annotation restarted-at)
Groups and aggregation, horizontal scaling, session affinity and routing, connection draining	Not taken: one instance per user and per project, one replica, no aggregation across users (isolation)

10. Offline checks

helm lint deploy/helm/enterprise-ai --set mcpRuntime.enabled=true
/tmp/toolchains/kyverno test deploy/policies/kyverno/tests
cd services/control-plane && .venv/bin/python -m pytest -q -p no:cacheprovider tests/test_platform_mcp_runtime.py

tests/test_platform_mcp_runtime.py reads the files, renders both ToolHive charts (helm template, network access to ghcr.io) and the platform chart with and without mcpRuntime.enabled, then runs kyverno test and kyverno apply; it fails when helm or kyverno is missing.