Skip to main content

ADR-0170: Internal MCP servers only, company MCP catalogue in SeaweedFS, destroyed on failure, woken on call and traceability by default

Numbered ADR-0168 when it was written, a number already taken by ADR-0168 on the inference engines. The applied migration sql/postgres/V201__mcp_runtime_lifecycle.sql (immutable file, its checksum is verified) and the task history of the project keep the former number.

Context​

ADR-0167 set the MCP Runtime on the ToolHive Kubernetes Operator. The user set three more requirements. Users never use an online MCP server: they host their servers internally or take those of the company catalogue, stored in SeaweedFS (installed with the Navin Manager). An instance that does not work is destroyed as soon as possible and recreated when the user calls it. All MCP activity is traced by default and centralised with the observability services of the platform (OpenTelemetry, metrics, logs). The ToolHive guide "Run MCP servers in Kubernetes" describes the phases and conditions of an MCPServer, telemetry (MCPTelemetryConfig), restart by annotation and troubleshooting; the operator never destroys nor recreates an instance by itself and has no sleep.

Decision​

  1. No online MCP server. The MCP Gateway reaches internal addresses only: servers hosted by the Navin Manager (EAI_EGRESS_PRIVATE_ALLOWLIST), MCP Runtime instances, internal hosts of EAI_MCP_INTERNAL_HOSTS. A public host is refused when it is registered and at every call (403 mcp_online_forbidden, audit security.egress.denied, metric eai_mcp_online_refused_total); a declared name that resolves to a public address is refused at connect time. An online preset of the Navin library (hosting: online) is installed only with the internal address where the company hosts it. The MCP registry of the Navin Manager applies the same rule and the mcp-host service starts its servers offline (npm, uv and pip without download, without proxy).
  2. Company MCP catalogue in SeaweedFS. The templates of the MCP Runtime are read from the object EAI_MCP_CATALOG_KEY of the platform storage profile EAI_MCP_CATALOG_STORAGE_PROFILE, published by the platform team with python -m eai.domains.mcp_runtime.catalog_cli publish and read again every EAI_MCP_CATALOG_REFRESH_SECONDS. An image must come from an internal registry (EAI_MCP_RUNTIME_IMAGE_REGISTRIES) and be pinned by digest. A version that cannot be read or is invalid never removes the templates in use: the last valid version keeps serving, marked stale. In production, without a profile, creation answers 503 mcp_catalog_not_configured; the starting templates of catalog/mcp-runtime/templates.yaml serve in dev and test only.
  3. Lifecycle (migration V201). An instance that does not work is destroyed at once: MCPServer Failed, a *Validated condition False, pods in CrashLoopBackOff or without image, MCPServer never picked up by the operator, start timeout, two failed /health probes of the proxy, proxy not answering a call. Its credentials are revoked and its resources deleted; it sleeps and keeps its gateway entry. An instance that is idle, at the end of the lifetime of a generation or stopped by its owner (POST .../stop) sleeps the same way. The next call of its owner or of one of its agents recreates it (new generation, new credentials, template read again from the catalogue and policy checked again) and waits until it is ready; a request that never reached the server, or without side effect, is retried once. After EAI_MCP_RUNTIME_FAILURE_LIMIT failures within the window, or a failure that recreating does not cure, the instance stays failed until it is started explicitly (POST .../start).
  4. Controller loop in the API pods. Only the API pods of the control plane may call the Kubernetes API (service account token, RBAC of the chart, Cilium policy deny-kube-apiserver): the loop runs in these pods, one replica at a time (PostgreSQL advisory lock), every EAI_MCP_RUNTIME_REAPER_SECONDS. It replaces the worker job mcp_runtime_reaper planned by ADR-0167.
  5. Traceability by default. Every project namespace gets the MCPTelemetryConfig eai-mcp-telemetry (traces and metrics of the ToolHive proxies over OTLP/HTTP with TLS to the platform collector, internal CA copied) and Kyverno refuses an MCPServer without telemetryConfigRef. The gateway propagates the W3C context to the proxy and counts every call (eai_mcp_calls_total); the lifecycle journal carries the trace_id of the request and the generation; the metrics eai_mcp_runtime_* and eai_mcp_catalog_* feed five alerts; the pod logs go to the separate index eai-mcp-runtime-audit.
  6. Operations from the Navin Manager. The identities of the ConfigMap kyverno/eai-mcp-runtime-operators may delete and restart an instance (annotation mcpserver.toolhive.stacklok.dev/restarted-at), never change its spec.

Alternatives considered​

  • Allow online MCP servers with a host list: the data of a project would leave the installation, against the requirement of the user.
  • Restart a failing instance in place (ToolHive annotation): its temporary credentials and resources would stay active during the failure; destroying it and recreating it on call revokes them at once.
  • Keep the loop in the worker: the worker cannot call the Kubernetes API.
  • Read the templates from a Git repository: the user chose SeaweedFS, installed with the Navin Manager.

Consequences​

A failure never leaves credentials active and the user gets the instance back by calling it. The MCP Runtime needs a platform storage profile to SeaweedFS and the internal CA (Secret eai-internal-ca) in the release namespace. The ToolHive features not taken (groups and aggregation, horizontal scaling, session affinity) stay out of scope: one instance per user and per project, one replica.

Evidence​

services/control-plane/tests/test_mcp_internal_only.py, services/control-plane/tests/test_mcp_catalog_seaweedfs.py, services/control-plane/tests/test_mcp_runtime.py, services/control-plane/tests/test_platform_mcp_runtime.py, navin-manager/server/tests/test_mcp_runtime_k8s.py, navin-manager/mcp-host/tests/test_host.py, portal/tests/navin-library.test.ts, deploy/policies/kyverno/mcp-runtime.yaml.