MCP Runtime per user and project

The MCP Runtime starts an isolated MCP instance for one user in one project, with the credentials of that user only, and destroys it when it is idle, fails or is no longer needed (ADR-0167, ADR-0170). This page explains how a developer or a user starts, calls and follows an instance, and what the platform guarantees about identity, limits and traces. The installation, the hardening and the operation of the ToolHive Operator are in MCP Runtime (operations); this page does not repeat them.

Before you begin

• The platform operator has enabled the runtime (EAI_MCP_RUNTIME_ENABLED, ToolHive Operator installed). Without it, the MCP instances tab shows "The MCP Runtime is not enabled on the platform" and creation answers 409 mcp_runtime_disabled.
• The company MCP catalogue is published in SeaweedFS. In production without it, creation answers 503 mcp_catalog_not_configured.
• Rights: the editor relation on the project (or the tenant_admin or developer role) to start an instance; tenant_admin to set the policy of the project.

How an instance is built

Element	Value
Namespace of the project	mcp-<12 hex> (sha256 of tenant and project), Pod Security restricted, ResourceQuota (EAI_MCP_RUNTIME_QUOTA_CPU, EAI_MCP_RUNTIME_QUOTA_MEMORY), LimitRange, NetworkPolicy default-deny
Instance	MCPServer inst-<10 hex> with its own audience, Cedar policy limited to the owner, its agents and the allowed tools, MCPToolConfig, Secret and CiliumNetworkPolicy of its own
Image	From the template of the company catalogue, internal registry (EAI_MCP_RUNTIME_IMAGE_REGISTRIES), pinned by digest
Credentials	none, secret_ref (personal secret of the user given by reference) or openbao_database (dynamic OpenBao lease)
Gateway entry	Server rt-<10 hex> of the project, auth.type mcp_runtime, callable by its owner and its agents only

The starting templates of catalog/mcp-runtime/templates.yaml are postgres, github, fetch and osv; they serve in dev and test only, production reads the company catalogue.

States of an instance

State	Meaning	What a call does
provisioning	ToolHive resources written, waiting for the MCPServer Ready	Waits up to EAI_MCP_RUNTIME_WAKE_TIMEOUT_SECONDS, then mcp_instance_starting
ready	Callable	Relayed to the ToolHive proxy
sleeping	Idle, end of lifetime of a generation, stopped by its owner, or destroyed after a failure: credentials revoked, resources deleted, gateway entry kept	Recreates the instance (new generation, new credentials, template read again, policy checked again) and waits
failed	EAI_MCP_RUNTIME_FAILURE_LIMIT failures within EAI_MCP_RUNTIME_FAILURE_WINDOW_SECONDS, or a failure that recreating does not cure	Refused with mcp_instance_failed until its owner starts it again
terminating, terminated	Destroyed for good; the journal is kept	Refused with mcp_instance_not_ready

Set the policy of the project

1. Write the policy (tenant_admin). PUT /v1/projects/{project_id}/mcp/policy sets the templates, hosts and database roles allowed, the idle time, the lifetime and the quotas. Empty allowed_templates allows every template; hosts and database roles must be listed to be used.

   client.mcp_runtime.put_policy("alpha", allowed_templates=["github", "postgres"],
                                 allowed_hosts=["<pg-host>"], allowed_database_roles=["readonly"],
                                 idle_timeout_seconds=900, max_lifetime_seconds=28800,
                                 max_instances_per_user=3, max_instances_per_project=20)

   Expected result: the answer holds the policy with configured: true; the change is audited as mcp.runtime.policy.

   If it fails: 422 mcp_policy_invalid lists the unknown templates, the invalid hosts and the invalid roles.

Field	Default	Bounds
allowed_templates	[] (every template)	100 names
allowed_hosts	[]	200 DNS names or *.suffix
allowed_database_roles	[]	100 names
idle_timeout_seconds	900	60 to 86400
max_lifetime_seconds	28800	300 to 86400
max_instances_per_user	3	1 to 20
max_instances_per_project	20	1 to 200

Start and call an instance

1. List the templates and the instances. GET /v1/projects/{project_id}/mcp/instances returns enabled, the policy, the state of the catalogue (source, version, stale), the templates and your instances. In the portal: Navin library > MCP instances.

   Expected result: each template shows "Allowed" or "Not allowed by the policy" and its kind of credentials.

   If it fails: "The catalogue cannot be read again" means the last valid version of the catalogue is in use.

2. Start an instance. POST /v1/projects/{project_id}/mcp/instances with the template, its parameters, a credential reference when the template needs one, the agents allowed to call it with you (20 at most), the allowed tools and an idle time (never above the idle time of the policy). In the portal: Activate on the template.

   instance = client.mcp_runtime.create_instance(
       "alpha", "github", credential_ref="openbao://users/alice/github-token",
       agents=["agent:support-bot"], allowed_tools=["search_issues"], idle_timeout_seconds=600)

   Expected result: 202 with status: provisioning; credential_active is true when a lease was issued.

   If it fails: 403 mcp_template_not_allowed, mcp_host_not_allowed or mcp_database_role_not_allowed (policy), 409 mcp_instance_quota or mcp_instance_exists, 422 mcp_credential_required, mcp_agent_invalid, mcp_tool_invalid or mcp_param_invalid, 502 mcp_credentials_unavailable (OpenBao).

3. Follow it until it is ready. GET /v1/projects/{project_id}/mcp/instances/{instance_id} reads its state again from ToolHive and returns its journal (events).

   Expected result: status: ready and a server_id; the journal holds created, credentials_issued, provisioned and ready.

   If it fails: failure_reason gives the cause (see the troubleshooting table).

4. Call it through the gateway. The entry is a regular MCP server of the project: call it with the SDK session or POST /v1/mcp/gateway/{server_id}, as the owner or as one of its agents.

   session = client.mcp.session(instance["server_id"])
   session.initialize()
   session.call_tool("search_issues", {"query": "is:open label:bug"})

   Expected result: the answer of the tool; a sleeping instance is recreated before the call.

   If it fails: the JSON-RPC error -32004 carries data.code (mcp_instance_starting, mcp_instance_busy, mcp_instance_unavailable, mcp_instance_failed, mcp_instance_not_ready) and retry_after_seconds.

Stop, start again and destroy

1. Stop it now. POST /v1/projects/{project_id}/mcp/instances/{instance_id}/stop (portal: Stop): credentials revoked, resources deleted, gateway entry kept.

   Expected result: status: sleeping, sleep_reason: user.

   If it fails: mcp_instance_busy while another lifecycle operation runs; try again.

2. Start it again. POST /v1/projects/{project_id}/mcp/instances/{instance_id}/start (portal: Start) restarts a sleeping or failed instance and resets its failure count.

   Expected result: 202, status: provisioning, a new generation.

   If it fails: the template or the policy refuses it again; the reason is in failure_reason.

3. Destroy it at the end of the task. DELETE /v1/projects/{project_id}/mcp/instances/{instance_id} (portal: Destroy): credentials revoked, resources deleted, gateway entry disabled, journal kept.

   Expected result: status: terminated and the audit event mcp.instance.destroy.

   If it fails: terminating stays while a deletion fails; the controller loop retries it every EAI_MCP_RUNTIME_REAPER_SECONDS.

Sessions

• The gateway keeps one MCP session (Mcp-Session-Id) per gateway entry. It sends initialize when the entry has no session, and once more when the server answers 404 for an expired session.
• A recreated instance is a new proxy: the gateway drops the former session and initializes again.
• When the proxy does not answer (connection refused, 502, 503, 504), the instance is destroyed at once. The call is retried once on the recreated instance when the request never reached the server or when the method has no side effect (initialize, tools/list, resources/list, resources/read, prompts/list, ping); a tools/call that may have run is never repeated and answers mcp_instance_unavailable.

Identity propagation

1. The caller authenticates to the Control Plane (Keycloak token or API key); the gateway checks the project, the access list of the entry (owner and agents) and OpenFGA can_invoke.
2. For each call, the gateway signs a short JWT with the key of the Context Authority: issuer EAI_MCP_RUNTIME_ISSUER, subject the caller, audience the instance (one audience per instance, so a token is never replayed on another one), lifetime EAI_MCP_RUNTIME_TOKEN_TTL_SECONDS (300 seconds by default).
3. The ToolHive proxy checks the token against GET /v1/mcp/runtime/jwks (public keys only), then applies the Cedar policy of the instance: only its owner and its agents call its allowed tools.
4. The MCP container gets the credentials of the owner only, written in the Secret of the instance; the lease id is sealed and cleared once revoked, a personal secret is kept as its reference only.

danger

Never share an instance by giving its credentials to another user: each user starts their own instance. An agent listed in agents acts with the credentials of the owner of the instance.

Limits

Limit	Value	Where it is set
Instances per user and per project	3 and 20 by default (only provisioning and ready count)	Policy of the project
Idle time	900 seconds by default, never above the policy	Policy, instance
Lifetime of a generation	28800 seconds by default	Policy
Wait of a call for its instance	90 seconds by default	EAI_MCP_RUNTIME_WAKE_TIMEOUT_SECONDS
Provisioning timeout	600 seconds by default	EAI_MCP_RUNTIME_PROVISIONING_TIMEOUT_SECONDS
Failures before failed	3 within 900 seconds by default	EAI_MCP_RUNTIME_FAILURE_LIMIT, EAI_MCP_RUNTIME_FAILURE_WINDOW_SECONDS
CPU and memory of a project namespace	8 CPU and 16Gi by default	EAI_MCP_RUNTIME_QUOTA_CPU, EAI_MCP_RUNTIME_QUOTA_MEMORY
CPU and memory of an instance	From its template (resources)	Company catalogue
Network of the MCP container	Hosts of its template only	CiliumNetworkPolicy of the instance

Observability of the calls

Signal	Content
Trace	Span eai.mcp.proxy of the gateway; the W3C context (traceparent) continues in the ToolHive proxy, whose traces go to the platform collector (EAI_MCP_RUNTIME_OTEL_ENDPOINT, sampling EAI_MCP_RUNTIME_OTEL_SAMPLING_RATE)
Metrics	eai_mcp_calls_total{target,method,outcome} (target instance, hosted or internal), eai_mcp_runtime_instances, eai_mcp_runtime_lifecycle_total, eai_mcp_runtime_failures_total, eai_mcp_runtime_wake_seconds
Lifecycle journal	events of GET /v1/projects/{project_id}/mcp/instances/{instance_id}: event, actor, details, generation and trace_id of the request; append only, kept after the instance
Audit	mcp.instance.create, mcp.instance.destroy, mcp.runtime.policy, and mcp.tools.call (and the other methods) for each call, in GET /v1/audit/events
Logs of the pods	Separate index eai-mcp-runtime-audit (400 days), read by the security officers

1. Find the trace of a call. Open the journal of the instance in the portal (Journal) and copy the "Trace identifier of the request" of the event.

   Expected result: the trace shows the gateway span and the spans of the ToolHive proxy.

   If it fails: an event written outside a request (controller loop) has no trace_id.

Next steps

• MCP Runtime (operations)
• MCP security
• Hosting MCP servers

Troubleshooting

Symptom	Cause	Fix
409 mcp_runtime_disabled	Runtime not enabled on the platform	Ask the operator (operations)
503 mcp_catalog_not_configured	No company catalogue in production	The platform team publishes the catalogue
failure_reason: image_pull	Image missing from the internal registry	Mirror the image with its digest
failure_reason: spec_rejected	ToolHive refused the configuration	Fix the template; start the instance again
failure_reason: provisioning_timeout or not_reconciled	Operator down or overloaded	Check the operator state in the Navin Manager
failure_reason: health or unreachable	Proxy failed its probes or did not answer	Call again: the instance is recreated
failure_reason: template_removed or policy	Template no longer in the catalogue, or no longer allowed	Change the policy or choose another template
JSON-RPC mcp_instance_failed	Too many failures in the window	Start it again with POST .../start after fixing the cause
JSON-RPC mcp_principal_not_allowed	Caller is neither the owner nor one of its agents	Start your own instance