Kubernetes mode

navin-manager manages hosts through its agent, and Kubernetes clusters through the Navin Platform Operator. A cluster of kind kubernetes is a bring your own cluster (kubeadm, RKE2, k3s, OpenShift) or a cloud managed one (EKS, AKS, GKE). The manager discovers it and writes the desired state of catalog components as ManagedComponent resources (platform.navin.ai/v1alpha1); the operator renders the chart of each component, applies it with server side apply and reports the state read from the real workloads. The console and the API show that confirmed state, never the requested action.

Prerequisites

Item	Requirement
Kubernetes	1.29 or later (CEL validation rules of the CRD, GA in 1.29); tested with the envtest API server 1.37.0
CRD	operator/config/crd/platform.navin.ai_managedcomponents.yaml installed by the cluster administrator
Operator	operator/config/manager/manager.yaml, service account and RBAC of operator/config/rbac
Charts	local charts mounted read only in --charts-dir, or OCI charts pinned by @sha256: digest
Manager access	a kubeconfig of the user navin-manager with the rights of operator/config/rbac/manager_access.yaml
Optional	metrics-server (HPA on CPU), KEDA (ScaledObject), a PDB aware drain procedure

The kubeconfig sent to the manager must be self contained: certificate-authority-data, a client certificate and key or a token, embedded. The manager refuses exec plugins (aws eks get-token, kubelogin), auth providers, file references, proxies, impersonation, insecure-skip-tls-verify and plain http.

Registration and kubeconfig guard

POST /api/v1/clusters/{cluster_id}/kubernetes
{"api_server": "https://api.k8s.bank.example:6443", "kubeconfig": "<yaml>", "namespace": "navin-platform"}

The API server is registered explicitly. The current context of the kubeconfig must point at exactly this address (same scheme, host, port and path), otherwise the request fails with 422 k8s_server_mismatch. The guard runs again at every use of the stored kubeconfig: changing the registered address without a matching kubeconfig is refused too. The kubeconfig is sealed with AES-256-GCM like every secret and is read back only as {"secret": true, "set": true}. The manager never reads KUBECONFIG, ~/.kube/config nor an in-cluster service account. A cluster that already manages hosts cannot become a Kubernetes target (409 k8s_cluster_has_hosts).

Discovery

POST /api/v1/clusters/{cluster_id}/kubernetes/discovery reads, with read only calls:

• the version of the API server and the distribution (eks, gke, aks, openshift, k3s, rke2, kubernetes), from which the mode is set: managed for a cloud control plane, else byo;
• every node: readiness, roles, taints, provider id, instance type, zone, OS image, kernel, container runtime, kubelet version, capacity and allocatable CPU (millicores), memory, ephemeral storage, pods and GPUs (nvidia.com/gpu, amd.com/gpu), plus the totals of the cluster;
• storage classes (provisioner, default, reclaim policy, binding mode, expansion) and ingress classes;
• CoreDNS (image and version, replicas);
• the platform add-ons found through the API groups their CRDs serve: cert-manager, Cilium, Kyverno, Argo CD, KEDA, the Navin Platform Operator, Prometheus Operator, Gateway API, volume snapshots, NVIDIA GPU Operator, OpenShift.

The last inventory is kept and returned by GET .../kubernetes/discovery.

Components and operations

Action	Request	Effect on the cluster
install	POST .../kubernetes/components {service, source, values, secret_refs, replicas, pdb, autoscaling}	creates the ManagedComponent
start, stop	POST .../components/{name}/actions {action: start or stop}	desiredState Running or Stopped (stop scales every workload to zero)
restart	{action: restart} or rolling_restart	new pod template annotation, rolling update within the disruption budget
scale	{action: scale, replicas} or {action: scale, autoscaling: {min_replicas, max_replicas, target_cpu_utilization_percentage, engine}}	replicas, HorizontalPodAutoscaler, or KEDA ScaledObject when the KEDA CRD exists
maintenance	{action: maintenance, enabled}	blocks stop, restart, scale in, configuration and removal
configure	PATCH .../components/{name} {version, source, values, secret_refs, pdb}	new chart inputs
remove	DELETE .../components/{name}	the operator deletes every object it applied, then the resource

Values never carry a secret: a key such as password, token or apiKey with a value is refused (422 k8s_values_secret) by the manager and by the operator. Secrets stay Kubernetes Secrets, referenced by secret_refs (secret_name, key, value_path of the chart value receiving the Secret name).

The state of a component has four separate dimensions:

Field	Values	Meaning
state	installing, starting, started, stopping, stopped, failed, uninstalling	observed from the Deployment and StatefulSet status (generations, replicas, revisions, conditions)
health	ok, warning, critical, unknown	available pods against desired pods
operation_state	running, succeeded, failed, blocked, idle	the last operation asked through the spec
security_state	compliant, non_compliant, unknown	inline secret or forbidden object rendered by the chart

converged is true only when the operator observed the latest generation and the observed state equals the desired state. A blocked stop in maintenance leaves state: started and health: ok: an administrative stop is not a security suspension and a blocked operation is not a health problem.

Protections

• PodDisruptionBudget: pdb.min_available or pdb.max_unavailable creates one PDB per workload. The Deployment rolling update never goes below it (maxUnavailable computed from the budget, maxSurge 1). A rolling restart waits while the live workload has fewer available pods than the budget requires (operation_state: blocked, reason DisruptionBudgetExhausted) and resumes by itself. A replica count below min_available is refused.
• Maintenance: disruptive operations are refused by the manager (409 k8s_maintenance) and by the operator, whoever writes the resource. Scaling out stays allowed.
• Charts may render only namespaced ConfigMap, Service, ServiceAccount, PersistentVolumeClaim, Deployment, StatefulSet, NetworkPolicy, Ingress, PodDisruptionBudget, HorizontalPodAutoscaler and ScaledObject objects, in the namespace of the component. RBAC objects, Secrets and cluster scoped objects are refused.
• Node drain stays an operation of the cluster administrator: the eviction API honours the PDBs the operator creates.

Operator RBAC (least privilege)

operator/config/rbac/role.yaml is generated from the RBAC markers of the controller. It is a ClusterRole bound with one RoleBinding per managed namespace (role_binding.yaml), never a ClusterRoleBinding, and the operator watches only the namespaces given to --namespaces.

Resources	Verbs
managedcomponents, status, finalizers	get, list, watch, update, patch (status and finalizers: update)
deployments, statefulsets, configmaps, services, serviceaccounts, persistentvolumeclaims	get, list, watch, create, update, patch, delete
networkpolicies, ingresses, poddisruptionbudgets, horizontalpodautoscalers, scaledobjects	get, list, watch, create, update, patch, delete
secrets	get (existence of a referenced Secret and key, never list nor watch)
leases	get, create, update (leader election)

The operator runs non root with a read only root file system, no privilege escalation and every capability dropped. Outside a cluster it accepts only an explicit --kubeconfig together with --api-server, with the same guard as the manager.

What the manager does and never does on a bring your own cluster

It does: read the version, nodes, storage classes, ingress classes, the CoreDNS Deployment and the API groups; create, patch and delete ManagedComponent resources in the registered namespace; read their status.

It never does: create or delete namespaces, nodes, CRDs, RBAC objects or Secrets; read Secret values; cordon, drain or evict; install the operator or the CRD (the cluster administrator does, through GitOps); run a command on a node; use a credential other than the registered kubeconfig. Unregistering the target or deleting the manager cluster leaves the workloads running: remove the components first.

Tests

The operator tests (cd navin-manager/operator && go test ./...) and the server tests (tests/test_kubernetes_*.py) start a kube-apiserver and an etcd with controller-runtime envtest, pinned to Kubernetes 1.37.0 in /tmp/envtest, never an existing cluster. envtest has no kubelet and no controller manager: the tests write the Deployment and StatefulSet status those controllers would write and check that the observed state converges. A missing Go toolchain or envtest binary fails the tests.

/tmp/gopath/bin/setup-envtest use 1.37.0 --bin-dir /tmp/envtest
cd navin-manager/operator && go vet ./... && go test ./...
cd navin-manager/server && .venv/bin/python -m pytest -q tests/test_kubernetes_unit.py tests/test_kubernetes_api.py tests/test_kubernetes_operator.py

MCP runtime (ToolHive Kubernetes Operator)

The manager installs the ToolHive Kubernetes Operator (Stacklok, Apache-2.0) and follows the ephemeral MCP instances the lifecycle controller of the Control Plane creates through it (page "MCP runtime" of the Cluster section; button "Install ToolHive" also in the Kubernetes page).

Route	Permission	Role
GET /api/v1/clusters/{cluster_id}/mcp-runtime	cluster:read	Deployment toolhive-operator of toolhive-system (ready, replicas, version), MCPServer CRD served or not, ToolHive ManagedComponents, managed instances of every namespace (tenant, project, template, owner fingerprint, phase, address, age, namespace) and counts by phase, tenant and project; tenant and project filter the items
POST /api/v1/clusters/{cluster_id}/mcp-runtime/install	services:install	writes the ManagedComponents toolhive-operator-crds then toolhive-operator in toolhive-system (409 mcp_runtime_installed when one exists)
DELETE /api/v1/clusters/{cluster_id}/mcp-runtime/instances/{namespace}/{name}	services:operate	deletes one managed MCPServer with a uid precondition
POST /api/v1/clusters/{cluster_id}/mcp-runtime/instances/{namespace}/{name}/restart	services:operate	restarts one managed MCPServer: merge patch of the annotation mcpserver.toolhive.stacklok.dev/restarted-at to a new RFC 3339 timestamp (rolling restart, uid precondition, audited)

Pinned charts (version 0.59.0, digests of the OCI manifests read on ghcr.io):

• oci://ghcr.io/stacklok/toolhive/toolhive-operator-crds@sha256:b7694eb741a04043818978c34b0136cf8cef30329a95ae16de518bd02daaf620
• oci://ghcr.io/stacklok/toolhive/toolhive-operator@sha256:0920e45a4d371a6a9096c0019afea688e3aecbd1167182f5dacfe86b37605449

Values of the operator: 2 replicas (leader election keeps one active controller), operator.rbac.scope: cluster, no experimental feature, no autoscaling, a disruption budget of one pod, and the restricted security context of the chart written explicitly (non root, read only root file system, no privilege escalation, every capability dropped, seccomp RuntimeDefault).

Only MCPServer objects labelled app.kubernetes.io/managed-by=eai-mcp-runtime in a namespace mcp-<12 hex> labelled eai.io/mcp-runtime=true are listed or deleted. Any other object answers 404 mcp_runtime_instance_not_found, exactly like a missing one, and the refusal is audited (mcp_runtime.instance.destroy, outcome denied, with the reason). The owner is shown only as the 16 hex fingerprint of eai.io/owner, never as an identity. Deleting a managed MCPServer is safe: the Control Plane controller then revokes the temporary credentials and closes the instance.

Rights the Kubernetes identity of the manager needs (listed in preset.required_rights; a missing right answers 502 mcp_runtime_forbidden with the verb and the resource): get of the Deployment toolhive-operator of toolhive-system, get and list of namespaces, get, list and delete of mcpservers.toolhive.stacklok.dev, and the ManagedComponents of toolhive-system. The import identity (navin-import) does not grant them by default.

Limit: the Navin Platform Operator renders only the namespaced kinds of its allow list (operator/internal/controller/workloads.go). The ToolHive charts hold CustomResourceDefinitions and cluster RBAC objects, outside that list: until the operator accepts them, ToolHive itself is installed by the platform deployment and the manager follows it.

cd navin-manager/server && .venv/bin/python -m pytest -q tests/test_mcp_runtime_k8s.py
cd navin-manager/ui && npx vitest run src/test/mcp-runtime.test.tsx