Governing AI agents in production

1. Executive summary

An AI agent is not a chatbot. It reasons, chooses tools and acts on real systems (email, databases, CRM, ERP, code, payments), and it does so in a partly unpredictable way. That autonomy creates value, but it shifts the risk: the incident is no longer "a wrong answer", it is "an irreversible action carried out with excessive privileges".

This paper argues a simple thesis:

Trust in an agent does not come from the model; it comes from the architecture around it.

A model can be deceived, hallucinate or drift. The enterprise must therefore make its agent governable by design, through five guarantees:

The following sections detail each guarantee, the technical mechanisms, the trade-offs and a progressive rollout path.

2. Why agents change the nature of risk

2.1 From generation to action

An LLM on its own produces text; the impact of an error is bounded by the judgment of the person reading it. An agent runs a loop: perceive → plan → call a tool → observe → repeat. Every iteration can produce a side effect. Three properties make the risk specific:

• Autonomy: the sequence of actions is not written in advance.
• Power to act: tools grant read and write access to production systems.
• Untrusted input: the agent reads emails, web pages, documents and tool results, any of which may contain hostile instructions.

2.2 The lethal trifecta

The most dangerous scenario combines three ingredients in a single agent:

1. 1.access to private data;
2. 2.exposure to untrusted content;
3. 3.the ability to communicate externally (sending email, HTTP requests, public writes).

When all three are present, a single sentence hidden in a document can trigger exfiltration. The design rule: never combine all three without a compensating control (isolation, approval, output filtering).

2.3 Why the prompt is not a barrier

Writing "never do X" in a system prompt is a preference, not a control. Models are probabilistic and sensitive to rephrasing and injection. Every security control must be deterministic, external to the model and executed before the effect takes place.

3. Threat model for AI agents

3.1 Assets to protect

Customer and personal data, secrets and tokens, intellectual property, integrity of business systems, budget (token and compute costs), reputation and compliance.

3.2 Threat catalog

3.3 Working assumption

Treat the model as an untrusted component inside the perimeter. It is useful, never authoritative. Security is provided by what surrounds it.

4. Guiding principles

1. 1.Least privilege, per task and per duration: minimal rights, short-lived tokens, narrow scope.
2. 2.Enforcement outside the model: policies are evaluated by an independent engine, on every call.
3. 3.Deny by default: any tool, data or destination that is not explicitly allowed is blocked.
4. 4.Proportionate human in the loop: the higher or more irreversible the impact, the stricter the approval.
5. 5.Defense in depth: no single layer is sufficient on its own.
6. 6.Everything is traced, nothing is assumed: every decision leaves verifiable evidence.
7. 7.Reversibility and fast stop: be able to undo, suspend and isolate.
8. 8.Governance as code: policies are versioned, tested and deployed like software.
9. 9.Continuous measurement: evaluate, attack (red team) and monitor at all times.
10. 10.Multi-tenant by design: customer isolation is an architectural property, not an option.

5. Security at every layer

Agent security is built in layers. Each layer assumes that the previous one may fail.

 User / Event
        │
 [1] Identity & access ──────────── who acts, on whose behalf?
        │
 [2] Input: filtering & DLP ─────── content, sensitive data
        │
 [3] Model gateway (LLM proxy) ──── routing, quotas, secrets
        │
 [4] Agent runtime & loop ───────── limits, budget, checkpoints
        │
 [5] Tool / MCP gateway ─────────── per-call authorization
        │
 [6] Execution sandbox ──────────── process/network isolation
        │
 [7] Data & memory ──────────────── classification, encryption
        │
 [8] Output: DLP & validation ───── before any external effect
        │
 [9] Network, platform, software supply chain

5.1 Layer 1: Identity, authentication and delegation

• Human identity: SSO, MFA, roles, tenant membership.
• Machine identity: every agent has its own identity (never a shared account), with short-lived certificates or tokens.
• Delegation: the agent acts on behalf of the user. The token carries both identities (actor and subject), and the effective rights are the intersection of the agent's rights and the user's rights. This neutralizes the confused deputy.
• Propagated context: identity, tenant, session and justification travel with every call (model, tool, data, sub-agent), so that every layer enforces the same decision.

5.2 Layer 2: Input protection

• Classification of incoming content by provenance (user, tool, external document, web) and tagging as "untrusted".
• Injection detection (heuristics, classifiers, rules) handled defensively: untrusted content is kept apart from instructions and never concatenated as a command.
• Inbound DLP: detection and tokenization of personal data, secrets and regulated data before anything is sent to an external model.

5.3 Layer 3: Model gateway

A single access point to models (cloud and local) provides:

• centralized authentication, with no provider key inside the agents;
• sensitivity-based routing (classified data → local model or sovereign region);
• quotas, rate limits and cost caps;
• automatic failover between providers;
• uniform logging of prompts, responses, tokens and costs.

5.4 Layer 4: Agent runtime

• Resource limits: number of iterations, duration, tokens, tool calls, sub-agent depth.
• Loop detection: stop when actions repeat without progress.
• Durable execution and checkpoints: recovery after an incident without replaying side effects that have already happened (idempotency).
• Sub-agent control: a sub-agent never receives more rights than its parent; delegation can only attenuate privileges.

5.5 Layer 5: Tool gateway and MCP

This is the most critical layer: it is where intent becomes action.

• Tool registry: only inventoried, versioned, signed and evaluated tools can be called.
• Authorization on every call (not per session): the agent, the user, the tool, the arguments and the context are evaluated by the policy engine.
• Strict schema validation of arguments; unexpected parameters are rejected.
• Tool classification: read-only, reversible write, irreversible write, financial, external. The class determines the approval level.
• Credential broker: the agent never holds the secret; the gateway injects a short-lived token at the last moment.
• Third-party MCP servers: version and description pinning (against "rug pulls"), allowlist, network isolation, review before activation.

5.6 Layer 6: Execution sandbox

Any code generated or run by the agent executes in a disposable environment: ephemeral file system, unprivileged user, restricted system calls, allowlisted network egress, bounded CPU, memory and time, and no privilege escalation.

5.7 Layer 7: Data, memory and RAG

• Classification of data (public, internal, confidential, regulated) applied from ingestion onward.
• Document-level access control in RAG: vector search only returns what the user is entitled to see.
• Encryption at rest and in transit, with customer-managed keys where required.
• Governed memory: controlled writes, provenance, expiry, right to erasure, protection against poisoning.
• Retention: defined periods, verifiable purge.
• Source traceability (lineage): knowing which data fed which answer.

5.8 Layer 8: Outputs

• Outbound DLP: blocking or masking sensitive data.
• Destination control (allowed domains, addresses and channels) to break exfiltration paths.
• Format and content validation before any effect: an outgoing email, a database write or a payment goes through a final checkpoint.
• Neutralization of covert exfiltration channels (remote images, generated links carrying data as parameters).

5.9 Layer 9: Platform and software supply chain

• Network segmentation, Kubernetes admission policies, network policies that deny by default.
• Signed images, SBOM, vulnerability scanning, a controlled artifact registry.
• Secrets in a vault, automatic rotation, secret detection in code and logs.
• GitOps deployment, infrastructure as code that is reviewed and audited.
• Tested backups and disaster recovery.

5.10 Multimodal security

Voice, images and video open other attack vectors: instructions hidden in an image, a cloned voice, personal data in an audio stream. The same controls (DLP, classification, policies) apply to media before and after processing.

6. Governance: policies, human approvals, emergency stop

6.1 Governing means deciding beforehand and proving afterward

Governance answers one question: who is allowed to deploy which agent, with which powers, under which control, and who is accountable for it?

Governance registry

Every agent is an inventoried object with: business and technical owner, purpose, accessible data, authorized tools, risk level, model(s) used, version, status (draft, approved, in production, suspended, retired) and last evaluation. No agent goes to production outside the registry.

Lifecycle

Design → evaluation (quality, security, red team) → risk review → approval → progressive rollout → monitoring → periodic reassessment → retirement.

Agent risk classification

Criteria: data sensitivity, reversibility of actions, external exposure, autonomy, affected population. Outcome: a level (low, medium, high, critical) that automatically determines the mandatory controls.

6.2 Policies

Policy as code

Rules are expressed in a declarative language, versioned in Git, tested automatically, reviewed through pull requests and deployed like software. The policy engine is the decision point; the gateways are the enforcement points.

What a policy can express

Possible outcomes of an evaluation: allow, deny, allow with obligations (mask, log, limit), require approval.

Good practices

• Deny by default, with explicit and dated exceptions.
• Composable policies: enterprise baseline → tenant → agent. A lower layer can only add restrictions.
• Simulation mode (dry run) before activation, to measure the impact on real traffic.
• Policy regression tests and sets of abuse cases.
• Explainable decisions: every denial states the rule that was applied.

6.3 Human approvals (human-in-the-loop)

What needs approval

Approving everything kills adoption; approving nothing kills trust. Approval must be proportionate:

Designing a good approval

• Sufficient context: the approver sees the exact action, its arguments, the agent's reasoning, the sources, the estimated impact and what changes if the request is denied. A blind approval is false security.
• What is approved is what is executed: the approval is cryptographically bound to the exact arguments; any change invalidates the decision.
• The right approver: routing to the competent and legitimate person, never the user who triggered the action when separation of duties requires it.
• Deadlines and escalation: expiry, reminder, delegate, escalation; an expired request counts as a denial.
• Four-eyes principle for critical actions.
• Countering approval fatigue: batch requests, reduce noise, measure the rate of "reflex" approvals and adjust thresholds.
• Channels: portal, enterprise messaging, mobile, with strong authentication for sensitive decisions.
• Durable pause: the agent is suspended without consuming resources and resumes exactly where it stopped once the decision is made.

Graduated autonomy

Autonomy is earned with data: observation mode (suggests), assisted mode (everything is approved), supervised mode (risky actions are approved), bounded autonomous mode (after-the-fact review). Moving from one level to the next is decided on indicators (error rate, incidents, quality), never for convenience.

6.4 Emergency stop (kill switch)

Why it is essential

When an agent drifts or is compromised, every minute counts. The emergency stop is the last safety net: it must work even when everything else is degraded.

Stop levels (from narrowest to broadest)

Properties of a good kill switch

1. 1.Fast: takes effect within seconds, ideally checked on every tool call (the decision is read at the moment of the effect, not only at launch).
2. 2.Independent: does not depend on the agent or the model; it acts on the gateway and the runtime.
3. 3.Fail-safe: when in doubt (decision service down), the default behavior is to block writes, not to let them through.
4. 4.Granular: isolate without shutting everything down.
5. 5.Accessible: can be triggered by on-call staff, by automation (critical alert) and through an API, with strong authentication and logging.
6. 6.Reversible and safe to resume: controlled restart, preserved state, in-flight side effects identified.
7. 7.Tested: regular exercises ("game days") that measure the actual time to stop.

Automatic triggers

Budget overrun, detected loop, repeated attempts at forbidden access, suspected exfiltration, anomaly score, confirmed injection detection, SOC alert. A circuit breaker can switch the agent to degraded mode (read-only) before a full stop.

After the stop

Isolate and preserve the evidence, qualify the incident, assess the impact (which actions took place?), compensate for or undo the effects, fix the root cause, replay in a test environment, then resume gradually.

6.5 Roles and responsibilities

A governance body (AI committee) rules on high and critical level agents.

7. Traceability and audit evidence

7.1 Objective

For any action, be able to reconstruct the full chain: who (human, agent) asked for what, based on which information, which policy decided, who approved, which effect took place, and at what cost.

7.2 What to record

7.3 Required properties of an audit log

• Complete: no action without a trace.
• Correlated: a single trace identifier links interface, model, tool, policy and approval (end-to-end correlation).
• Tamper-evident: append-only writes, hash chaining or signatures, immutable storage (WORM), separation of rights between those who produce the logs and those who administer them.
• Reliably timestamped: synchronized time source.
• Protected: logs contain sensitive data; masking, encryption, controlled and audited access.
• Retained for a defined period: aligned with regulatory requirements and the right to erasure, keeping evidence separate from personal data where possible.
• Usable: queryable, exportable, admissible as evidence.

7.4 Replayability

Keep enough information (inputs, versions, tool results) to replay a run in an isolated environment: incident analysis, debugging, validation of a fix, demonstration to an auditor. A replay must never reproduce real external effects (simulated tools or dry-run mode).

7.5 Evidence trails for audit

Produce automatic "evidence packages": list of agents and owners, policies in force and the history of their changes, a sample of approvals, incidents and resolution times, evaluation and red team results. Compliance becomes a by-product of operations rather than a one-off project.

7.6 Cost traceability

A cost ledger attributes every token and every second of compute to a tenant, an agent, a task and a cost center: the basis for chargeback, anomaly detection and profitability management.

8. Agent observability

8.1 Beyond traditional monitoring

An agent can be "up" and still be wrong, dangerous or ruinously expensive. Observability must cover three dimensions: technical, behavioral and economic.

8.2 Signals

Distributed traces

Every run is a trace: steps, model calls, tool calls, sub-agents, waits for approval. The decision tree can be visualized and the failing step pinpointed. Recommended standard: OpenTelemetry, extended with the semantic conventions for generative AI.

Metrics

Structured logs

Standardized format, sensitivity level, correlation identifiers, automatic masking of sensitive data.

Continuous evaluations

Reference test sets replayed on every change of model, prompt, tool or policy, plus sampled evaluation in production (rule-based, model-as-judge, human review). Detection of silent regressions when a provider ships an update.

8.3 Anomaly detection and alerting

• Behavioral baseline per agent: usual tools, volumes, hours, destinations.
• Alerts on deviations: a tool never used before, unusual read volume, unknown destination, cost spike, abnormal denial rate.
• Prioritized alerts, each with an owner, a runbook and a direct link to the trace.
• Integration with the SOC (SIEM) and the incident management process.
• Automatic link from alert to action: circuit breaker, degradation, emergency stop.

8.4 Dashboards by audience

• Operations: health, queues, latencies, saturation, real-time costs.
• Security: threats, policies, incidents, coverage.
• Business: value delivered, quality, adoption.
• Executive / compliance: inventory, risk levels, evidence, trends.

8.5 Red team and adversarial testing

Observability is complemented by planned attacks: a library of injections, exfiltration scenarios, tool abuse, escalation between agents. The results feed policies, filters and evaluation sets, and are replayed in continuous integration.

8.6 Privacy of observability

Trace without exposing: masking at the source, role-based access, sampling for high volumes, separation by tenant, appropriate retention periods.

9. Scalability and resilience

9.1 Challenges specific to agents

• Long-running tasks (minutes to days) with unpredictable duration.
• Bursty load that depends heavily on rate-limited external providers.
• Expensive resources (GPUs, tokens) that must be shared fairly.
• The need for strong isolation between customers on shared infrastructure.
• State that must survive failures and deployments.

9.2 Reference architecture

Separate control plane and execution plane The control plane (identity, policies, registries, approvals, audit) is lightweight and highly available. The execution plane (runtimes, sandboxes, inference) scales horizontally, by region or by tenant. An execution failure must not compromise governance, and conversely policy enforcement must stay local and fast (a cache of signed decisions with a short expiry).

Durable execution Agents run as durable workflows: persistent state, recovery after a crash, timers, waits for approval that consume no resources, idempotent effects. This is the precondition for scaling to thousands of concurrent agents without losing a task.

Asynchrony and queues Everything that is not interactive goes through queues with priorities, back-pressure, bounded retries with backoff, and monitored dead-letter queues.

Inference gateway Load balancing across providers and local models, semantic and prefix caching, request batching, routing by cost, latency and sensitivity, automatic failover on outage or throttling.

9.3 Scaling resources

• Autoscaling on suitable metrics: queue depth, tokens per second, GPU utilization, rather than CPU alone.
• GPU pooling (partitioning, time-slicing) with per-tenant accounting.
• Hierarchical quotas: enterprise → tenant → team → agent; fairness and protection against the "noisy neighbor".
• Workload separation: interactive (latency) versus batch (throughput) on separate pools.
• Cost control: a model sized for the task (small model first, escalation to a large model), bounded context, reuse of results, budgets with cut-off.

9.4 Multi-tenancy and isolation

Isolation of data (schemas or databases, per-tenant encryption keys), of compute (namespaces, quotas, segmented network), of secrets and of logs. Automated isolation tests (cross-access attempts) run continuously. Deployment options: shared multi-tenant, dedicated cells, or customer-hosted, sovereign or offline deployment for sensitive sectors.

9.5 Resilience and continuity

• Multi-zone redundancy; multi-region for critical services.
• Explicit RPO / RTO objectives per component, tested backups, recovery exercises.
• Circuit breakers, timeouts and rate limits wherever a call crosses a boundary.
• Graceful degradation: if approval, the primary model or a tool is unavailable, the system falls back to a restricted mode (read-only, backup model, queuing) rather than failing completely or bypassing controls.
• Progressive deployments (canary, blue/green) with automatic rollback when quality or security indicators degrade.
• Chaos and load testing, including on the "policy → approval → emergency stop" chain.

9.6 Governance performance

A control that is too slow gets bypassed. Aim for a policy evaluation cost in the order of a few milliseconds, through local evaluation, rule compilation and short-lived decision caching, without ever reducing coverage.

9.7 Governing at scale

Beyond a handful of agents, you need: reusable agent templates ("approved patterns"), catalogs of validated skills and tools, inherited policies, guarded self-service for teams (built-in guardrails), and automated risk review.

10. Compliance and reference frameworks

A useful mapping: the human oversight required by the AI Act translates into approvals and the emergency stop; the logging obligation, into tamper-evident traceability; risk management, into the registry, classification and continuous evaluations; data protection, into DLP, classification, retention and sovereignty of processing.

This document is technical guidance and does not constitute legal advice; each organization must validate its obligations with its own counsel.

11. Implementation roadmap

Phase 0: Scoping (2 to 4 weeks)

Inventory of use cases and existing agents (including "shadow agents"), risk classification, appointment of owners, choice of principles.

Phase 1: Foundations (1 to 3 months)

Machine identity and delegation, a single model gateway, a secrets vault, correlated logging, a first policy engine that denies by default, an agent registry.

Phase 2: Controlling action (2 to 4 months)

Tool and MCP gateway, tool classification, sandbox, inbound and outbound DLP, approval workflow, kill switch with drills, resource limits.

Phase 3: Observability and assurance (3 to 6 months)

AI traces, metrics, continuous evaluations, anomaly detection, automated red team, dashboards, audit evidence packages.

Phase 4: Scaling up (6 to 12 months)

Durable execution, autoscaling, hierarchical quotas, multi-region, governed self-service, catalog of approved patterns, cost optimization.

Rollout principle

Start with low-risk, high-value use cases in observation mode, measure, then extend autonomy based on the evidence.

12. Steering indicators

13. Common mistakes

1. 1.Trusting the prompt to enforce the rules.
2. 2.A single, over-privileged service account for every agent.
3. 3.Approving without context (reflex clicks) or approving everything (fatigue).
4. 4.Logging without correlating: millions of lines and no usable story.
5. 5.A kill switch that is never tested, or that depends on the failing component.
6. 6.Ignoring indirect sources (documents, emails, web pages, tool outputs).
7. 7.Enabling third-party tools without review or version pinning.
8. 8.Evaluating once at launch and never measuring again.
9. 9.Separating security from the platform: controls that can be bypassed or cannot be enforced.
10. 10.Discovering the cost too late: no budget or cap per agent.
11. 11.Fail-open: letting traffic through when the decision service is unavailable.
12. 12.Treating multi-tenancy as an application layer rather than an architectural property.

14. Conclusion

AI agents are set to become full-fledged actors in information systems. The challenge is not to choose between innovation and security, but to build the framework that makes autonomy acceptable: clear identities, policies enforced outside the model, humans making the decisions where it matters, complete evidence, real-time visibility, the ability to stop everything, and an architecture that scales without diluting the controls.

The organizations that succeed will be those that treat governance not as a brake, but as the infrastructure that makes it possible to grant more autonomy, faster, with confidence.

Appendix A: Agent production readiness checklist

Identity and access

• Dedicated machine identity, short-lived tokens
• User delegation with intersection of rights
• Documented least privilege (tools, data, network)

Policies and governance

• Agent recorded in the registry, owners named
• Risk level assessed and approved
• Policies tested in simulation, deny by default
• Action classes and approval levels defined

Security

• Injection filtering and tagging of untrusted content
• Inbound and outbound DLP
• Validated tools, strict schemas, pinned MCP servers
• Sandbox with allowlisted network egress
• Secrets only through the credential broker
• Lethal trifecta analyzed and compensating controls in place

Traceability and observability

• End-to-end correlated trace
• Tamper-evident log retained according to policy
• Quality, security, cost and governance metrics
• Alerts with runbooks, SOC integration
• Evaluation sets and red team tests executed

Resilience and scale

• Limits: iterations, duration, tokens, cost
• Durable execution, tested recovery, idempotency
• Per-tenant quotas and isolation
• Degraded mode defined, fail-safe on writes
• Kill switch tested, time to stop measured

Compliance

• DPIA / impact assessment where required
• Retention periods and right to erasure addressed
• Evidence package can be generated

Appendix B: Policy example (illustrative, pseudo-language)

policy: support-agent-prod
applies_to: { agent: support-assistant, env: production }
default: deny

rules:
  - id: read-ticket
    allow: { tool: tickets.read }
    obligations: [log]

  - id: read-customer
    allow: { tool: crm.customer.read }
    when: { user.role: [support, support_lead] }
    obligations: [log, mask_pii_in_output]

  - id: send-email
    require_approval:
      tool: email.send
      approver: team_lead          # never the initiator
      expires_in: 30m              # expiry counts as a denial
      bind_arguments: true         # the approval is bound to the exact arguments
    deny_if:
      - recipient.domain not in allowed_domains
      - context.contains_untrusted_content and data.class >= confidential

  - id: refund
    require_approval:
      tool: payments.refund
      when: { amount_gt: 100 }
      approvers: 2                 # four-eyes

  - id: delete-account
    deny: { tool: accounts.delete }

limits:
  max_steps: 40
  max_runtime: 15m
  max_cost_usd: 2.00
  max_subagent_depth: 1

killswitch:
  triggers: [budget_exceeded, loop_detected, exfiltration_suspected]
  on_trigger: degrade_to_read_only_then_suspend
  on_policy_service_unavailable: block_writes   # fail-safe

Appendix C: Glossary

• Agent: a system that uses a model to plan and carry out actions through tools.
• Confused deputy: a privileged component misused by a less privileged actor.
• DLP: data loss prevention.
• HITL: human in the loop.
• Prompt injection (direct / indirect): a malicious instruction given by the user or hidden in a source the agent reads.
• Kill switch: emergency stop mechanism.
• MCP: standard protocol for connecting models to tools and data sources.
• Policy engine (PDP) / enforcement point (PEP): the components that decide / enforce.
• RAG: retrieval-augmented generation.
• RPO / RTO: maximum tolerable data loss / maximum tolerable downtime.
• Lethal trifecta: the combination of private data + untrusted content + external communication.
• WORM: write once, read many (immutable storage).

Appendix D: Implementation in Navin AI Platform