- 01
Governing an agent means governing its actions and authority, not just the quality of its text output.
- 02
Authority limits belong in the tool layer, enforced in code — prompt instructions are guidance, not controls.
- 03
Treat all retrieved and ingested content as untrusted input; prompt injection is a supply-chain risk for anything that acts.
Why agentic systems need different governance
Governance frameworks written for predictive models assume the system produces an output that a human or a downstream process then uses. The controls follow from that assumption: validate the model, monitor drift, document the decision logic, review outcomes for bias. Agentic systems break the assumption. They select actions, invoke tools, write to systems of record, send communications, and chain steps whose intermediate outputs were never reviewed by anyone. The consequential unit is no longer a prediction but an action with an effect in the world — a refund issued, a record updated, a message sent, an entitlement granted. That shifts governance from evaluating output quality to constraining and evidencing behaviour. Three properties make this harder than it first appears. Non-determinism means the same input may produce different action sequences, so testing must be statistical and continuous rather than a one-time certification. Composition means an agent that behaves correctly in isolation may behave differently when chained with another, so the unit of governance is the workflow rather than the component. And environmental dependence means behaviour changes when the retrieved content, the tool responses or the underlying model version change, none of which are necessarily under the deploying team's control. Governance that does not account for these three properties produces documentation rather than safety.
Authority: the central control
The most important governance question for any agentic workflow is what it is permitted to do, and the answer must be expressed as enforced limits rather than intentions. Define authority along four axes. Scope: which tools and which records the workflow may reach, resolved through the caller's entitlements so an agent acting for a customer cannot read another customer's data. Value: monetary or quantitative thresholds above which a human must approve, held as configuration under change control. Irreversibility: actions that cannot be undone — external communications, payments, deletions, regulatory submissions — should default to human confirmation regardless of value until a workflow has an extended production track record. Rate: how many actions of a class the workflow may take in a window, which limits the blast radius of a malfunction from catastrophic to noticeable. Enforce all four in the tool layer, in code, on the server side. Instructions in a system prompt describing what the agent should not do are useful for shaping behaviour and worthless as controls, because they can be argued around by a sufficiently unusual input and provide no evidence to an auditor. When an enterprise can produce the authority configuration for a workflow, show that it is enforced outside the model, and demonstrate its change history, most of the remaining governance conversation becomes straightforward.
Risk tiering that keeps delivery moving
Uniform governance guarantees one of two failures: controls too heavy for low-risk work, which drives teams to build outside the process, or controls too light for high-risk work, which produces an incident. Tier by consequence and reversibility. Tier one covers internal, read-only or easily reversible workflows — knowledge assistants over internal content, drafting tools, classification with human confirmation. These need a registry entry, an evaluation set, basic monitoring and an owner, approvable in days by the delivery team against a checklist. Tier two covers customer-facing or externally visible workflows with bounded authority — status handling, servicing actions within limits, outbound notifications. These add pre-release evaluation against a golden set, security review including injection testing, defined escalation behaviour, and monitoring with alert thresholds. Tier three covers workflows affecting money, eligibility, safety, employment or regulatory positions. These require documented human decision points, disparate-outcome testing, full evidence retention, legal and compliance sign-off, and periodic re-certification. Publish the tiers with concrete examples drawn from work already in flight, name the approvers, and commit to a review turnaround. The commitment matters as much as the framework: a tiering model with an unbounded queue behaves, from the delivery team's perspective, exactly like a prohibition.
Evaluation as a release gate
Agentic workflows need evaluation that tests behaviour, not only language quality. Maintain a golden set of real cases with expected outcomes, built with the domain experts who own the process and grown from every production disagreement and human override. Test at three levels. Component: retrieval precision and recall, extraction accuracy against ground truth, classification quality. Trajectory: whether the workflow selects appropriate actions in the right order, respects authority limits, and escalates when it should — including deliberately adversarial cases designed to tempt it past a limit. Outcome: whether the end state is correct, and whether the human handoff contains what the human needs. Run the suite in the deployment pipeline so a regression blocks release, and re-run on every model version change, prompt change, tool change and significant content change, since any of them can alter behaviour. Track evaluation results over time rather than treating each run as pass or fail, because gradual degradation is the common failure mode and only a trend reveals it. Enterprises that own their evaluation harness — in their repositories, runnable by their engineers, independent of any vendor — retain the ability to change models and platforms. Those that rely on a vendor's evaluation tooling have made a switching decision without noticing.
Security: injection, entitlements and the supply chain
An agent that reads content and takes actions has an attack surface that traditional application security models do not fully describe. Treat every piece of retrieved or ingested content as untrusted: documents supplied by customers or suppliers, web content, email bodies, ticket text, and even internal wiki pages that any employee can edit. Instructions embedded in that content must never be able to escalate the agent's authority, because the authority is enforced outside the model. Resolve entitlements per request from the identity of the caller rather than granting the workflow a broad service account, which is the single most consequential design decision for data exposure. Constrain tool inputs with strict schemas and validate outputs before acting on them. Log every tool invocation with its parameters and result. Extend third-party risk management to model providers, orchestration platforms and any component that sees your data, covering data handling, retention, sub-processors, regional processing and incident notification. Maintain an inventory of which models and providers are in use in which workflows, so a provider incident or a deprecation notice can be assessed in hours rather than through a discovery exercise. Red-team high-tier workflows before release and periodically afterwards, with the specific goal of inducing an action beyond authority.
Monitoring, incidents and re-certification
Production governance is continuous. Monitor quality against golden sets on a schedule, input distribution shift, escalation and human override rates, authority-limit rejections, tool error rates, latency, cost per task, and safety events. Set thresholds that trigger review rather than relying on someone noticing a chart. Define what constitutes an AI incident — an action beyond authority, a materially wrong outcome reaching a customer, an unintended data exposure, sustained quality degradation — and write the response: who is notified, who can suspend the workflow, how in-flight cases are handled, what the manual fallback is, how affected parties are identified, and what evidence is preserved. Rehearse a suspension at least once, because a fallback that has never been exercised is an assumption. Re-certify workflows on a cadence set by tier, and treat a model version change, a material prompt change or a new tool as an event that triggers re-evaluation regardless of schedule. Finally, publish governance operating metrics — workflows by tier, first-time pass rate, median review turnaround, monitoring coverage, overdue re-certifications, incidents and their causes — and review them monthly. Governance that reports on itself improves; governance that only reports on others becomes an obstacle teams learn to route around.
Documenting a workflow for approval
A tier-two or tier-three submission should fit on a few pages and answer a fixed set of questions, so that teams can prepare it in hours and reviewers can assess it consistently. What business outcome does the workflow produce and who owns it. What actions can it take, on which systems and records, with what value and rate limits, and where are those limits enforced. Whose entitlements govern its data access. What is the human decision point and what does the human see when a case is routed to them. What content does it retrieve and who maintains that content. What evaluation exists, what does it currently score, and what is the release gate. What is monitored in production and what thresholds trigger review. What is the manual fallback if the workflow is suspended, and who can suspend it. What evidence is retained per action and for how long. What third parties see the data. Standardising this set does more for review throughput than any additional reviewer, because most delay comes from submissions that arrive incomplete and cycle back rather than from genuine deliberation about risk.
- Governing an agent means governing its actions and authority, not just the quality of its text output.
- Authority limits belong in the tool layer, enforced in code — prompt instructions are guidance, not controls.
- Treat all retrieved and ingested content as untrusted input; prompt injection is a supply-chain risk for anything that acts.
- Tier by consequence and reversibility so low-risk workflows are not queued behind the controls a payment agent needs.
- Define an AI incident, name who can suspend a workflow, and rehearse the suspension before you need it.
Questions leaders ask us
- How is agentic governance different from model governance?
- Model governance evaluates outputs — accuracy, bias, drift. Agentic governance constrains actions: what the system may do, to which records, up to what value, how reversibly, and how often. The central artefacts are authority limits enforced in the tool layer, trajectory-level evaluation, action-level audit logs and an incident path that can suspend a live workflow.
- Can we control agent behaviour through prompts?
- Prompts shape behaviour usefully but are not controls. They can be circumvented by unusual or adversarial input and they produce no evidence an auditor can rely on. Anything that must not happen should be impossible in the tool layer — enforced server-side, versioned as configuration, and logged when rejected.
- How do we stop governance from blocking delivery?
- Tier by consequence, automate evidence collection from the pipeline the team already runs, ship templates that pass the lowest tier by default, commit publicly to a review turnaround, and publish your own operating metrics. Teams route around governance when its cost is unpredictable, not when it is strict.
- What evidence should we retain per action?
- The input, the caller identity and resolved entitlements, the configuration and model version in force, the retrieved content, the tool calls with parameters and results, the action taken or the escalation reason, any human decision, and timestamps. Retain to the tier's requirement and make it queryable, so an audit sample takes hours rather than a project.
- Who should own agentic governance?
- Risk or compliance should own the framework, tiers and approval authority; the platform team should own the enforcement mechanisms, evaluation harness and monitoring; the business owner of each workflow owns its outcomes and holds the authority to pause it. Governance owned solely by a central function without enforcement capability becomes advisory, and advisory governance is not evidence.
Sources
- [1] AI risk management should be proportionate, continuous across the lifecycle, and evidenced through governance, mapping, measurement and management functions. AI Risk Management Framework (AI RMF 1.0) — NIST, 2023
- [2] Prompt injection is the leading security risk for applications built on large language models, particularly those with tool access. OWASP Top 10 for Large Language Model Applications — OWASP, 2025
- [3] High-risk AI systems are subject to requirements covering risk management, data governance, logging, human oversight and post-market monitoring. Regulation (EU) 2024/1689 (AI Act) — European Union, 2024