AI Agent Governance

Many AI governance practices were designed around outputs that a person reviews before acting. An agent can also choose steps, call tools, read and write data, and trigger actions in other systems. Anthropic's engineering guidance draws a useful line between workflows, where models and tools follow predefined code paths, and agents, where the model directs its own process and tool use. The more the model directs, the more governance has to move from reviewing outputs to bounding actions.

Agentic AI governance therefore answers practical questions: who owns each agent, what it is allowed to touch, which actions need a human decision, what evidence is recorded, how failures are contained, and who approves changes to prompts, tools, permissions or memory. It covers the whole AI agent lifecycle, from proposal and design through operation, evaluation, change and retirement.

Governance is not the same as compliance paperwork, and it is not a product. Frameworks such as the NIST AI Risk Management Framework offer a voluntary structure for managing AI risk; NIST describes it as voluntary guidance rather than a certification. Organizations still have to translate any framework into operating decisions for their own agents, data and people.

Model-review practices such as data quality checks, bias testing, model approval and periodic review still matter. On their own, however, output-focused and model-review practices are insufficient for agentic systems, because several of their working assumptions do not hold.

• Behavior depends on runtime context. The same agent can take different paths depending on retrieved memory, tool responses and instructions from other agents.
• Risk sits in actions, not only outputs. Sending a payment, changing a record or emailing a customer has consequences a text answer does not.
• Permissions accumulate quietly. Agents are often given broad credentials during prototyping that never get narrowed.
• Configuration changes constantly. Prompt edits, new tools and memory updates can change behavior without any model retraining event that would trigger review.
• Accountability blurs across handoffs. When an outcome is produced by several agents and people, it is easy for no one to own it.

The OWASP GenAI Security Project describes this class of problem as excessive agency: damaging actions made possible by excessive functionality, excessive permissions or excessive autonomy. Its mitigations are operational rather than documentary: give agents the minimum tools and permissions they need, enforce authorization in downstream systems instead of trusting the model, require human approval for high-impact actions, and log and rate-limit activity.

Multi-agent governance introduces problems that do not exist when a single assistant serves a single person. These are the ones that tend to surface first.

Delegation chains

A planning agent asks a research agent for data, which asks a drafting agent for a summary, which is then sent by an execution agent. Each step may be reasonable, but the combined chain can perform an action nobody explicitly approved. Governance needs a rule that delegated work never carries more authority than the original request allowed.

Approval and evidence handoffs

When work moves from one agent to another, the receiving agent needs to know what was already checked, by whom, and what is still pending. Without an explicit handoff record, approvals are either repeated wastefully or, worse, assumed. A simple pattern is to pass a handoff note with every transfer: the request, the authority it came with, the evidence gathered, approvals obtained, and open questions.

Shared state and conflicting writes

Agents that read and write the same memory can overwrite each other or build on an unverified claim. One agent's guess becomes another agent's fact.

Emergent load and cost

Agents that call other agents can loop or fan out. Anthropic notes that agentic systems trade latency and cost for task performance, which is why it recommends starting with the simplest solution that works. Governance should include budgets and stop conditions, not only content rules.

Every agent should have a named human owner who is accountable for its purpose, permissions and performance. Ownership should be recorded somewhere people can find it, alongside the agent's scope, the systems it can reach and its escalation contact.

Ownership of an agent is not the same as ownership of an outcome. In a coordinated system, the business outcome usually belongs to the person who requested the work or the process owner, while each agent owner answers for their agent's contribution. Making both explicit avoids the common failure where every participant believes someone else was responsible for the final check.

• Agent owner: accountable for purpose, configuration, permissions and retirement.
• Process or outcome owner: accountable for the result delivered to the business or customer.
• Approver: a person with authority to authorize specific high-impact actions.
• Reviewer: checks evidence, evaluations and incidents, and can pause an agent.

Agent permissions should be designed the same way you would design access for a new contractor: the minimum required, scoped to a purpose, time-limited where possible, and revocable. Several practical rules follow from the OWASP guidance on excessive agency.

• Give each agent its own identity rather than a shared service account, so actions can be attributed.
• Prefer narrow, purpose-built tools over general ones. A tool that can update one field is easier to govern than one that can run arbitrary queries.
• Separate read from write and write from irreversible action.
• Enforce authorization in the downstream system. The agent asking politely is not a control.
• Act with the requesting user's permissions where possible, so an agent cannot reach data the requester could not.
• Review permissions on a schedule and whenever the agent's purpose changes.

Tools are where agent decisions turn into effects. Keep an inventory of tools available to each agent, with a short description of what each can change. Classify tools by impact: read-only, reversible write, external communication, financial or legal effect, and irreversible change. Higher classes warrant tighter limits, approval requirements and stronger logging.

New tools should go through the same change control as new permissions. Rate limits and spend caps belong on the tool, not only in the prompt. When tool responses come from outside the organization, treat them as untrusted input that could contain instructions.

Shared memory lets an organization stop relearning the same lessons, but it is also a channel through which errors and sensitive data spread. Govern it as an organizational asset.

• Record provenance: which agent or person wrote an entry, when, and from what evidence.
• Distinguish verified facts, working assumptions and opinions.
• Scope access so agents only retrieve what their purpose requires.
• Set retention and correction rules, including how a wrong entry is removed and who is notified.
• Keep personal and confidential data out of general memory unless there is a clear basis and control.

This is the Memory capability in the Swarm Loop: context that improves future work only if it can be trusted and audited.

Human-in-the-loop is often described as a single switch. In practice it is a set of decisions about where human judgment adds the most value. Approval points should be placed on actions, not on every output, and should be proportionate to impact and reversibility.

• Approve before: irreversible, external, financial, legal or safety-relevant actions.
• Review after: reversible internal changes, sampled for quality.
• Escalate when: confidence is low, inputs conflict, policy is unclear, or the request falls outside the agent's scope.

Approvers need the evidence to decide, not just a yes or no button. A good approval request shows what will happen, why, what evidence supports it and what the alternatives are. Watch for approval fatigue: if people approve everything without reading, the control has failed and the threshold needs redesign.

AI agent oversight depends on being able to reconstruct what happened. For each consequential task, the record should show the request, the agents involved, tools called, data retrieved, approvals given, outputs produced and final effect. That trail serves incident response, internal review and learning.

Monitoring should watch for signals that matter operationally: unusual tool volume, denied actions, escalation rates, repeated retries, cost spikes and drift in evaluation results. Dashboards are useful only if someone owns acting on them.

Evaluate agents against the tasks they actually perform, including edge cases and adversarial inputs, before deployment and after meaningful changes. Evaluate coordinated workflows end to end as well; agents that pass individually can still fail together.

Plan for failure in advance. Each agent and workflow should have a defined way to pause, roll back recent actions where possible, notify owners and fall back to a human process. Treat incidents as information: record what happened, what control should have caught it, and what will change.

Agentic systems should improve, but improvement without control is just drift. Any change to prompts, models, tools, permissions, routing rules or memory structure should have an owner, a reason, a test, an approval proportionate to its risk and a way to reverse it. Keep a change history next to the agent's ownership record.

Learning loops are where governance pays off. When evaluations, incidents and approver feedback are reviewed regularly, the organization can widen autonomy where evidence supports it and narrow it where it does not.

A regional services company wants agents to help handle supplier invoice disputes. An intake agent classifies incoming disputes, a research agent gathers contract terms and payment history, a drafting agent prepares a response, and an execution agent can issue credit notes in the finance system. A finance analyst owns the process.

Notice what the matrix makes explicit: no agent can both decide and execute a credit, approvals travel with the work as evidence, and the boundary where authority changes is a number someone can review. If the analyst later finds drafts consistently accurate, the outcome owner can propose reducing review to sampling, through change control, with the evidence recorded.

The Swarm Loop describes eight organizational capabilities: Governance, Planning, Memory, Routing, Execution, Observation, Learning and Simulation. Governance appears first because it sets the boundaries the other capabilities operate within, but it depends on all of them.

• Governance defines ownership, permissions and approval rules; the Govern stage guide covers the related checks.
• Planning and Routing decide which agent or person receives work, and must respect those rules.
• Memory carries context and evidence between participants, with provenance.
• Execution performs actions through scoped tools.
• Observation records what happened so it can be audited.
• Learning turns evaluations and incidents into controlled change.
• Simulation lets teams test a new rule or agent before it touches live work.

For how a coordination layer can enforce these rules across agents, see the AI Control Plane. For how governance maturity develops over time, see the AI Agent Maturity Model, and for the organizational structure in which these controls sit, see Agentic Organization.

Use this as a starting review for any agent or multi-agent workflow. It is a practical prompt for discussion, not a compliance standard.

1. 01Each agent has a named owner, documented purpose and listed systems it can reach.
2. 02Each workflow has an outcome owner distinct from agent owners.
3. 03Agents have individual identities and least-privilege permissions, enforced downstream.
4. 04Tools are inventoried and classified by impact, with limits on the tool itself.
5. 05Delegated work cannot exceed the authority of the original request.
6. 06Handoffs carry the request, authority, evidence, approvals and open questions.
7. 07High-impact and irreversible actions require human approval with evidence.
8. 08Escalation triggers are defined and routed to someone with authority.
9. 09Shared memory records provenance, distinguishes facts from assumptions and has correction rules.
10. 10Consequential tasks leave an audit trail that can be reconstructed.
11. 11Agents and workflows are evaluated before launch and after meaningful change.
12. 12Every agent has a pause, rollback or fallback path.
13. 13Changes to prompts, tools, models, permissions and routing go through change control.
14. 14Incidents and approver feedback are reviewed regularly and drive changes.

To get a directional view of where your organization stands across these areas, the Swarm Readiness Assessment offers a self-rated starting point. Its results reflect your own answers and are not an audit.