Skip to content
EngineeringTheo Lindqvist

AI Agent Governance: Which Actions Need Human Approval? Ask What You Can Actually Undo

Sort every recurring AI agent action by what it can affect and whether you can restore the prior state before harm spreads. That gives you three classes: allow with limits, require review, prohibit.

The question is whether you can still pull the action back, and whether you have checked that you can.

Your support agent has drafted a $20 refund. It has the payment tool. Should it press the button?

Require human approval when an agent action can do meaningful harm and nothing contains it in advance. Contained means a hard limit bounds the loss, or a tested undo restores the prior state before the harm reaches anyone. OWASP's guidance on excessive agency states the core plainly: "require a human to approve high-impact actions before they are taken." [3] A contained action can run alone inside its limit, even when it is final. The harder part of AI agent governance is deciding which actions count as high-impact. Two questions do most of that work for anything an agent repeats: what can it affect, and can you restore the prior state before the harm spreads? The answers put the action into one of three classes. Allow it with limits that actually stop it. Require a human to review the exact action before it runs. Or prohibit it by taking the capability away. Having the tool connected settles none of this, because access is not authority. Then name one person who can grant exceptions, and decide in advance what will make you rewrite the rules.

That is the whole method. The rest of this piece is about where the lines fall, and where they quietly move.

Why the question is on the table now

On 18 November 2026 Microsoft announced a runtime policy service for agents. Its description is blunt about where the decision lives: "Each evaluation returns a decision for the agent host to enforce." [1] The announcement lists "transaction limits" and "human-approval steps" among the business rules organizations need, and its control table includes "Refunds and transfers stay within approved limits" and "No binding customer commitment without review." [1]

That is a vendor describing its own design, so it says nothing yet about whether the approach works. But it makes a useful point by its structure: the rule about what an agent may do sits outside the agent and outside the tool. Two days earlier, Axios's CEO newsletter relayed advice from Damon Kachur, a chief executive it quoted: "Clearly define what each agent can access, what actions it can take and when a human must step in." [2] It is one executive's advice, and Axios presents no survey behind it. Still, it is the right question, and each source reviewed for this piece answers only part of it.

Does technical access mean the agent has authority?

No. An agent's permissions, the tools it is connected to, tell you what it can call. They say nothing about what the agent may do with them, for which customer, up to what amount.

The same announcement draws that line at the protocol level: "HTTP 200 means the evaluation completed, not that the action is permitted." [1] OWASP's guidance on excessive agency goes further and tells builders to "implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not." [3]

The practical consequence for a small team: write the rule per action, and enforce it somewhere the agent cannot talk its way past. A sentence in the system prompt asking the agent to be careful with refunds is a preference. A payment integration that rejects the sixth refund of the day is a policy.

Two questions that sort every action

What can it affect? Name who or what is touched: money, a customer, a record other people rely on, a running service, someone's permissions. Then name the worst credible outcome and how far it travels. NIST's AI Risk Management Framework asks organizations to identify and document the "likelihood and magnitude of each identified impact." [4] You do not need a risk register to do that for ten actions. You need ten honest sentences.

Can you restore the prior state before the harm spreads? This is where teams fool themselves, so the definition has to be strict. Undo means you have tested putting things back the way they were, and you can do it before anyone downstream acts on the change. An apology is not an undo. A database rollback cannot pull back a refund that has already reached a bank. Restoring a customer row from version history leaves last hour's emailed report exactly as it was sent.

A "no" to the second question raises the bar: the action now needs either a hard cap that bounds the loss or a human who reviews it first. Irreversibility alone does not force review. A small refund cannot be recalled, but a hard cap keeps the exposure to an amount you have already decided you can live with, so it runs inside the limit. Larger irreversible actions, and anything past the cap, go to review. A "yes" counts only when the undo is tested and finishes before anyone is affected; a rollback that works after customers have already met the problem is a "no" for this purpose. Prohibition is for actions outside the agent's job, actions that widen its own authority, and actions whose worst case you would not accept even with a human signing off.

When either answer is "I don't know", treat it as the worse case. If the action belongs in the agent's job and a human can check it, it goes to review. If not, block it until someone can answer.

Decision flow: start with an agent action; ask what it can affect and whether it can be restored before harm spreads; low exposure, a hard cap, or a tested undo that finishes before anyone is affected leads to allow with limits; meaningful impact that nothing contains, that is in scope and that a human can verify before it runs leads to require review; an action outside scope, one that expands the agent's own authority, or one whose worst case is unacceptable even with approval leads to prohibit; unknown answers route to review if checkable and in scope, otherwise block.

Two questions, three outcomes. "Unknown" is never treated as "low".

What can an AI agent do without asking, and what needs a human?

Only actions whose exposure is small or contained, by a hard cap or by a tested undo that finishes before anyone is affected. Everything else either waits for a person or should not be possible at all, and the three classes say which.

Allow with limits. The action is low in consequence, or its exposure is contained: a cap you set in advance bounds it, or a tested undo restores the prior state before anyone is affected. The agent does it without asking. The cap has to be mechanical: when it is reached, the next action fails or moves to review. A cap that exists only in the agent's instructions depends on the agent remembering it. OWASP suggests exactly this kind of damage limit for a mail-sending tool: "implementing rate limiting on the mail-sending interface." [3]

Require review. The action can do meaningful harm that neither a cap nor a fast, tested undo contains, but it is part of the agent's legitimate job and a reviewer can check it before it runs. This is human-in-the-loop review in its narrow sense, the OWASP mitigation quoted at the top: a human approves the specific action before it runs. [3] The reviewer sees the exact diff, recipient, amount and rollback status, and the approval covers that action only. If the agent changes the amount after approval, it needs a fresh one.

Prohibit. The action is out of scope, would let the agent widen its own authority, or has a worst case you would not accept even with a reviewer's sign-off. Here the right control is removal. OWASP's example is an extension that only needs to read documents but "also includes the ability to modify and delete documents." [3] Take the delete function away. A prohibited action does not become available through an exception. If a human needs it done, the human does it through their own access.

None of these buckets is new. Dataiku's governance guide, for one, describes policy boundaries as "what an agent is authorized to do, what it is prohibited from doing, and what requires human approval before proceeding." [5] What the published guidance tends to leave to you is the line itself: where a two-person team running two products should draw it on a Tuesday.

An AI agent governance matrix for a two-product team

The team below is illustrative, not a case study. A founder and one operator run a booking app and a reporting app, with agents doing support and data work across both. Every cap in the table is an example chosen to show the shape of a rule, not a recommendation and not a measured threshold.

ActionWhat it can affectReal undo?Class and boundaryBasis
Draft replies or release notes (both apps)Nothing until sentYes, discard the draftAllow. Drafts only, no send or publish accessOWASP: minimise functionality [3]
Add private routing labels to tickets (booking)Internal queue orderYes, if labels are versionedAllow, up to a daily cap, one queue. Raise the class if a label triggers a customer messageEditorial; OWASP rate limiting [3]
Issue a routine refund (booking)Customer moneyNo, once it reaches the bankAllow only matched, pre-authorised refunds to the original payment method, e.g. $20 each and $100 a day. The sixth $20 refund goes to reviewMicrosoft refund-limit control [1]; caps editorial
Pay a supplier (booking)Company moneyNoReview amount, payee, account and purpose. Prohibit any change to payout detailsEditorial, from the refund logic
Send a customer a commitment, such as a promised date or credit (booking)Customer relianceNo, a correction does not erase itReview the exact text and recipient before sendMicrosoft binding-commitment control [1]
Edit source-of-truth customer rows (reporting)Every report built on themPartly: the row restores, sent reports do notReview the diff, scope and downstream effectsOWASP database permissions [3]; editorial
Deploy (reporting)Running serviceStaging: yes, nobody outside the team sees it. Production: partly, code rolls back but users meet the bad release firstAllow to staging with fixture data and no production secrets. Review production with release diff and tested rollbackEditorial
Bulk permanent deletion (reporting)Data with no copyNoProhibit for the agent. A human runs maintenance deletesOWASP delete example [3]
Change roles, tokens or its own permissions (both)The agent's own authorityTechnically, but trust is goneProhibit. A named admin handles it separatelyOWASP least privilege [3]
Spend on metered compute or APIs (reporting)Shared budgetNo, spend is spentAllow under a hard daily and per-job budget that actually stops spend. Review upgrades and commitmentsEditorial; Microsoft transaction limits [1]
Read the "Real undo?" column against the class and the pattern is plain. Where undo is real, tested and faster than the harm, the agent acts inside a limit. Where it is not, the agent acts alone only when a hard cap bounds the loss; otherwise a human reviews. That is why the two deploys split: a bad staging release reaches only test data and rolls back unseen, while a bad production release reaches customers before any rollback finishes, so the tested rollback limits the damage without containing it.

Can a reversible action still need review?

Yes, and this is the case the matrix exists to catch. The customer-row edit is reversible in the narrow sense: version history will put the row back. But if a weekly report has already gone out built on the wrong number, restoring the row fixes the database and nothing else. The useful test is whether anything has already acted on the change.

The same applies to volume. One $20 refund inside the cap is a small, contained action. Five in a row against the same account is a pattern somebody should look at. That is why the cap routes the sixth refund to review instead of letting the agent retry in parallel. Small actions can add up to a large exposure, and a cap is how you notice.

Who grants exceptions, and when should the policy be revisited?

Pick one owner. In the example, the founder owns the policy and every exception. The operator reviews only the action classes the policy assigns to review, and cannot widen them. NIST's framework expects this kind of clarity: "organizational roles and responsibilities clearly defined, including determining the frequency of periodic review." [4] GAO's accountability framework makes governance and monitoring two of its "four complementary principles." [6] Neither tells a two-person team how to run it day to day, so here is one workable shape.

An exception record holds the action and its target or amount, the reason, who approved it, its scope, when it expires, and what happened after it ran. Two defaults keep it honest. If nobody with authority is available, the action does not run. If an approval has expired, it does not run either.

Then set the triggers for rewriting the rules. A fixed date works as a backstop; in the example it is the end of the first month. The better triggers are events: a near miss, a cap that keeps getting hit, a new integration, a new kind of customer data, a rollback that failed when you tried it, or a review queue that is growing faster than anyone clears it.

The strongest objection: this just rebuilds a human queue

It might. If every action needs approval, the agent works at human speed, and the reviewer learns to click approve without reading. No source reviewed here measures how often that happens, so treat it as a design risk to plan for rather than a known rate.

The matrix answers this in three ways. First, most of the volume should sit in "allow with limits", where caps stop the damage and a sample of actions gets checked afterwards. Second, every review is one concrete decision: this diff, this recipient, this amount, this rollback status. If the reviewer cannot verify it, the action waits. Third, review is for actions that are legitimately in scope. Anything that should never happen is removed, so it never reaches the queue.

The other objection is that least privilege and sandboxing make all this unnecessary. They remove capability the agent should not have. They cannot judge whether a refund the agent is allowed to issue should be issued to this customer, today, for this amount. OWASP lists minimising permissions and requiring human approval as separate mitigations for a reason. [3] You need both.

Where the matrix stops working: this is not compliance guidance, and regulated sectors have their own rules. Teams whose products share a budget or a database need caps that add up across products rather than per product. And if you have never tested a rollback, treat every "yes, we can undo it" as "unknown".

Sort your three most frequent actions this week

Take the three actions your agents run most often. Write down what each can affect and whether you have actually tested putting it back. Put each one in a class.

The action you cannot place is a rule your AI agent governance has not written yet. If one of yours resisted every class, reply with it. Those are the cases worth arguing about.


References

  1. Microsoft. "Introducing agent controls: An enterprise control plane for agent behavior." Published 2026-11-18. https://commandline.microsoft.com/introducing-agent-controls-an-enterprise-control-plane-for-agent-behavior/. Accessed 2026-11-19.
  2. VandeHei, Jim. "Axios CEO newsletter: Reining in AI agents." Axios. Published 2026-11-16. https://www.axios.com/2026/11/16/ai-agent-accountability-safety-workforce. Accessed 2026-11-19.
  3. OWASP Gen AI Security Project. "LLM06:2025 Excessive Agency." https://genai.owasp.org/llmrisk/llm062025-excessive-agency/. Accessed 2026-11-19.
  4. National Institute of Standards and Technology. "AI RMF Core," AI Risk Management Framework 1.0. https://airc.nist.gov/airmf-resources/airmf/5-sec-core/. Accessed 2026-11-19.
  5. Dataiku. "AI agent governance guide." https://www.dataiku.com/blog/ai-agent-governance-guide. Accessed 2026-11-19.
  6. U.S. Government Accountability Office. "Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities," GAO-21-519SP, 2021-06-30. https://www.gao.gov/products/gao-21-519sp. Accessed 2026-11-19.

Related reading

Go deeper