Human in the loop for AI agents: from a policy line to a proxy that holds the call

Almost every AI agent policy contains a sentence like this one: high-risk actions require human review. It is a reasonable sentence. It is also unenforceable as written, and the gap between the sentence and reality is where agent incidents live.

The problem is not the intent. It is that the sentence lives in a document, while the action lives in a tool call that an agent framework executes a few hundred milliseconds after the model proposes it. Nothing stands between the two. The agent’s own code may ask for confirmation on some paths, if a developer thought to add it, and that code changes with every sprint.

IBM’s agentic AI governance playbook puts it bluntly: the shift from predictive models to agents moves governance “from validating the answer to controlling the actions”, and authority limits have to be “enforced technically”. This article is about what that looks like in practice for the human-review requirement specifically.

What “high-risk action” actually means

Start by being concrete. When people say an action is high risk, they almost always mean one of three things:

1. A particular tool is risky in itself. Anything that deletes, pays, suppresses an alert, or messages a customer. The risk is in the verb, regardless of arguments.
2. An argument crosses a threshold. Refunds are fine up to a limit. Ticket changes are fine unless the ticket is a P1. Transfers are fine below an amount. The risk is in a value the model chose.
3. The payload contains something sensitive. Credentials, personal data, card numbers in a message body that is about to leave the building. The risk is in what the arguments carry, not which tool receives them.

Written down this way, each of these is a condition over a tool call: its name, its arguments, or the content of its arguments. That is a condition software can evaluate. The only question is where.

Where the call can be seen

An agent’s model call goes out to a provider and comes back with a response. If the model decided to act, the response contains the tool call: the tool’s name and the full arguments the model wants to pass to it. The agent framework reads that response and executes the tool.

A proxy between the agent and the provider sees exactly that response, before the framework does. It is the one place in the architecture where the proposed action is fully formed and has not yet happened. That is where human-review triggers belong.

In UrNammu the triggers are a structured list on the agent’s registry entry. Each is one of the three shapes above:

• Any call of a tool — a pattern such as payments/* or delete_*.
• An argument condition — a tool pattern, a path into the arguments (amount, payment.total, items.0.sku), a comparison and a value: refund.amount > 1000. Numeric strings are coerced, so "$1,200.50" compares as 1200.5.
• Sensitive data in the arguments — the same detector that scans responses for leakage runs over the call’s arguments, optionally limited to categories such as credentials or personal data.

The proxy evaluates every tool call the model makes against these, in streaming and non-streaming responses alike. Streaming is the awkward case: the arguments only exist once the model has finished generating them, so in enforcement mode the proxy buffers the stream, decides, and either forwards it intact or withholds it. The agent waits a little longer for a response it was going to wait for anyway.

Two modes, deliberately

A trigger can do one of two things when it matches.

Monitor records the match as a dry-run denial, raises an alert, and lets the response through. This is where every new trigger should start. It tells you how often the condition actually fires, which is almost never what the policy author guessed, and it catches a trigger written wrongly before it stops anything real.

Enforce withholds the model’s response and returns a 403 to the agent with a type of human_review_required. The agent did not get the tool call, so it cannot execute it. The loop stops. This is the sentence from the policy, finally doing something.

The right order is monitor for a week, read the matches, fix the thresholds, then enforce. Skipping the first step is how teams end up with an enforce-mode trigger on amount > 100 that holds every refund and gets switched off by Friday.

A held call needs somewhere to go

Withholding a call and returning an error is necessary but not sufficient. If the only thing that happens is an alert, the reviewer has no view of what was proposed and no way to say yes. The agent is simply broken until someone edits the trigger.

So a withheld call becomes a pending review request: the agent, the tool, the trigger that matched, the detail (amount = 5000), and the call’s actual arguments so a reviewer can judge the specific action rather than the category. Identical re-runs collapse onto the same request and count up, which is itself a signal: an agent retrying a held call forty times is telling you something about its retry logic.

The reviewer can approve it one of two ways, or reject it:

Decision	What it allows
Approve this exact call	The same tool with the same arguments, once, within a day
Approve any matching call	Any call matching this trigger, a handful of times, within the hour
Reject	Nothing; the call stays held and a retry is held again

An approval is a waiver, not a replay. Nothing is sent on the agent’s behalf. The agent, or whoever operates it, re-runs the call; the proxy recognises the waiver, lets the matching call through, and consumes one use. The distinction matters: the reviewer approved an action the agent proposed, under conditions the reviewer set, and the agent still has to be the one that acts.

What the agent’s code has to do

This changes one thing about how an agent is written: it has to handle a 403 from its model provider that carries a review id. The right behaviour is to stop, surface the id to a person, and resume only after approval. The wrong behaviour is to retry. A framework that retries on any error will hammer the held call and learn nothing.

That is a small change to make once in the agent’s base client, and it is the price of a human-in-the-loop requirement that is enforced rather than hoped for.

Knowing it is working

Three things tell you the control is real rather than decorative.

Every withheld call shows up as an enforced denial with the review request it created, so an auditor can walk from “the policy says high-risk actions are reviewed” to the specific calls that were held, who decided, and what they decided.

A behaviour baseline over the agent’s recent traffic flags the day its denial rate jumps, a new caller drives it, or it starts calling tools at three in the morning. Those are usually the days a trigger needs attention.

And a governance posture score, built from the agent’s own record, awards credit for authority only when triggers are both declared and enforced. Declared-but-monitored is visible as exactly that. The score is not a gate, but it makes the gap between the policy sentence and the proxy’s behaviour impossible to miss.

Where this sits against the frameworks

NIST AI RMF’s Manage function asks for mechanisms to intervene and for risk responses that are “documented and monitored”; a held call with a recorded decision is both. ISO/IEC 42001’s Annex A controls on human oversight and on the AI system lifecycle want oversight that is designed in rather than described. The EU AI Act’s Article 14 requires that high-risk systems “can be effectively overseen by natural persons”, including the ability to intervene or interrupt. A review trigger that stops a tool call until a person approves it is the plainest possible implementation of that requirement.

None of the frameworks say “put a proxy in front of the model”. They say the human must be able to intervene before the action. The proxy is just the place where that is possible.