What actually stops an agent is code it cannot reach - tearing down two agent prototypes

In July 2025 a developer working with Replit's AI agent had a code freeze in place. During that freeze the agent made unauthorised changes to live infrastructure and deleted the production database, which held records covering more than 1,200 executives and more than 1,190 companies.¹

A freeze is a policy, and the only place you can put a policy where an agent will read it is the context window. So the freeze was a sentence, and it held about as well as a comment in the code.

Jason Lemkin, the user involved, asked the question that should end most agent governance pitches: how could anyone use this in production if it ignores all orders and deletes your database.¹ Replit's chief executive Amjad Masad answered honestly, and his wording is the part worth keeping. Not that it should never happen. "Unacceptable and should never be possible."¹

One detail in the same reporting got less attention than the apology, and it matters more. The agent also said a rollback would not work. Lemkin recovered the data himself, and concluded the agent had either fabricated that answer or did not know what recovery options existed.¹ So the confession was vivid and the damage report inside it was wrong. Hold that, because it is mechanism two.

Every fix Replit shipped afterwards was structural. Automatic separation between development and production databases. Better rollback. A planning-only mode where the agent can collaborate without touching a live codebase.¹ Not one of them was a better instruction. That is this entire piece, delivered by the company that paid for the lesson.

A system prompt is not a permission. An instruction is not a constraint. A self-reported confidence score is not a measurement. Anything inside the context window is advisory, because the model is a text generator and the context window is text. The only things that actually stop an agent are the things it has no ability to influence: a decision computed from signals it does not produce, a credential it does not hold, a record it cannot rewrite, and data it never receives.

The most-quoted number in enterprise AI is not one you can plan against

Most decks on this subject open with the MIT finding that roughly 95% of enterprise generative AI pilots deliver no measurable return. It comes from "The GenAI Divide: State of AI in Business 2025", out of MIT's NANDA initiative, and it was reported by Fortune in August.² I am not opening with it, because I could not get it to hold still.

Fortune describes the study as 150 interviews with leaders, a survey of 350 employees, and analysis of 300 public AI deployments.² Summaries written from the report itself describe 52 executive interviews, 153 survey responses and 300-plus implementation reviews, and label the findings preliminary.³ Those are not two descriptions of the same study. The definition of failure is contested too: one critic who read it argues the report counts any pilot that did not reach full production deployment as a failure,⁴ which is a deployment-stage measure wearing the clothes of a P&L measure.

I have no view on who is right, and that is the point. A number whose sample size changes depending on which write-up you read is a poor foundation for a governance budget.

There is a better 95% available and it points the other way. Simon Willison, writing about the prompt-injection risk he calls the lethal trifecta, notes that guardrail vendors advertise catching around 95% of attacks.⁵

Same number, opposite meaning, and the second one is useful. In security the residual 5% is the whole story, because an attacker needs the gap once. Willison is blunt about the state of the art: we still do not know how to 100% reliably prevent this from happening.⁵

Here is a figure you can check yourself. Zhan and colleagues took eight published defenses against indirect prompt injection on LLM agents and bypassed all eight with adaptive attacks, consistently achieving an attack success rate above 50%.⁶ Eight out of eight. If your agent's safety story is a defense living in the prompt, or a classifier over the prompt, that result is your prior.

Why the gate cannot live in the model

The clearest statement of the principle I have read is in a 2025 paper on design patterns for securing LLM agents, written jointly by researchers from ETH Zurich, EPFL, Google, Microsoft, IBM and Invariant Labs.⁷

Impossible, not unlikely. The same paper is direct about the alternative: heuristic approaches provide some protection but they do not provide guarantees.⁷ All six of its patterns are architectural rather than linguistic. Action-Selector, where the model picks from a fixed menu and never composes. Plan-Then-Execute, where the plan is frozen before untrusted data is read. LLM Map-Reduce. Dual LLM, where a privileged model holds the tools and never touches untrusted data while a quarantined model touches the data and holds no tools. Code-Then-Execute. And Context-Minimization, which removes content from the context once it has done its job, so an injection sitting in that content has nothing left to act on.⁷

OWASP arrives at the same place from the practitioner's side. Its 2025 entry on prompt injection says plainly that it is unclear whether fool-proof prevention exists.¹⁰ Read its mitigations and every one is a boundary rather than a wording: handle privileged functions in code rather than providing them to the model, put a human in the loop for privileged operations, and separate and clearly denote untrusted content.¹⁰

Google's engineering write-up on agent security is the most useful vendor document I found, because it concedes the limit of the thing vendors usually sell. It splits defense into two layers: layer one is deterministic runtime policy enforcement that intercepts actions and evaluates them against predefined rules, layer two is reasoning-based, using models to judge inputs and internal reasoning for risk. On layer two it says the strategies are non-deterministic, cannot provide absolute guarantees, and are inadequate on their own for scenarios demanding absolute safety, especially where actions are critical or irreversible.¹¹ Its three principles: agents must have well-defined human controllers, agent powers must have limitations, and agent actions and planning must be observable.¹¹

If you want the price of the architectural version, CaMeL is the cleanest measurement available. It extracts control flow and data flow from the trusted user query up front so that, in the authors' words, the untrusted data retrieved by the LLM can never impact the program flow.⁸ On AgentDojo, the benchmark of 97 realistic tool-use tasks and 629 security test cases across environments including an email client, an e-banking site and travel booking,⁹ it solves 77% of tasks with provable security against 84% for an undefended system.⁸ Seven points of utility. That is a number you can take into a room, and a great deal cheaper than the alternative price, which is the paragraph at the top of this piece.

Five things that actually stop an agent

What follows is from two prototypes I built and run in public on this site: Praxis, an agent control plane, and Azeon, a governed customer-operations agent. Both are independent concept studies, self-built on public data, never client work, and not affiliated with, endorsed by or commissioned by any company. Azeon here is my own demo and shares its name with an unrelated commercial product I have no connection to; nothing below describes or evaluates that product. I am using these two because I can quote the actual lines instead of describing a diagram, and because each mechanism is answering a specific documented failure.

1. A decision computed from signals the model does not produce

Praxis routes every live agent run to one of three outcomes: auto, review, escalate. The route is a pure function of four scalars, and it is module-level code, not prompt text.

// src/prototypes/praxis/lib/livetrace.ts
export const EVIDENCE_FLOOR: Record<string, number> =
  { L1: 0.3, L2: 0.36, L3: 0.42, L4: 0.5 };
export const GROUND_FLOOR = 0.3;
export const AUTONOMY_CAP: Record<string, number> =
  { L4: 5000, L3: 50000, L2: 500000, L1: 0 };

export function enforceDecision(
  autonomy: string,
  evidence: number,
  value: number | null,
  grounded?: number | null,
): { decision: "auto" | "review" | "escalate"; reason: string }

autonomy is the agent's declared tier. evidence is a retrieval signal computed server-side. value is the money blast radius, regex-parsed out of the case text. grounded is an overlap check run against the answer after the model has produced it. Three of the four are model-independent by construction; the fourth is measured against the model's output rather than reported by it. There is no argument the model can make here, because there is no channel through which to make one.

The evidence signal is real search, not a vibe. Around 680KB of corpus in /public/praxis/data is chunked into roughly 450 passages and indexed with BM25 at k1=1.4 and b=0.75, cached at module scope. The score is evidence = 0.6 * coverage + 0.4 * sat(topBM25) where sat(s) = s / (s + 10). Coverage is IDF-weighted over the scenario's own informative terms only: expansion terms taken from the department and agent names are used to retrieve but are deliberately excluded from the coverage denominator, so you cannot inflate the signal by padding the query with domain vocabulary. Terms the corpus has never seen carry a fixed weight of 1.5, so unknown jargon counts against the score rather than being silently ignored.

Two details in those constants matter more than the formula. The first is the inversion in AUTONOMY_CAP: of the tiers allowed to act on money, the most autonomous one, L4, has the lowest cap at 5,000 rupees, while L2 has the highest at 500,000, and L1 is capped at zero because it only ever suggests. That is deliberate. High autonomy is for high-volume low-stakes work. A tier that acts without asking should only be allowed to act on things that do not matter much, and the moment the blast radius grows, the same tier has to stop. Most autonomy ladders get this backwards and grant more authority and more money in the same step.

The second is that the floors are calibrated rather than chosen. The comment in the file records that the six seeded scenarios were scored against an off-corpus control that lands around 0.21, and the floors sit above it. That is the difference between a threshold and a magic number, and it is the first thing I would ask any vendor for. If nobody can tell you what a deliberately bad input scores, nobody knows what the gate is doing.

When the gate fires, the reason string is written for the person it lands on and it says the quiet part out loud: computed evidence of 34% from retrieval coverage is below the L3 floor of 42%, so the case routes to a human, and the model's self-reported confidence played no part. An escalation that cannot explain itself gets rubber-stamped.

2. A hard overwrite of the model's account of what happened

In 2024 the British Columbia Civil Resolution Tribunal decided Moffatt v. Air Canada, 2024 BCCRT 149.¹⁵ The airline's chatbot told a bereaved passenger he could apply for a bereavement fare retroactively, contradicting the airline's own published page. Air Canada argued the chatbot was in effect responsible for its own statements. The tribunal called that a remarkable submission, found that Air Canada did not take reasonable care to ensure its chatbot was accurate, and ordered it to pay the fare difference.¹⁵¹⁶ One line does the work: "While a chatbot has an interactive component, it is still just a part of Air Canada's website."¹⁵

That is the legal counterpart to the engineering argument, and the best answer I know to why a governance engine has to overwrite the model's prose rather than merely disagree with it. If the model says the refund is approved, a tribunal may treat that as the company saying so. And as the rollback claim showed, the model's account of what happened can be wrong in whichever direction happens to read most fluently. The gate is not only a safety feature. It is the boundary of what you are liable for.

Azeon's answer is a four-gate action gateway running in fixed order: identity, then eligibility, then penalty, then logic. The header comment in the file is the design brief. LLMs converse; this decides. No gate, no action. Rules are rows in a Postgres table keyed by action type, gate and rule key, each dispatched to a named deterministic evaluator. The model can call propose_action. There is nothing it can call that skips the gateway.

The system prompt's first rule tells the model never to narrate governance: never state in prose that an action is approved, denied, eligible, refunded, rescheduled or done, and never summarise gate results. By my own thesis that rule is advisory, and I want to be precise about why it is still worth writing. It is a request for cooperation that reduces confusing output. The control is somewhere else. The action_status block the customer sees is injected by the server from the gate report, rendered as its own component, ordered ahead of the model's prose, and persisted regardless of what the model wrote. The prompt asks. The render does not depend on getting a yes.

3. Grounding measured against the retrieved text, not asserted by whoever wrote it

NIST's Generative AI Profile gives this failure its best name. Confabulation is defined there as the production of confidently stated but erroneous or false content, colloquially hallucinations or fabrications, by which users may be misled or deceived.¹³ Confidently stated is the operative half.

Which is why self-reported confidence must not route anything. A 2024 evaluation across six language and vision-language models found high calibration error and overconfidence most of the time, which the authors summarised as a poor capability for uncertainty estimation.¹⁴ A score that is high when the model is right and also high when it is wrong carries no information, and wiring it to a threshold produces a gate that opens on exactly the wrong cases.

So Praxis displays the model's self-reported confidence and groundedness in the trace, clearly labelled, and routes on neither. What routes is a lexical overlap check computed between the final answer and the passages that were actually retrieved, gated at GROUND_FLOOR = 0.3. An answer that has drifted from its own sources cannot ship autonomously however certain it sounds.

Two smaller controls close the obvious cheats. The model is handed N passages and told to cite them inline as [S1] through [SN]; the response is post-processed to strip any marker outside that range, so it cannot cite a fifth source when four were supplied. Proposed tool calls are filtered against a Set of the agent's declared tool names, so it cannot invent a tool. Three lines each. Both are the difference between an audit trail and a story.

4. A ledger whose integrity does not depend on the application layer

The EU AI Act makes this a compliance requirement rather than an engineering preference. Article 12 requires that high-risk systems technically allow for the automatic recording of events over the lifetime of the system, sufficient to identify risk situations, support post-market monitoring and monitor operation.¹⁸ Article 19 requires providers to keep those logs, to the extent the logs are under their control, for a period appropriate to the purpose and at least six months.²⁰ An evidence ledger is not an indulgence. It is a stated obligation with a retention floor.

Azeon's ledger is hash-chained, and the chain is computed inside Postgres rather than in application code.

-- supabase/migrations/20260706_azeon_setup.sql
select seq, hash into v_seq, v_prev
  from portfolio.azeon_evidence_events
  where conversation_id = p_conversation
  order by seq desc limit 1
  for update;
if v_seq is null then v_seq := 0; v_prev := 'genesis'; end if;
v_hash := encode(extensions.digest(
  v_prev || '|' || p_event_type || '|' || p_actor
        || '|' || coalesce(p_payload::text, '{}'),
  'sha256'), 'hex');

The companion function, azeon_verify_chain, recomputes the whole chain from the genesis sentinel and returns valid, the event count, and first_bad_seq. It does not merely report that the ledger was tampered with. It reports where. That is the difference between a claim of integrity and a demonstration of one, and it is about twenty lines of PL/pgSQL.

The execution path applies the same idea to atomicity. azeon_execute_action performs the domain mutation, the action status update, the evidence append and the outcome-billing insert inside one function in one transaction. There is no code path that produces an executed action without its evidence event, because there is no code path that writes one without the other. It refuses to run unless the action's status is proposed, approved or pending approval, so an executed action cannot be replayed. And the billing row carries the evidence_seq of the event that justifies it, so every charge points at the ledger entry that earned it.

5. Data the model never receives, and instructions it is structurally unable to obey

Willison's lethal trifecta is the cleanest threat model I have found for agents: access to private data, exposure to untrusted content, and the ability to communicate externally.⁵ Any two are survivable. All three is an exfiltration channel. The design move is to remove a leg, not to police the middle.

Azeon removes the first leg conditionally. For financial-services tenants, while identity is unverified, balances, EMI details and transactions are never assembled into the context at all. The account line the model receives is a masked number, a type, a status, and the note that the rest is locked until verification. This is not an instruction not to disclose balances. The balances are not in the window. It is the Context-Minimization pattern, applied at assembly time.⁷

It removes the third leg with an allowlist I am still slightly pleased with. The agent can send real email, but the server-side guard requires identity verification and requires the recipient to be either the address on file or an exact email token the customer actually typed in this conversation. Exact token, not substring. The comment explains why: substring matching would let x@y.com be waved through by notx@y.com.evil appearing somewhere in the text, which is precisely how an injected instruction would try to walk data out. A guard that is right in spirit and wrong in its matching rule is not a guard.

Two more, briefly. Malformed model output is dropped rather than rendered: blocks are validated by shape and chart values are coerced through Number() with non-coercible entries removed, so a hallucinated value cannot break a render. And the Praxis live endpoint rate-limits at ten runs per IP per minute, with a comment noting that the demo runs under the same blast-radius caps it preaches. If a governance demo is itself ungoverned, do not trust its diagrams.

One honest note about the injection screen on that endpoint. There is a regex first layer, and the comment above it in my own code calls it a coarse first layer, trivially paraphrased around. That is the correct description, given eight out of eight published defenses fell to adaptive attack.⁶ The structural mitigation is separate and stated separately: the scenario is framed to the model as untrusted data, and nothing the model says can move the route. The filter is hygiene. The architecture is the control.

What most implementations get wrong

Five failure patterns show up repeatedly, and four of them are one-line fixes nobody makes until after an incident.

• Rule registries that fail open. A policy row whose rule key has no registered evaluator should deny the action. In Azeon an unknown rule key produces a failed check reading no evaluator registered, which denies. The common alternative is to skip unknown rules, which means the policy you wrote but did not implement silently permits everything it was supposed to stop.
• Boolean decisions. Passing a gate and needing a human are orthogonal axes, not two ends of one. Azeon's penalty gate always passes its amount check but sets requiresApproval above the threshold, which is why the gateway's decision is three-valued: deny, approval, execute. Collapse that into a boolean and you will either block legitimate work or auto-execute large refunds.
• Denials that are refusals instead of diffs. Every check carries an expected and an actual, so a denial reads: within the 7-day window of delivery, expected at most 7 days, got 11 days since delivery. That is a sentence a customer can act on and an auditor can check. "This action is not permitted" is neither.
• Gateways that answer for records they should not see. Azeon loads the order, account or transaction scoped to both tenant and customer before any rule runs, so a reference belonging to someone else returns not found. Skip that scoping and your policy engine becomes an enumeration oracle: an attacker learns which order numbers exist from how the denials differ.
• Approval queues with no veto rate. Article 14 of the EU AI Act requires that the human overseer be able to disregard, override or reverse the output, and to interrupt the system through a stop button that brings it to a halt in a safe state.¹⁹ It also requires that the overseer be enabled to remain aware of automation bias, the tendency to over-rely on the output.¹⁹ NIST calls the same thing excessive deference to automated systems, and notes it can exacerbate confabulation.¹³ A regulator has written the rubber-stamp problem into statute. If your human-in-the-loop queue runs at a 100% approval rate, it is not oversight, it is a log line with a person's name on it.

The weakness in my own gate

The Praxis evidence signal is lexical. BM25 coverage plus a saturation term is a bag-of-words measure, and it will under-score a correct answer expressed in vocabulary the corpus does not happen to use. A domain expert phrasing something precisely but unusually can score below an off-topic passage that shares the right nouns. That is a real limitation, and semantic retrieval would reduce it.

What makes it acceptable is the direction of the error. An under-scored case routes to a human, so the failure mode is too many escalations, which you discover in a queue-depth chart and fix by tuning. The opposite failure mode, a gate that over-scores and lets things through, is discovered in an incident report. When you cannot make a gate perfect, make it wrong in the survivable direction, and say out loud which direction that is.

Five questions, all answerable by reading code

If you are evaluating an agent platform, yours or someone else's, these separate a control plane from a control-plane-shaped diagram.

1. Show me the function that makes the route decision. Which of its arguments does the model produce?
2. What does a deliberately bad input score, and where is that number written down? If nobody knows the floor's calibration, the floor is decoration.
3. Who writes the sentence the customer reads when an action is refused, the model or the gate? If the model can write the word "approved", you are liable for the word.
4. If I edit a row in the audit log, what breaks, and does anything tell me which row?
5. For a privileged action, what was the approval queue's veto rate last quarter? A rate of zero is the answer to a different question.

Run them and try to talk one past the gate

Both prototypes are live on this site and both are built to be falsified rather than admired. The Praxis live agent shows the retrieved passages, the computed evidence score, the measured grounding, the parsed blast radius and the model's own confidence side by side, with the confidence marked as unused; the interesting exercise is writing a scenario that talks it into auto-approving. Azeon runs its four gates against real Postgres rows, and the evidence view has a verify button that recomputes the hash chain and names the first sequence number that fails, and it is open to anyone. If either does something the mechanisms above say it cannot, I would like to know.