Every team using a coding agent has this argument eventually. One person has built the scaffolding — an instruction file, a pre-commit gate, a hook that refuses "done" until tests have run. Someone else points out the agent passes the tests anyway. Then the gate blocks six runs in a morning, and someone asks whether the guardrails pay or are just ceremony.
Rakesh Menon ran 270 unattended runs to find out: three models, seven tasks, five runs per task/condition/model, conditions one layer apart — none, rules (an AGENTS.md), and gates (rules plus a pre-commit gate, a stop hook, a destructive-command hook, and an instruction to show test evidence).
The answer is more interesting than yes or no.
Pass rate barely moved
All 270 runs were accepted. First-pass results on the hidden tests: 90/90 with nothing, 89/90 with rules, 88/90 with gates. No statistically clear difference for any model.
If your metric is "did the agent complete the task," the guardrails did nothing — and cost you some waiting. That's the result most teams would stop at, and it's the wrong place to stop.
What changed is the residue
The interesting measure is what a run leaves behind at "done." On the migration task (T7), 14 of 15 runs without an instruction file changed the database schema without the test the team's rule requires — a rule those runs never saw. With the rule, 0 of 15 did.
Pooled over the four harder tasks, defects at done were 15/60 with nothing, 14/60 with rules, and 9/60 with gates. The widest gap was Sonnet at 1 versus 5 — p = 0.18, a lead rather than proof. Files changed outside the task by the large model: 10/35 with nothing, 2/35 with rules, 1/35 with gates.
So the guardrails didn't make the agent better at the task. They made it less likely to hand over something quietly broken or out of scope. That's insurance, not acceleration, and it's exactly the kind of benefit that never shows up in a demo.
A guardrail can create its own failure
The best part of the write-up is the T4 anomaly. That task can't be completed without changing an existing test, and the instruction file says tests are read-only for agents.
Under that rule, 23 of 30 runs built the feature, left the old test failing, and asked a human to change it. Nobody came — it's an unattended run — and "asked a person" counted as a defect, just like leaving a bug.
The rule was still right; it exists so agents don't delete or weaken tests to turn red green. But the experiment exposes the cost precisely: a guardrail that requires human action in an unattended pipeline converts a code failure into a stall. If you're going to install that rule, you need a queue, a notification, and someone whose job is to answer it. Otherwise the gate is a very polite way to ship nothing.
The economics are dominated by the model
Cost per accepted change, on the harder tasks:
- Haiku: $0.04–0.06.
- Sonnet: $0.27–0.30.
- Opus: $0.82–0.97.
The harness moved cost by cents; the model moved it by more than tenfold. All 270 runs cost an estimated $115.58 at list prices. If you're optimising spend, the model choice is the lever and the guardrails are noise — though the cheap model's success was on "small, fully specified tasks," which is doing a lot of work in that sentence.
What the experiment can't say
Five runs per cell is the protocol minimum. Seven tasks, one small codebase, one agent, three models from one vendor, in October 2026. The author wrote the tasks, the tests, and the harness — which could favour the harness — and the defect measure uses its own guards. All of that is stated plainly, which is why the result is worth taking seriously.
The lesson
"Did it pass?" is a completion metric, and completion metrics are exactly what agents are good at optimising. The useful question is what the run left behind: an untested migration, a file outside the task, a failing test with a note for a human.
Guardrails are not there to raise the pass rate. They're there to shrink the tail of plausible-looking work that isn't done. The T7 run that passed every hidden test while leaving a schema change with no test is the whole argument in one example: perfect by one metric, defective by another, and which one you pick decides whether the guardrail was worth it.
Source: Do Guardrails Make AI Coding Agents Better? What 270 Runs Showed — Rakesh Menon.