Project Zero spent six months comparing two automated bug-finders on V8, Chrome's JavaScript engine. One is Fuzzilli, a coverage-guided fuzzer that generates JavaScript and watches for crashes. The other is BigSleep, an LLM-based agent that reads new commits and tries to prove bugs exist.
The headline number looks like a tie: 45 actionable vulnerabilities for BigSleep, 37 for Fuzzilli. The more interesting number is 25 — the percentage of those bugs that both systems found.
The tally is the least interesting part
Raw bug reports favour BigSleep heavily: 441 to 316. Filter to type:vulnerability and it flips: 178 to 95 for Fuzzilli. Filter again to bugs that were actually fixed by a commit referencing them, and you get the 45–37 split.
That filtering exercise is worth sitting with, because it shows how easily an AI-versus-tool comparison can be made to say anything. BigSleep's early reports included a lot of configuration and integration issues — real problems, but not security bugs. Count everything and the agent wins by 40%. Count only what matters and it wins by 20%. Count overlap and you learn something neither number tells you: roughly three-quarters of the vulnerabilities were found by only one of the two systems.
A controlled experiment pushed the overlap higher — around 60% when BigSleep was asked to rediscover 112 historic Fuzzilli bugs from the commits that introduced them, and about 70% with extra prompt variations. But even at 60%, a large share of bugs remained visible to only one approach.
What the agent is actually good at
The case studies are the most useful part of the post, because they show a shape rather than a score.
One BigSleep find involved V8's temporal dead zone. A let variable lives in a special "hole" state until initialised, and the bytecode compiler removes redundant hole-checks for performance. A 2024 fix made the compiler careful about do-while loops containing break. The agent read that patch, reasoned about which loops could skip their condition without registering a break, tried try-catch, then tried labeled breaks, and eventually built a testcase using a labeled break inside a do-while inside a switch. Each step is a hypothesis, tested and revised.
Another find was almost embarrassingly simple once stated: a handler-table field stores bytecode offsets in 28 bits, but V8 allows 512 MB of bytecode. A function big enough to exceed 256 MB of bytecode, with an exception handler at the end, jumps to the wrong place. That is the kind of bug you find by checking units against limits, not by fuzzing.
The fuzzer's showcase bug went the other way. Two Maglev optimisations were each correct in isolation — one emitted an unchecked Smi untag based on feedback, the other removed a checked untag because all Phi inputs looked like int32 — and together they corrupt a pointer once a loop counter exceeds the Smi range. Neither optimisation looks wrong on its own. A coverage-guided fuzzer found it; the agent did not, in any of the experiments.
The finding that undercuts the obvious conclusion
If agents were simply better at hard bugs, you could move budget from fuzzing to agents. Project Zero checked, and the data points the other way: bugs that took Fuzzilli a long time to find were also harder for the agent to rediscover. The correlation is statistically significant but not large. The practical reading is that "AI will eat fuzzing" is not supported yet — the two approaches are bad at overlapping things.
There is also a per-component wrinkle: overlap was noticeably lower in the JIT compilers, the area Fuzzilli specialises in most.
The methodology is the quiet achievement
The post does something most AI evaluations don't: it treats its own numbers as suspect.
It explains why the production comparison is unfair in both directions. It deduplicates across three separate mechanisms — bugtracker duplicates, Clusterfuzz's own dedup, and multiple bugs fixed by the same commit. It restricts the experiment to bugs that reproduce on a standard build with standard flags, and then lists the ways that restriction biases the result against the agent. And it checks for memorisation by plotting rediscovery rates against the model's knowledge cutoff, finding no significant effect before 2026 — while noting the check should be repeated for every new model.
That last point matters more than it sounds. A benchmark where the vendor wins every row is marketing. A benchmark that publishes its own confounders, and the conditions under which its conclusion could be wrong, is evidence.
What I take from it
"Complementary" is the right word, but it undersells the asymmetry. The agent reasons about intent: it can read a fix, infer what the fix missed, and construct a scenario that violates the assumption. It is also distractible — the post notes BigSleep sometimes missed a deeper bug because it found a shallow one first.
The fuzzer has no theory of the code at all, which is both its weakness and its immunity. It cannot be persuaded by a plausible narrative, because it doesn't have one. It just runs things until something breaks, and it will keep doing that at three in the morning without developing an opinion.
The unresolved question is cost. Project Zero says outright that neither system is calibrated for cost-per-bug, so the comparison is about capability, not economics. That is the number a team actually needs: not how many bugs each finds, but how many unique bugs each finds per euro, and whether the marginal ones justify the second pipeline.
Run both, for now. Measure overlap, not volume. And be suspicious of any benchmark that reports a winner without publishing what it counted.
Source: Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines — Project Zero, Samuel Groß.