Every model launch follows the same arc. It's incredible for two weeks. Then the complaints start: it got dumber, they quietly cut reasoning effort, they're serving a cheaper checkpoint. The denials are equally predictable: nothing changed, you're noticing limitations you didn't see before.
Both sides usually argue from anecdotes. Nathan Langley built an instrument instead.
LiveNerf has been running daily measurements of Opus 5.5 since late September, and the design is the interesting part. He screened 2,336 questions from GPQA Diamond, MMLU-Pro, competition math, and AIME, sampling each four times. Opus 5.5 got about 93% right on the first attempt, and roughly 97% of questions were either always right or always wrong — useless for detecting small changes. That left a frozen panel of 78 questions the model sometimes solves and sometimes misses, plus a 12-question control arm, run every day through a pinned CLI with fixed prompts, grading, and harness settings.
Thirty-four consecutive days. Two completed 10-day comparison windows against a 10-day baseline. Pre-registered thresholds.
The result, with error bars
Baseline: 58.4%. Window one: 59.2%, a change of +0.7 ± 1.8 percentage points. Window two: 57.1%, −1.5 ± 1.8. Neither met the threshold for declaring a change.
That's the headline, and it's deliberately narrow: no sustained change detected on this panel. The minimum detectable effect is about 6.6 percentage points per 10-day window. A smaller real change could hide inside that.
This is the part the discourse usually omits. A benchmark with a ±1.8pp standard error cannot adjudicate a claim like "it feels 5% worse at coding," and neither can a screenshot of two prompts. The honest statement is "we didn't detect it," not "it didn't happen."
The positive controls are the best idea in the piece
Anyone can run a benchmark and find no change. The question is whether the instrument could have found one. Langley tested this by deliberately reducing reasoning effort:
- Medium effort: −4.2 ± 3.9 pp accuracy, −26% output tokens.
- Low effort: −8.3 ± 4.5 pp accuracy, −62% output tokens.
The accuracy deltas are noisy. The token deltas are enormous. That suggests a quiet effort reduction would show up in output-token distributions before it showed up in benchmark accuracy — a much cheaper and more sensitive signal than correctness on a 78-question panel.
He also tried swapping Opus 5 for Opus 5.5. The measured accuracy difference was −3.8 ± 6.3 pp, with about 23% fewer tokens. At the pre-registered 99% confidence threshold, the instrument could not reliably distinguish the two models on accuracy. That's a genuinely uncomfortable finding, and he states it plainly: an instrument that can't detect a specific substitution shouldn't be used to claim that substitution didn't happen.
The questions are also broken
During an item audit, he found eight answer keys that looked incorrect and around thirty ambiguous questions among eighty reviewed. In a panel deliberately selected for questions the model sometimes gets right, that's not surprising — the questions that discriminate are often the ones that are hard to grade — but it means the measurement error is not only statistical.
He kept the panel frozen and published a sensitivity analysis rather than quietly removing items after the data came in. That's the right call, and it's rarer than it should be.
What this says about the debate
The "nerfing" argument is usually conducted at a level of precision the evidence can't support, in both directions. Users have a real signal — something in their workflow changed — but attribute it to model weights. Providers have a real defense — the weights didn't change — but the product around the weights (system prompts, context management, tool behavior, inference settings) is exactly what could change someone's experience without showing up on a benchmark.
Langley's own conclusion is careful about this: the model is only part of the product, and his panel doesn't measure coding agents or long conversations.
The gap he identifies is institutional. We have enormous infrastructure for comparing models at launch and almost none for verifying that they keep behaving the same way afterward. That's not a technical limitation; it's a missing habit. Continuous, independent, pre-registered evaluation is cheap relative to training and mostly absent.
The takeaway
If you think a model got worse, the useful move is not a vibe check but a number: the same prompts, the same grading, repeated over time, with a baseline captured before you suspect anything. Token counts first, accuracy second. And when the answer comes back "no significant change," read the error bars before you read it as vindication.
The most valuable sentence in the post is the least dramatic one: we haven't detected a sustained change, and here is exactly how much change we could have detected.
Source: We measured Opus 5.5 for 30 days — Nathan Langley, LiveNerf.