fix

Release-Day Issue Counts Are a Bad Metric. This Is a Better One.

DuckDB 2.0 shipped on 21 October. Query Farm did something I wish every project did: instead of asking "does this release feel buggy," they measured issue intake across 46 DuckDB repositories against each release's own pre-release baseline, and published the method along with the result.

Short version: 239 issues in the six days around release. That's 2.9× what the baseline predicted. For comparison, 1.5 was 2.6× and 1.4 was 1.6×. So 2.0 looks worse than 1.5 — until you account for the fact that issue activity was already rising, and that the 1.5 baseline included quieter holiday weeks.

The more useful findings are in the shape of the data, not the multiple.

Where the reports landed

53% of the 239 issues were filed against the core engine. The rest went to clients, extensions, ecosystem projects, and documentation. The core's increase was 2.55× baseline; at 1.5 it was 3.29×. By that measure the engine did better this time, despite the higher raw count.

Clients and installers told a different story: 27 issues against 11.25 expected, about 2.4×. The authors are careful to note that this ordering flips depending on which baseline you use, so they don't claim it's worse than 1.5. That kind of caveat is rare and worth noticing.

The number I find most interesting is first-time reporters: 4.6× baseline, against 2.1× at 1.4 and 2.2× at 1.5. And 67 of those 72 new reporters filed in exactly one repository.

That's a signature. Experienced users hitting a regression tend to file in the core engine, where the machinery lives. New users hitting an install problem, a Python binding that won't build, or a docs page that doesn't match reality file in one client repo and often never come back. A 4.6× rise in first-timers alongside a smaller core spike reads less like "the engine is broken" and more like "the release brought in people whose first experience was friction."

Why release-day counts mislead

Issue volume is a function of three things: defects, users, and reporting cost. A major version changes all three at once. It attracts new users, breaks workflows, and — increasingly — makes it cheap to file a report, because an AI agent can turn a confusing error into a plausible-looking issue in thirty seconds.

The Query Farm post names that last one explicitly: "increasingly common AI agents make it easier to write and submit reports." They can't identify how many reports involved AI, so they don't try. But the observation matters for every open-source project. The signal-to-noise ratio of an issue tracker is a moving target, and the old heuristic — "lots of new issues means a bad release" — gets weaker every year.

What replaces it is composition. Who is filing, where, and at what rate per reporter. In this dataset, issues per reporter actually fell to 0.8× baseline while daily reporter participation rose to 3.4×. More people, each filing slightly less. That's a different phenomenon from a small group of power users finding many regressions, and it points at a different fix.

The part I'd copy

The methodology is the contribution. They define a baseline (weeks 12 to 9 before release), match weekdays, align releases on day zero, and report the sensitivity of the result to an alternative baseline. They separate "first observed issue" from "first ever issue" and de-duplicate people by ID. They exclude bots. They're explicit that they haven't classified regressions, so the counts are not a defect rate.

That last admission is what makes the rest credible. A vendor publishing release-quality analysis has an obvious incentive to make its own product look fine; the way to be trusted is to state what the data cannot show.

Their actual recommendation is appropriately boring: start evaluating 2.0 now, pay particular attention to the clients and installers you use, and watch whether 2.0.1 — planned for 18 November — arrives with fixes and a falling report rate.

What this means outside DuckDB

Three things generalize.

Release-day issue counts are a vanity metric. They measure attention as much as quality, and attention is exactly what a major version is designed to generate.

Baselines beat absolutes. The same 239 reports mean different things against different histories, and the only honest comparison is a project against itself.

And "who filed it" is often more diagnostic than "how many." First-time reporters clustering in one repo is an onboarding problem; veterans clustering in core is an engine problem; a spike in one file's issues after a dependency bump is a supply-chain problem. The count alone can't tell them apart.

The Query Farm post is also, quietly, a product argument — they sell DuckDB integration help, and the piece ends with a consult link. That doesn't make the analysis wrong. It makes it a good example of the kind of marketing that earns attention by being useful, which is a higher bar than most release blog posts clear.

Source: DuckDB 2.0: Upgrade Now or Wait for 2.0.1? — Query Farm.

#databases#releases#metrics#engineering