TPC-D was retired because people optimized for it so hard that query times converged toward zero through caching. TPC-H fixed that by declaring itself ad hoc and forbidding exploitation of prior knowledge and result caching. Thomas Neumann's post argues that ClickBench is now going the same way, accelerated by AI code generation.
The examples are specific and damning.
A system called pgrust performs group-by aggregations on string columns purely by hash value, ignoring the long string content. That's a large speedup, and for the benchmark it's fine, but it can produce wrong results. Another trick: ClickBench repeats predicates across queries, so pgrust caches the predicate and eliminates the second query's cost. And for one query that returns tuples grouped by userid with a limit and no order by, it exploits the fact that tuples are roughly clustered by userid and that any userid will do — so it never reads most of the data.
Neumann's point isn't that these are cheating in a narrow sense. It's that a human wouldn't do them, because they're unlikely to be relevant in general use. An AI will, because it can look at each query in detail and tune for it.
The pressure changed, not the principle
Goodhart's law is old. What's new is the cost of fitting.
Historically, benchmark overfitting was limited by the fact that vendors also had to be useful to customers. That constraint is weak for a system built to win a leaderboard, and it's weaker still when an AI can iterate on every query individually. Ten thousand queries were once a defense against overfitting — no human would fine-tune against a workload that large. Neumann notes that with AI, that defense is gone too.
That's the sharp version of the argument: the benchmark's difficulty used to be a proxy for generalization, because optimizing it was expensive. When optimization becomes cheap, the proxy breaks.
The comment thread is the real debate
The most interesting part of the post is the exchange with pgrust's creator, Michael Malis, who defends the optimizations: ClickHouse does all three; he used a 128-bit salted cryptographic hash, so collision probability is effectively zero; and in a world of bespoke OLAP, maybe visible query sets aren't the problem.
Neumann's reply is the one I'd frame: collisions are unlikely, but aggregating on a hash instead of the value "is a violation of the user contract." The user asked for a group-by on a string; doing it on the hash computes a different result, at least potentially. And group-by strings are often user- or attacker-controlled, so the collision assumptions are worse than random. A user choosing a UUID primary key is different — that's the user's choice.
That's the crux. The benchmark doesn't specify whether semantic shortcuts are allowed, so a system that takes them can win without being comparable. Malis's "does another database do it" heuristic is a reasonable engineering test, but it's not a defense: ClickHouse doing it means ClickHouse has the same contract question, not that the question is settled.
What would make a benchmark meaningful
Neumann's proposals are concrete:
- Be ad hoc and forbid exploiting prior knowledge and any form of intermediate result caching.
- Include an update stream, or read-only systems will always beat transactional ones.
- Require that the systems be older than the benchmark, so they can't have been designed around it.
And the borrowing from ML: Kaggle ranks on a hidden test set, LiveBench releases new questions to prevent contamination. For databases, that means keeping actual queries secret, publishing a representative sample for development, or generating a fresh workload each evaluation round.
The last one is the most interesting. SQLStorm already generates queries; rotating the generated set is cheap. The benchmark becomes a distribution rather than a fixed exam.
My take
The benchmark is a specification, and specifications need adversarial maintenance. If the rules don't say "you may not aggregate on a hash," then a system that does it is compliant, and the leaderboard is measuring something other than what readers think.
That's not a vendor problem; it's a design problem. TPC-H's forbidding of caching is a specification decision, and it's why TPC-H numbers were comparable for years. ClickBench's underspecification is why its numbers now need an asterisk.
AI doesn't create the incentive to game a benchmark. It removes the cost, which means the specification has to carry the weight that friction used to carry. Hidden queries, rotating workloads, and update streams are all ways of making the specification harder to satisfy by accident.
The uncomfortable conclusion for anyone who reads leaderboards: a fast number is a claim about a system under a specification. If the specification is loose, the number is a claim about the specification.
Source: The Benchmarking Crisis — Thomas Neumann, Database Architects.