HitMind fetched robots.txt from the top 10,000 domains and found something that should be taught in every configuration-language course: 29.8% of the files that block paths for everyone and also name an AI crawler let that crawler straight past those blocks.
1,105 files block some paths under User-agent: * and name at least one AI crawler. 329 of them give the named crawler access to at least one path the wildcard group forbids. In 182 of those, the crawler's group says nothing but Allow: / or an empty Disallow:.
The cause is one line of RFC 9309: a crawler looks for the group that names it, and only if there is none does it fall back to *. Google documents the same rule — groups are not combined. So the moment you add a group for GPTBot, even one written to welcome it, GPTBot stops reading every rule under User-agent: *.
What gets let back in
The paths are exactly the ones site owners block on purpose: site search (in 47.1% of the gap files), login and account pages (41.3%), API endpoints (28.0%), admin areas (24.0%), cart and checkout (21.3%), feeds (13.4%).
None of these are secrets — robots.txt is public, and the study is careful to say it isn't access control. But they are the paths that cost money to serve, spawn near-duplicate URLs, or expose an attack surface. Blocking them under * and then adding a cheerful User-agent: GPTBot / Allow: / is a way of undoing the block without noticing.
The most common shape is the blanket allow. The other 147 files give the crawler rules of its own but simply omit some of the disallows.
Why people do it
Three reasons, all reasonable in isolation.
To welcome a crawler. Someone reads that AI crawlers respect robots.txt, and adds a permissive group to make sure their site is included. The group does what they wanted — and silently disables the rest of the file for that crawler.
To block training but allow answers. This is the recommended split: refuse GPTBot and CCBot, allow OAI-SearchBot and PerplexityBot. But if you write named groups for all of them, you have to repeat every disallow in each group, and most people don't. Only 7.1% of files block at least one training crawler while leaving all nine answer engines open; 64.6% of the files that block a training crawler also block an answer engine.
Because generators do it. Some robots.txt generators add a permissive group per crawler. The output looks thorough and has the inheritance bug baked in.
The fixes are simple
If the group exists only to welcome the crawler — nothing but Allow: / — delete it. The crawler falls back to * and follows the same rules as everyone else.
If the crawler needs rules of its own, repeat every Disallow from * inside its group. That's mechanical and easy to get wrong by hand, which is why the study's checker exists.
Or don't name crawlers at all: a single * group applies to everything that follows the standard, including AI crawlers, and there's no inheritance to get wrong.
What robots.txt can't do
The study's own framing is the right one: robots.txt is a request. Well-behaved crawlers follow it; others don't, and some user-initiated fetchers say it doesn't apply. Perplexity states that Perplexity-User generally ignores robots.txt because a user asked for the page.
So the blocked-path gap matters for the crawlers that do obey — which is most of the training and answer crawlers — and not at all for the ones that don't. Anything that must stay private needs authentication or a server-side rule.
The practical numbers are worth keeping: 14.8% of parseable files block GPTBot entirely (21.6% among the top 1,000), CCBot is blocked most often at 15.3%, and 14.4% of sites serve a valid llms.txt. Also 2.7% carry Cloudflare's Content-Signal line, which is a preference outside the RFC and not required to be read.
My take
This is the configuration-inheritance problem in its purest form: a specific rule set replaces the default rather than extending it, and the replacement is invisible in the file's layout. You read the file top to bottom and see the blocks; the crawler reads it by group and sees an exception.
The lesson generalizes to every layered config system — nginx location blocks, firewall rules, CSS specificity, feature flags. The composed policy is not the policy you wrote; it's the policy the resolver produces.
The test is cheap: for each named group, ask what that crawler can now reach that the wildcard group blocks. The study's checker does it for robots.txt. The rest of us should do the same for whatever config we're writing.
Source: AI crawlers in robots.txt: a study of the top 10,000 sites (2026) — HitMind.