Before an AI engine can cite your page, something has to fetch it. That step is easy to overlook, and it is where many sites make costly mistakes. Some owners leave every door open. Others panic about AI training and lock everything, then wonder why they vanish from AI answers. Good AI SEO begins with deciding, deliberately, which automated visitors you welcome.
Three Kinds of AI Bot, Three Different Decisions
The key idea is that “AI crawler” is not one thing. A 2026 access-control matrix puts it plainly: training and search are now separate bots, so GPTBot is not OAI-SearchBot and ClaudeBot is not Claude-SearchBot, and each can be controlled independently.
In practice, vendors run up to three types:
- Training crawlers collect text to build models. Examples listed in the sources include GPTBot, ClaudeBot, CCBot and the Google-Extended token. One guide notes that training crawlers send you nothing back.
- Search crawlers build the index that answers cite from. The same guide says blocking one of these is the same decision as blocking Googlebot in 2010. Examples include OAI-SearchBot, Claude-SearchBot and PerplexityBot. contextbolt
- User-triggered fetchers, such as ChatGPT-User and Claude-User, retrieve a page because a person asked an assistant about it.
The vendor-by-vendor breakdown above comes from third-party reference tables, so confirm current bot names in each company’s own documentation before editing anything. Names and behavior change.
The Mistake That Costs Citations
The most damaging error is a blanket block. One developer guide warns that disallowing GPTBot, OAI-SearchBot, ClaudeBot and Claude-SearchBot together succeeds at blocking training but also closes the only channel by which your site gets cited in ChatGPT and Claude search.
The reverse assumption is just as wrong. According to one practitioner write-up, blocking GPTBot does not block ChatGPT search, because OAI-SearchBot and ChatGPT-User are independent. If your goal is to stay out of training but remain citable, you need rules for each bot, not one rule for “OpenAI.”
Google Is a Special Case
Google does not follow the same three-bot pattern. Googlebot is the crawler that matters for visibility, and one cheat sheet states that Googlebot powers Search and AI Overviews, while Google-Extended powers Gemini training. Mixing these up is a common trap.
Google-Extended is also not a normal crawler. A reference describes it as a robots.txt control token, not an HTTP user-agent, and says Google states it does not affect Search inclusion or ranking. So opting out of Gemini training through that token should not remove you from results. Blocking Googlebot itself, however, would remove your traditional visibility and with it your path into AI Overviews.
What Publishers Are Actually Doing
Real-world behavior shows the split in action. A summary of Cloudflare Radar data (snapshot from 31 August 2026) reports that publishers block GPTBot 2.3 times for every allow, yet allow OAI-SearchBot outright. The listed counts show GPTBot with 696 disallow rules against 299 allows, while OAI-SearchBot sits near even at 250 against 266. That summary comes from a secondary site reading Cloudflare’s dashboard, so treat the exact figures as indicative. The direction is clear, though: many sites block training while keeping search access open.
Another write-up repeats a Cloudflare-derived claim that Anthropic’s training bots make 38,000 crawls per referral visit, compared with 14 for Google. I could not trace that to Cloudflare’s original publication, so I would not quote it without checking. It does illustrate why some owners question the trade-off of training access.
Limits of robots.txt
Robots.txt is a request, not a wall. One source notes that well-behaved crawlers respect it, but blocking bad actors reliably needs server-side rules. User-triggered fetchers are a particular gray area. A reference table marks ChatGPT-User as partially respecting robots.txt, since it acts on a user’s request. Another wiki adds that such fetchers generally ignore robots.txt by design, so blocking them requires CDN-level controls. robots txt ai crawlers +2
The opposite risk exists too. A security layer can block legitimate bots without you knowing. One reference title warns about the AI crawlers your WAF is silently blocking. A permissive robots.txt means nothing if your firewall returns errors to the bots you want.
A Five-Step Access Audit
- Read your current file. Open the robots.txt in your browser and list every AI-related rule, including old ones added in a rush.
- Decide your stance per category. Training, search and user fetchers each get a deliberate yes or no.
- Check your firewall and CDN. Review bot-management and rate-limit rules for blocks on search crawlers.
- Verify in server logs. Confirm that the bots you allowed are actually fetching pages and receiving successful responses.
- Recheck quarterly. Vendors add and rename bots, so a file that was correct last year may be wrong now.