Most AI SEO advice starts with content. This article starts with something duller and more decisive: whether the systems behind ChatGPT, Claude, Perplexity and Gemini are allowed to read your pages at all.
If they are blocked, nothing else matters. The best article in your industry cannot be cited by a system that never fetched it. And the blocking is often accidental, caused by a default setting someone applied years ago.
“AI crawler” is not one thing anymore
A few years ago you could talk about “the AI bot.” Today the major companies run separate bots for separate jobs. One 2026 decision guide sums up the shift: training and search are now separate bots, so GPTBot is not OAI-SearchBot, and ClaudeBot is not Claude-SearchBot, and each has its own user-agent that can be controlled independently in robots.txt.
It helps to sort bots into three groups:
- Training crawlers collect text used to build or improve models. Examples listed in one guide include GPTBot, Google-Extended, Bytespider and CCBot.
- Search crawlers index pages so an assistant can cite them. OAI-SearchBot and PerplexityBot are given as examples. Anthropic’s Claude-SearchBot belongs here too.
- User-triggered fetchers load a page because a person asked an assistant to look at it. ChatGPT-User and Claude-User are the usual examples.
The practical lesson comes from one source: blocking a training crawler keeps your content out of model training, while blocking a search crawler can remove you from AI answers and citations, so each should be decided separately.
The common mistake: blocking one bot and losing another
A frequent error is blocking the famous bot and assuming the job is done. According to one technical guide, blocking GPTBot without allowing OAI-SearchBot still removes your content from ChatGPT Search results, so both directives are needed. The names look related, but the permissions are not. pixis
The opposite mistake happens too. Another source notes that many sites inherit default robots.txt or firewall rules that block AI bots without the team realizing, leaving them invisible to AI engines regardless of content quality. That claim comes from a vendor glossary, so treat the wording with care. But the pattern is easy to believe, because security plugins and hosting providers often add bot rules automatically.
Decide what you actually want
There is no single correct policy. It depends on what your content is worth to you and what you want from AI visibility. Three common positions:
Open. You want maximum exposure. Allow training, search and user-fetch bots. This suits businesses whose main goal is awareness, such as local service firms or consultancies.
Visible but not for training. You want citations but object to your writing feeding model training. Several guides call this the defensible default: disallow training crawlers while allowing search and retrieval crawlers. A simple version looks like this:
Closed. You publish paid or premium material and want it out of both. Block everything relevant, and accept that AI answers will rarely mention you.
At Sarkar SEO, the first conversation with a new site is usually about which of these three it is, because the answer changes every technical recommendation that follows.
Be careful with Google-Extended
This token causes the most confusion, and the sources I found disagree. Some say Google-Extended is used only for Gemini training and AI Overview generation, and that it is separate from Googlebot. Others describe it more narrowly, as a control that opts you out of training only and does not affect AI search.
Because these accounts conflict, I’d treat Google’s own documentation as the authority, not any third-party guide. Check it directly before you decide. Also keep Googlebot itself open. One audit checklist advises confirming that Googlebot is not blocked and that the split between Googlebot and Google-Extended is correct.
Not every bot obeys
Robots.txt is a request, not a lock. A reference table from one technical guide marks most major bots as respecting it, but lists ChatGPT-User as “no longer guaranteed.” I could not verify that against OpenAI’s documentation, so confirm it yourself if it matters to you. nav43
The bigger point is that robots.txt only governs polite bots. If your content is truly sensitive, put it behind a login, not a text file.
A twenty-minute access audit
You don’t need special software to check your own door.
- Open your robots.txt. Type your domain followed by /robots.txt into a browser. Read every line. Note any bot you don’t recognize.
- Look for blanket rules. A single “Disallow: /” under a wildcard user-agent blocks everything.
- Check your firewall and CDN. Bot protection tools can return errors to crawlers even when robots.txt says yes. Ask your host or developer whether AI user-agents are being challenged or rate-limited.
- Read your server logs. If you can, search them for the bot names above. A bot you have allowed but never see may be getting blocked elsewhere.
- Test a key page. Make sure it loads without JavaScript trouble, returns a normal status code, and is not hidden behind a cookie wall or pop-up.
- Write down your policy. One short paragraph stating which categories you allow and why. This stops an agency, plugin or new developer from quietly changing it.
- Recheck every quarter. One source recommends a quarterly audit of robots.txt against the current documented crawler list as the minimum cadence. Bot names change, and new ones appear.
What access does not do
Letting a bot in guarantees nothing. It only removes a barrier. Whether an assistant cites you still depends on whether your page is useful, trustworthy and relevant to the question. Think of access as the front door, and content as the reason anyone walks through it.
