Before an AI assistant can cite your page, a bot has to be allowed to read it. That sounds obvious, yet it is one of the most commonly botched parts of AI SEO. A single misplaced line in robots.txt can remove a site from ChatGPT’s search answers, and many owners never find out. Here is how AI crawlers work and how to set the rules deliberately.
Three Jobs, Three Kinds of Bot
AI crawlers are not one thing. One industry guide sorts them into three groups: training crawlers such as GPTBot, ClaudeBot and Google-Extended, search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, and user-triggered fetchers such as ChatGPT-User and Claude-User. Each answers a different question: lilachbullock
- Training: may your words help build future models?
- Search: may an assistant index your pages so it can cite them?
- User-triggered: may a live session fetch a page because someone asked about it?
Treating these as a single decision is where most mistakes begin.
What OpenAI Says
OpenAI’s documentation is the clearest primary source. It describes OAI-SearchBot as the crawler that surfaces sites in ChatGPT’s search features, and states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. GPTBot is separate: it crawls content that may be used to train OpenAI’s generative AI foundation models.
The third bot behaves differently. ChatGPT-User acts when a person asks a question, and OpenAI notes that because these actions are user-initiated, robots.txt rules may not apply, and it should not be used to manage search opt-outs. OpenAI also says a robots.txt change can take about 24 hours to take effect for search, so don’t judge a fix after an hour.
Other Assistants
Anthropic documents separate tokens too. One configuration guide reports that Anthropic states all three of its bots honor robots.txt, including the non-standard crawl-delay extension. The practical consequence is that blocking ClaudeBot does not automatically block Claude-SearchBot, because the two do different jobs. For Perplexity, a secondary guide says Perplexity’s own bot documentation states PerplexityBot is not used for training. Since I have not read Perplexity’s page directly, check it before relying on that point.
Google is a separate case. Its AI Overviews and AI Mode draw on the regular Search index, so ordinary Googlebot access and snippet eligibility matter most there.
The Common Trap: Blocking by Accident
Many sites block AI bots deliberately, and some of that is reasonable. A Zyte analysis of robots.txt files found training crawlers blocked far more often than search ones: GPTBot is blocked at 5:1, while OAI-SearchBot is blocked at a 2:1 ratio, the most balanced in the dataset. That pattern suggests many publishers are making a conscious split between bots that compete with them and bots that send readers.
The trouble comes from blanket rules. The same analysis notes that some sites have made a general decision to close their robots.txt to all automated access, and AI crawlers are caught as part of that. Those owners may have never decided anything about AI visibility. They simply inherited a default.
The reverse error is just as costly. One guide points out that blocking OAI-SearchBot while allowing GPTBot hands OpenAI your content for training while removing you from ChatGPT Search, which is usually the opposite of what a brand wants.
A Simple Decision Framework
- Do you want to appear in AI answers? For most businesses selling services or products, yes. Allow the search bots: OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking them gains almost nothing for marketing content.
- Are you comfortable with training use? This is a business and ethics choice, not an SEO one. Publishers with paid content or licensing concerns often block training bots. Sarkar SEO suggests writing that decision down, with a named owner, rather than leaving it to whoever last touched the file.
- What about user-triggered fetchers? Allow them if you want live assistant sessions to open your pages, and accept that these may ignore robots.txt anyway.
Mistakes Worth Checking Today
- Stale files. If your robots.txt predates the split between training and search bots, it may not reflect today’s goals.
- Firewalls and CDNs. A bot can be allowed in robots.txt yet blocked at the network edge. One guide advises allowlisting verified OpenAI IP ranges where ChatGPT visibility matters.
- Wrong file location. Each domain and subdomain needs its own robots.txt at the top level.
- Confusing blocking with deindexing. A Disallow rule stops crawling. It is not the same as telling a search engine to keep a page out of its index.
- Spoofed user agents. Anyone can claim to be a famous bot. OpenAI publishes IP lists for its crawlers so you can verify requests.
How to Verify It Is Working
Open your server logs and count visits by bot name over a few weeks. A healthy site open to search usually shows steady OAI-SearchBot activity, and one guide treats zero visits over a long period as a red flag. Cross-check suspicious entries against published IP ranges. Then run a handful of your target questions through each assistant and note whether your pages appear. Sarkar SEO recommends repeating this check every quarter, since crawler names and policies keep changing.
