Most AI SEO advice starts with content. This article starts one step earlier. If an AI system cannot fetch your page, nothing you write on it can be cited.
Many robots.txt files were last edited for Googlebot. As one industry write-up puts it, such a file almost certainly does not account for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or the dozen other AI crawlers that have appeared since 2023. That gap can quietly cost you visibility, or quietly give away content you meant to protect.
One brand, several bots, different jobs
The key idea is that a single AI company usually runs more than one crawler, each with a different purpose. OpenAI’s documentation states that each setting is independent of the others, so a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to signal that content should not be used for training.
Independent write-ups describe the same pattern for other vendors. Anthropic runs ClaudeBot for training and Claude-SearchBot for retrieval. Google-Extended is different again: it only controls Gemini and Vertex AI training, and does not affect Google Search crawling or ranking, which is governed by Googlebot. somethingincstudiohawk
That produces three separate decisions, not one.
The three-decision model
Sarkar SEO uses a simple model for this. It is our own working framework, built on the vendors’ published guidance.
Decision 1: Training. Do you want your content used to train models? Crawlers here include GPTBot, ClaudeBot, CCBot and Google-Extended. Blocking them limits future training use. It does not remove you from live answers.
Decision 2: Search and retrieval. Do you want to be indexed so assistants can cite you? This covers OAI-SearchBot, Claude-SearchBot and PerplexityBot. The stakes are higher here. OpenAI recommends allowing OAI-SearchBot in your robots.txt file, and allowing requests from its published IP ranges, to help ensure your site appears in search results.
Decision 3: User-triggered fetches. When a person asks an assistant to read a specific page, a different agent makes the request. OpenAI says of ChatGPT-User that because these actions are initiated by a user, robots.txt rules may not apply. Independent analysis reports similar behavior from some Google and Perplexity fetchers. Treat this tier as outside robots.txt control.
The common mistake is merging these into one choice. Blocking “AI” wholesale can remove you from answers your buyers read. Allowing “AI” wholesale may hand over content you wanted to license or protect.
Which policy fits which site
There is no universal right answer, only trade-offs.
- Businesses that sell services or products. Visibility is usually the goal. A common setup is to allow search crawlers and decide on training separately.
- Publishers with paid or licensed content. Blocking training and retrieval crawlers may be reasonable, accepting the loss of citations. Some large news sites already block many of these bots, according to one public crawler checker.
- Sites with sensitive areas. Block internal paths, carts and parameterized search URLs from all crawlers, as you would for any bot.
A single illustrative setup for a services business that wants citations but not training would disallow GPTBot, ClaudeBot and Google-Extended while allowing OAI-SearchBot, Claude-SearchBot and PerplexityBot. Check each vendor’s current documentation for exact user-agent names before deploying, because they change.
Robots.txt is a request, not a lock
Two limits matter. First, robots.txt is an agreement, not a technical barrier, and well-behaved bots honor it. Poorly behaved scrapers may not. If content truly must be private, use authentication, not a text file.
Second, some controls are invisible. One analysis notes that Google-Extended and Applebot-Extended make no HTTP requests and generate no log lines. You cannot confirm them in server logs, so your only record of intent is the robots.txt itself.
The hidden failure: the file says yes, the firewall says no
A permissive robots.txt does not guarantee access. Security tools, CDNs and bot-protection rules can block crawlers before they ever read your file. This is practitioner experience rather than something the sources above test, but it matches OpenAI’s advice to allow its published IP ranges.
A quick audit looks like this:
- Read your live robots.txt. Look for a blanket Disallow: / that catches every bot.
- Check for conflicting rules. Each crawler needs its own clearly labelled group.
- Review firewall and CDN settings. Confirm that legitimate AI search crawlers are not challenged or rate-limited.
- Verify by IP, not just name. Anyone can fake a user-agent string. OpenAI publishes real address ranges for its bots, so match requests against those before trusting a log line.
- Wait for changes to apply. OpenAI notes that search-related changes can take about 24 hours to take effect.
- Re-test after site migrations. Staging rules that block everything often leak into production.
For reference, OpenAI’s crawler documentation lists the current user agents and IP ranges.
What access does and doesn’t buy you
Allowing a crawler makes you eligible, not guaranteed. Being fetched is the entry ticket; whether you are cited still depends on relevance, clarity and trust. Equally, blocking a training bot is a content-rights decision, not a ranking tactic, and it should not be sold as one.
Review logs monthly. Count hits from each named crawler separately, since a rise in retrieval crawlers can signal growing interest, while a sudden drop may mean a rule or firewall change has cut you off.
