Most advice about AI SEO is about content. Yet an answer engine cannot cite a page it was never allowed to read. Before you rewrite a single paragraph, check who can reach your site, because one line in a robots.txt file or one aggressive firewall rule can remove you from an entire platform.
The Three Jobs an AI Bot Can Do
Official documentation from OpenAI, Anthropic and Perplexity points to the same split. Sort every bot into one of three jobs, and the right decision becomes clearer.
Training bots collect content that may be used to build models. OpenAI’s GPTBot and Anthropic’s ClaudeBot fall here. Blocking them signals that your content should not be used for training. Anthropic notes this applies to future material, not content already collected.
Search bots index pages so they can appear in an assistant’s search results. OAI-SearchBot (ChatGPT search), Claude-SearchBot and PerplexityBot belong here. Block these and you reduce your chance of being surfaced. OpenAI says a blocked site may still appear as a plain link, but not in answers.
User-initiated fetchers visit a page because a person asked a question that needs it. ChatGPT-User, Claude-User and Perplexity-User work this way. They do not roam the web on their own.
That keeps you out of training while leaving the door open for search. Blanket rules such as “block all AI bots” treat these three jobs as one and often cost visibility without achieving what the site owner intended.
The Fetcher Wrinkle
User-initiated fetchers behave differently. OpenAI says robots.txt rules may not apply to ChatGPT-User, since a person triggered the visit. Perplexity says Perplexity-User generally ignores robots.txt for the same reason. So a robots.txt block is not a guarantee that no AI product will ever read a public page.
If content truly must stay private, put it behind a login. Robots.txt is a polite request for crawlers, not a lock.
Where Google Fits In
Google’s controls are a separate case and are widely misunderstood. According to Google’s crawler documentation, as summarized by Search Engine Journal, the Google-Extended token governs whether content may train Gemini models or ground responses in Gemini Apps and Vertex AI. It does not control AI Overviews or AI Mode, and does not affect Search inclusion.
Two real levers exist for Google’s AI search features, and both carry costs:
- Nosnippet can remove content from AI Overviews, but it removes ordinary snippets too.
- A Search Console setting lets some site owners exclude their site from AI Overviews, AI Mode and Discover’s generative features. Search Engine Journal reported in August 2026 that it was rolling out to a subset of owners, and that Google began honoring changes on June 17. Google’s own help page is the authority on availability.
Opting out may reduce exposure in AI answers, and measurement is thin: Google’s AI performance report shows impressions by page, country, device and date, but not clicks or queries.
The Silent Problem: Your Own Security Layer
The most common access failure is not a deliberate block. It is a web application firewall, CDN bot filter or hosting default that treats unfamiliar crawlers as threats. Perplexity’s documentation advises allow rules that match both user agent and published IP ranges, placed ahead of any blocking rules, and refreshed regularly because ranges change. OpenAI similarly recommends allowing its published ranges for OAI-SearchBot.
Anthropic adds a caution that applies broadly: blocking by IP address alone may fail, because it can stop the crawler from even reading your robots.txt. Use the documented user agents.
How to Audit Your Access in an Hour
- Read your robots.txt. Look for blanket disallow rules, old plugins’ leftovers and wildcard blocks that catch bots you never named.
- Check subdomains. Anthropic notes rules must be set for each subdomain.
- Review security settings. Find your firewall or CDN bot rules and confirm documented AI search crawlers are allowed.
- Read server logs. Verify which bots visit, how often and which pages they get blocked from. Confirm identity using each vendor’s published IP lists, since user-agent strings can be faked.
- Wait, then retest. OpenAI cites roughly 24 hours for robots.txt changes to take effect, and Perplexity says similar.
- Confirm exact tokens. Anthropic’s page shows its robots.txt example only for ClaudeBot, so check each vendor’s current documentation before writing rules for the others.
Sarkar SEO treats this audit as the first step of any AI search project, because it is cheap, binary and often reveals a problem no content upgrade could solve.
What Access Does and Doesn’t Guarantee
Being crawlable makes you eligible. It does not make you cited. Once the door is open, the usual work applies: clear, accurate, original pages from a source people can verify. Sarkar SEO pairs the access audit with a content review so the pages bots can now reach are worth citing.
