AI SEO and Your robots.txt: Who Should Be Allowed to Read Your Site?

Before an AI system can cite your page, it has to be allowed to read it. That sounds obvious, yet many sites now block AI crawlers through a template, a plugin or a copied snippet, without anyone deciding to. The result can be invisible: nothing breaks, traffic looks normal and your brand quietly drops out of AI answers.

This is the least glamorous part of AI SEO, and one of the easiest to get wrong. Here is how to make the decision on purpose.

Not All AI Bots Do the Same Job

The core idea is that AI companies run different crawlers for different purposes. Industry guides sort them into two groups.

Training crawlers collect content to build or improve models. Examples named across sources include GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) and Google-Extended.

Search and retrieval crawlers index or fetch pages so answers can cite them. Examples include OAI-SearchBot (ChatGPT search), Claude-SearchBot and PerplexityBot. Some assistants also use user-triggered fetchers, such as ChatGPT-User, when a person asks about a specific page.

These roles are separate, so the permissions are separate. Blocking a training bot is a statement about how your content may be used to build models. Blocking a search bot can remove you from the answers themselves. Several guides warn that blocking GPTBot without allowing OAI-SearchBot would still keep you out of ChatGPT search, so each bot needs its own rule.

Google Is a Special Case

Google’s setup trips people up most often. According to the guides I reviewed, Google-Extended controls whether your content is used for Gemini training and related AI products. It does not control Google Search crawling or rankings, which belong to Googlebot.

That matters for AI Overviews. Google’s own documentation says AI Overviews and AI Mode draw on its normal search systems, and that preview controls such as snippet restrictions can limit how your content appears. So if you block Googlebot, or apply a strict “nosnippet” rule to pages you want surfaced, you can disappear from AI features even though Google-Extended is untouched. Aleyda Solis’s checklist flags the nosnippet risk specifically.

Three Policies, and Who They Suit

Think of your choice as one of three positions.

1. Open. Allow every crawler. This suits businesses that live on discovery: local services, ecommerce, consultants and publishers whose revenue doesn’t depend on exclusivity. You maximize your chance of being cited, and accept that your content may also feed model training.
2. Selective. Block training crawlers, allow search crawlers. Several sources call this the most commonly recommended approach. It says “don’t train on my work, but cite me when someone asks.
3. Closed. Block everything. This fits paywalled research, proprietary data products and sites whose content is the product. You protect your material, but you give up AI citation visibility. For most marketing sites, that trade is hard to justify.

Which one is right depends on how you earn money. If a stranger reading your answer in a chatbot is a lead, closed is expensive. If that same stranger replaces a paying subscriber, closed may be rational.

Limits Worth Knowing

• robots.txt is a request, not a lock. Well-behaved crawlers honor it, but nothing technically enforces it. One guide notes that some bots are accused of not fully complying, and Perplexity has faced such accusations. If content is truly sensitive, put it behind authentication.
• Bot names change. New crawlers appear and existing ones are renamed. Before editing rules, check the current documentation published by each AI company.
• Newer signals are emerging. One guide describes an IETF effort called Content-Signal, which would let sites declare separate permissions for search, AI input and AI training. It is a proposal, not a universal standard, so don’t rely on it alone.
• llms.txt is unproven. One study cited in earlier GEO research found no citation benefit from it, and Google’s guidance says no special files are required. Don’t treat it as a substitute for access rules.

An Audit You Can Finish Today

At Sarkar SEO, we start AI visibility reviews with access, since optimizing pages no bot can fetch is wasted work. Try this sequence:

1. Read your live robots.txt. Open yourdomain.com/robots.txt and look for blanket “Disallow: /” rules, or lists of AI bots added by a security plugin or hosting provider.
2. Check your CDN and firewall. Bot-protection tools can block crawlers before robots.txt is even read. Review their settings and logs for AI user agents.
3. Scan page-level directives. Look for noindex tags and nosnippet or max-snippet limits on pages meant to be cited.
4. Confirm server responses. Pages should return normal status codes to crawlers, not challenges, redirects or empty rendered content. Content that appears only after heavy client-side scripting may not be read by every bot.
5. Write your policy down. Record which category you chose and why, so the next developer doesn’t undo it.
6. Review quarterly. Crawler lists and company policies change often enough to justify a regular look.

A Common Misreading

Some teams hear “block AI” and assume it protects rankings or privacy. It protects neither automatically. Blocking training bots has no effect on Google rankings, according to the guides, and it does nothing about content that is already public and copied elsewhere. It is a licensing preference, not a security control.

The opposite mistake is also real: assuming that allowing crawlers guarantees citations. Access is only the entry requirement. Whether you get cited still depends on relevance, clarity and trust.