AI SEO is usually discussed as a content problem, but it begins with a technical one: can the right systems reach your pages at all? Your robots.txt file, your rendering setup and your search settings decide whether your content can appear in AI SEO answers, and whether it can be used to train models you may never see credited.
The short answer: separate AI bots by purpose, allow the ones that send readers, decide deliberately about training, and never rely on one file or one setting to do the whole job.
Three Kinds of AI Bots, Three Different Decisions
Treating every AI bot alike is the most common mistake. Most major operators run up to three kinds of access.
Training crawlers collect data for future models. Examples are OpenAI’s GPTBot and Anthropic’s ClaudeBot.
Search crawlers build the index behind an assistant’s answers. OpenAI’s OAI-SearchBot, Anthropic’s Claude-SearchBot and PerplexityBot belong here.
User-triggered fetchers visit a page because someone asked an assistant to read it. ChatGPT-User, Claude-User and Perplexity-User fit this group.
The consequence is practical. Blocking GPTBot does not remove you from ChatGPT Search, because OAI-SearchBot is separate. Blocking the search crawler while allowing the training crawler does the opposite of what most site owners want: your content is available for training but absent from answers.
Google works differently. Google-Extended is a robots.txt token controlling certain Gemini training and grounding uses. It does not control AI Overviews or AI Mode, which run through normal Search systems.
What Robots.txt Can and Cannot Do
Robots.txt is a policy signal, not a lock. Anthropic states that all three of its bots honor it. OpenAI’s position on user-initiated visits is looser, and robots.txt may not apply to them. Cloudflare reported in August 2025 that Perplexity accessed content through rotating IP addresses and generic browser identities after declared crawlers were blocked, and delisted it from its verified-bot list. That is one vendor’s allegation, but it explains why security-minded sites pair robots.txt with firewall rules.
One more trap: your host or CDN may block bots by default. Test what your live server returns to each user agent, not what your robots.txt file says. At Sarkar SEO, we start every technical review with that live test, because a clean file and a blocked response coexist more often than people expect.
Do Not Expect Much From llms.txt
The llms.txt file is a proposed plain-text guide for AI systems. Adoption has surged, but use has not. Originality.ai counted 36,120 sites with the file by May 2026, up from 4,088 a year earlier. Yet Ahrefs, analyzing server logs from 137,000 domains, found 97% of those files received zero requests that month. Retrieval bots made only 1.1% of requests, while SEO audit tools were the biggest visitors.
Google’s guidance says the file will not help or hurt visibility in Search, and a June 2026 note adds that it is not required. The sensible stance is low priority: harmless if cheap, pointless as a strategy. Note that these figures come from secondary reports of the two studies, and the methods are only briefly described.
Rendering: The Quiet Reason Pages Go Unseen
Reports on AI crawler behavior suggest many do not execute JavaScript. If your key content loads only after scripts run, a crawler may see an empty shell. Check the raw HTML response of important pages. If your headings, main text and product details are missing from it, move that content to server-side rendering. This is a hypothesis worth testing on your own site, since crawler behavior varies and changes.
The Trade: Visits Given Versus Pages Taken
Cloudflare’s crawl-to-referral ratio compares pages crawled with visitors referred back. In 2025 data, Google ran at roughly 5:1 by July, OpenAI near 1,091:1, and Anthropic at about 38,065:1 after a sharp decline from far higher. Perplexity was much lower, though rising. Cloudflare notes some apps send no referral information, so these ratios are upper bounds.
Use this to decide with eyes open. A bot that takes thousands of pages per visitor may still deliver brand exposure through citations, but you cannot count on traffic. If server load or content value is your concern, rate limits and selective blocking are legitimate choices.
Google’s Opt-Out: A Decision, Not a Default
In mid-2026 Google began rolling out a Search Console control, following a UK regulatory requirement, that lets sites exclude content from AI Overviews, AI Mode and Discover’s generative features. Google says it is not a ranking signal. The catch is measurement: the generative AI report shows impressions only, combined across features, with no clicks or queries, so you cannot easily price what opting out costs. Click data is expected later under UK requirements. The nosnippet directive also removes content from AI Overviews, but it strips traditional snippets too.
For most commercial sites, staying included is the stronger default, because that is where attention is moving. Publishers with licensing or subscription concerns have a different calculation.
A Short Decision Framework
- List the bots that matter, grouped as training, search and user-triggered.
- Allow search crawlers for assistants where you want citations.
- Choose your training stance on business grounds, not fear.
- Verify live server responses and raw HTML for each important template.
- Review monthly, because bot names, policies and controls keep changing. Sarkar SEO suggests keeping a one-page record of every bot rule and the reason behind it.
