Before an AI system can cite your page, something has to fetch it. That makes crawler access the least glamorous and most easily broken part of AI SEO. One misplaced line in robots.txt can keep you out of ChatGPT’s search answers while leaving the door open to bots you never meant to feed.
The principle is simple: AI companies run different bots for different jobs, and each job deserves its own decision. Sarkar SEO treats this as the first check in any AI visibility review, because no amount of good writing helps a page nobody can retrieve.
Three Bots, Three Jobs
AI crawlers fall into three groups. Training crawlers collect content that may be used to build or improve models. Search crawlers build the index behind an assistant’s web answers. User-triggered fetchers visit a page because a person just asked about it.
OpenAI documents this split clearly. OAI-SearchBot surfaces sites in ChatGPT’s search features. GPTBot crawls content that may be used to train its foundation models. ChatGPT-User handles actions a user starts, such as opening a page during a conversation. OpenAI says each setting is independent, so a site can allow OAI-SearchBot while disallowing GPTBot.
Other vendors follow a similar pattern. One 2026 industry guide lists ClaudeBot for training, Claude-SearchBot for search and Claude-User for on-demand requests, with PerplexityBot and Perplexity-User in the search and on-demand roles. Treat that as a starting map and confirm names in each vendor’s own documentation, because user-agent names change.
What the Documentation Actually Says
Two details in primary documentation are easy to miss.
First, OpenAI states that a site opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though it may still appear as a navigational link. Blocking is therefore a visibility decision, not a neutral privacy setting. OpenAI also says robots.txt changes can take about 24 hours to take effect for search results, so do not judge an edit after an hour. Because ChatGPT-User actions are initiated by people, robots.txt rules may not apply to them.
Second, Google-Extended is a robots.txt token rather than a separate crawler. Google says it controls whether content Google crawls can be used to train future Gemini models, and that it also covers grounding, where Search index content is supplied to Gemini at prompt time. Google states that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” That page does not discuss AI Overviews, so do not assume the token governs them. Check Google’s current documentation before relying on it.
The training row is a genuine trade-off. Some argue that allowing training helps models recognise your brand without searching; that is plausible but unproven in anything I reviewed. Blocking protects content you consider proprietary. Publishers with paid or licensed material usually have stronger reasons to block than a service business seeking maximum exposure. Decide on principle, not on promised gains.
Remember that robots.txt is a request, not a lock. Bots that ignore it have to be stopped at the firewall or CDN level, which acts before robots.txt is read.
The llms.txt Question
llms.txt is a proposed file, first published in September 2024, that offers models a Markdown guide to your site. It is often sold as an AI SEO shortcut. The evidence does not support that.
According to an October 2026 review, Google says its search systems do not use such files. The same review reports that a June 2026 Ahrefs analysis of 137,210 domains found 28% published one, and 97% of those files received no requests that month. Semrush and SE Ranking reportedly found no correlation with AI citations either. These are vendor studies of their own data, and the review’s publisher sells related services, so read the figures as indicative, not definitive.
The practical conclusion: the file costs little and may help coding tools, but it should never take time from crawl access or content quality.
Four Checks Before You Edit Anything
- Open your live robots.txt and list every AI-related user agent it mentions.
- Look for blanket rules under “User-agent: *” with broad disallows, which can catch search bots by accident.
- Review firewall and CDN settings, since a security rule can silently block a bot that robots.txt allows.
- A week after any change, ask assistants questions your pages answer and record whether you are cited.
Sarkar SEO advises keeping a dated log of each edit. Without one, you cannot tell whether a change in visibility came from your policy, your content or the assistant’s own updates.
The Limits of Access Control
Access is necessary, not sufficient. Allowing every search bot gets a page into the candidate pool; it does not earn a citation. Assistants also differ in how they retrieve and whom they favour, so a clean robots.txt is a floor, not a strategy.
Nor is this a one-time job. Vendors add and rename bots, and their policies on training and search keep shifting. Review the file quarterly, and whenever a vendor announces a new crawler.
Make the Gate Work for You
The best AI SEO policy is one you chose deliberately and can explain in a sentence: who may read your pages, for what purpose, and why. Many sites have never written it down, and some are invisible to assistants by accident.
If you are unsure what your current file is doing, Sarkar SEO can audit your crawler rules, flag accidental blocks and help you set a policy that fits your business. Start with the four checks above, then build the content worth being found.
