Before an AI system can cite your site, it has to reach it. That sounds basic, yet many sites block the very crawlers that feed AI search, often by accident through a security setting or a copied robots.txt file. This guide covers the technical side of AI SEO: what each major crawler does, how to set rules that match your business, and what the evidence says about the much-discussed llms.txt file.
One Company, Several Crawlers
The first mistake is treating “AI bots” as a single group. OpenAI’s documentation shows why. It lists separate agents with separate jobs:
- OAI-SearchBot surfaces sites in ChatGPT’s search features. OpenAI says sites that opt out will not appear in ChatGPT search answers, though they can still show as navigational links.
- GPTBot collects content that may be used to train OpenAI’s foundation models. Disallowing it signals that your content should not be used for training.
- ChatGPT-User visits a page when a person asks ChatGPT a question that needs it. OpenAI notes that because these visits are user-initiated, robots.txt rules may not apply.
Each robots.txt setting is independent, so you can allow OAI-SearchBot and disallow GPTBot. That combination lets your pages appear in ChatGPT search while opting out of training use. OpenAI says search changes can take about 24 hours to apply after you update the file. Other AI companies run their own crawler families with similar splits, so read each vendor’s documentation instead of assuming the pattern holds.
Why the Trade-Off Is Real
Crawling costs money, and the return varies. Cloudflare publishes a crawl-to-referral ratio, which compares how many pages a platform’s crawlers fetch with how many visitors it sends back. According to PPC Land’s summary of Cloudflare data, between January and July 2025 OpenAI’s ratio moved from 1,217:1 to 1,091:1, Perplexity’s rose from 54.6:1 to 194.8:1, and Google’s settled around 5.4:1.
Treat those numbers carefully. Cloudflare itself says some apps and browsers send no referrer, so published ratios are best read as upper bounds. Training crawlers also generate no referrals by design, so judging them on referrals is unfair to their purpose. And one site’s logs can differ sharply from the network average. The ratios show a direction, not a precise price.
A Decision Guide by Business Type
The matrix below is Sarkar SEO’s working judgment, not research data. Adjust it to your legal position and goals.
The logic is that being mentioned in answers is a visibility channel, while contributing to training is a separate decision about your content’s value. Many businesses will make different choices for each.
What the Evidence Says About llms.txt
The llms.txt file is a proposed plain-text index meant to help AI systems read a site. It is attractive because it is easy to add. The available data is not encouraging.
An article from GEO agency Something Inc. summarizes two 2026 studies. SE Ranking scanned 300,000 domains, found the file on about 10% of them, and saw no clear, consistent citation lift for sites that had it. Trakkr scanned 37,894 domains and also found no measurable citation advantage. Neither study found that the file hurts. The author, who had previously recommended llms.txt to clients, now treats it as inert and points to crawler access, page structure and outside corroboration instead.
Two caveats apply. This is a secondary summary from an agency, and the article cites no statement from Google or an AI company confirming how crawlers treat the file. Absence of a measured lift is not proof of uselessness either. A sensible stance is to add one only if it costs you almost nothing, and never to let it displace real work.
A Five-Point Crawl Audit
- Read your robots.txt line by line. Look for broad disallow rules inherited from a template or a staging site.
- Check your CDN and firewall. Security tools can block bots at the network level, which robots.txt never shows. Test with a real request, not just a settings page.
- Review server logs. Logs show who actually visits, how often, and which pages they request. Analytics scripts often miss bot traffic entirely.
- Verify the visitor is genuine. User-agent names can be faked. OpenAI publishes IP ranges for its search bot, so allow those rather than trusting a name alone.
- View a key page without JavaScript. If your main content only appears after scripts run, some crawlers may see an empty page. Serve important text in the initial HTML where you can.
Fixing Access Does Not Create Visibility
Access is a prerequisite, not a strategy. A crawler that can read your pages still needs a reason to cite them: a direct answer, a clear source, and a reputation beyond your own site. Sarkar SEO sees crawl access as the cheapest item on the AI SEO list, which is exactly why it should be checked first.
Maintenance Habits
Review the setup quarterly, because vendors add and rename bots. Keep a dated note of every robots.txt change so you can connect shifts in traffic to specific decisions. After any redesign, migration or security upgrade, repeat the audit above.
