TopicHow AI Reads the Web
AI crawlers fall into three kinds, training crawlers (GPTBot, ClaudeBot, Google-Extended) that feed model training, search crawlers (OAI-SearchBot, PerplexityBot) that index you for AI answers, and live fetchers (ChatGPT-User) that read pages on demand. Blocking the search and fetch kinds removes you from AI answers. Check robots.txt before anything else.
The crawler roster, 2026
| User agent | Operator | What it does | Block it and... |
|---|---|---|---|
GPTBot | OpenAI | Collects content for model training | Future models know less about you. Citations unaffected directly |
OAI-SearchBot | OpenAI | Builds the index behind ChatGPT search | You drop out of ChatGPT's retrievable web |
ChatGPT-User | OpenAI | Fetches pages live when a user's chat needs them | ChatGPT cannot read or quote your pages on demand |
ClaudeBot | Anthropic | Training and index crawling for Claude | Reduced presence in Claude answers |
PerplexityBot | Perplexity | Indexes the web for Perplexity answers | You disappear from Perplexity citations |
Google-Extended | Opt-out token for Gemini training (not a crawler itself) | Gemini training opt-out. Search and AI Overviews unaffected | |
Bingbot | Microsoft | Classic search index, which ChatGPT retrieval leans on | You vanish from Bing and much of ChatGPT retrieval |
Training vs retrieval, the distinction that changes the decision
The publishing industry's "block the bots" reflex conflates two different things. Blocking training crawlers is a copyright stance with little visibility cost. Blocking retrieval crawlers and live fetchers is a marketing decision. It removes you from the answers your buyers are reading. A business that wants AI engines to recommend it accurately should allow retrieval even if it opts out of training.
The half-measure that backfires is blocking everything while competitors stay open. Engines still answer the question. They just cite someone else, or describe you from stale third-party data you cannot correct.
A sensible robots.txt posture for a commercial site
- Allow
OAI-SearchBot,ChatGPT-User,PerplexityBotandClaudeBot. These are how engines read and cite you. - Decide
GPTBotandGoogle-Extendedon principle, not panic. They affect training, not today's answers. - Never block
Bingbot. Verify in Bing Webmaster Tools and submit your sitemap, since ChatGPT retrieval rides on Bing. - Check your CDN and security layer too. Cloudflare-style "block AI bots" toggles override a permissive robots.txt.
- Re-test quarterly. New agents appear, and one plugin update can silently rewrite your rules.
How to verify what crawlers actually see
- Fetch
yourdomain.com/robots.txtand read it. The number of firms who have never looked is remarkable. - Grep server or CDN logs for the user agents above. Absence of
OAI-SearchBotover a month is a finding. - Search
site:yourdomain.comin Bing. Thin coverage there throttles ChatGPT retrieval. - Ask ChatGPT (with search enabled) to summarise a specific page URL. A refusal or garbled summary points to fetch blocking.
- Check that your main content renders server-side. Several AI fetchers execute little or no JavaScript.
Common questions
Do AI crawlers respect robots.txt?
The major named agents (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended) document compliance and observably honour it. Disreputable scrapers do not, but your policy should be set for the engines that send buyers, not the noise.
Will allowing AI crawlers hurt my SEO?
No. AI agents are separate from Googlebot and Bingbot, and allowing them has no ranking effect. The only SEO-adjacent risk is server load, which is negligible for normal sites.
Should I block AI bots to protect my content?
If you sell content (publishers, course creators), a training block is a defensible commercial position. If you sell services or products, your content is marketing. Blocking the engines that quote it is blocking the channel.
Does llms.txt control AI crawlers?
No, llms.txt is a curation suggestion no major engine confirms reading. robots.txt is the file with teeth. Get that right first.
Keep reading
Put this to work on your business
We build exactly this, SEO, AI visibility, PPC, email and automation, as one measured system. Talk it through with a real person, no obligation.
