AI Crawlers Explained, from GPTBot to PerplexityBot

Every AI crawler that matters in 2026, what each one does, its robots.txt token, and the allow/block decision that quietly decides your AI visibility.

TopicHow AI Reads the Web

AI crawlers fall into three kinds, training crawlers (GPTBot, ClaudeBot, Google-Extended) that feed model training, search crawlers (OAI-SearchBot, PerplexityBot) that index you for AI answers, and live fetchers (ChatGPT-User) that read pages on demand. Blocking the search and fetch kinds removes you from AI answers. Check robots.txt before anything else.

The crawler roster, 2026

User agentOperatorWhat it doesBlock it and...
GPTBotOpenAICollects content for model trainingFuture models know less about you. Citations unaffected directly
OAI-SearchBotOpenAIBuilds the index behind ChatGPT searchYou drop out of ChatGPT's retrievable web
ChatGPT-UserOpenAIFetches pages live when a user's chat needs themChatGPT cannot read or quote your pages on demand
ClaudeBotAnthropicTraining and index crawling for ClaudeReduced presence in Claude answers
PerplexityBotPerplexityIndexes the web for Perplexity answersYou disappear from Perplexity citations
Google-ExtendedGoogleOpt-out token for Gemini training (not a crawler itself)Gemini training opt-out. Search and AI Overviews unaffected
BingbotMicrosoftClassic search index, which ChatGPT retrieval leans onYou vanish from Bing and much of ChatGPT retrieval

Training vs retrieval, the distinction that changes the decision

The publishing industry's "block the bots" reflex conflates two different things. Blocking training crawlers is a copyright stance with little visibility cost. Blocking retrieval crawlers and live fetchers is a marketing decision. It removes you from the answers your buyers are reading. A business that wants AI engines to recommend it accurately should allow retrieval even if it opts out of training.

The half-measure that backfires is blocking everything while competitors stay open. Engines still answer the question. They just cite someone else, or describe you from stale third-party data you cannot correct.

A sensible robots.txt posture for a commercial site

  • Allow OAI-SearchBot, ChatGPT-User, PerplexityBot and ClaudeBot. These are how engines read and cite you.
  • Decide GPTBot and Google-Extended on principle, not panic. They affect training, not today's answers.
  • Never block Bingbot. Verify in Bing Webmaster Tools and submit your sitemap, since ChatGPT retrieval rides on Bing.
  • Check your CDN and security layer too. Cloudflare-style "block AI bots" toggles override a permissive robots.txt.
  • Re-test quarterly. New agents appear, and one plugin update can silently rewrite your rules.

How to verify what crawlers actually see

  1. Fetch yourdomain.com/robots.txt and read it. The number of firms who have never looked is remarkable.
  2. Grep server or CDN logs for the user agents above. Absence of OAI-SearchBot over a month is a finding.
  3. Search site:yourdomain.com in Bing. Thin coverage there throttles ChatGPT retrieval.
  4. Ask ChatGPT (with search enabled) to summarise a specific page URL. A refusal or garbled summary points to fetch blocking.
  5. Check that your main content renders server-side. Several AI fetchers execute little or no JavaScript.

Common questions

Do AI crawlers respect robots.txt?

The major named agents (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended) document compliance and observably honour it. Disreputable scrapers do not, but your policy should be set for the engines that send buyers, not the noise.

Will allowing AI crawlers hurt my SEO?

No. AI agents are separate from Googlebot and Bingbot, and allowing them has no ranking effect. The only SEO-adjacent risk is server load, which is negligible for normal sites.

Should I block AI bots to protect my content?

If you sell content (publishers, course creators), a training block is a defensible commercial position. If you sell services or products, your content is marketing. Blocking the engines that quote it is blocking the channel.

Does llms.txt control AI crawlers?

No, llms.txt is a curation suggestion no major engine confirms reading. robots.txt is the file with teeth. Get that right first.

Keep reading

Put this to work on your business

We build exactly this, SEO, AI visibility, PPC, email and automation, as one measured system. Talk it through with a real person, no obligation.

Book a free strategy call or take the two minute scorecard