AI Crawler Directory
Nearly every piece of robots.txt advice about AI bots treats them as one category, which is why so much of it is wrong. They do three different jobs. One trains future models, one builds the index an assistant searches at answer time, and one fetches a single page because a human asked it to. Block the first and you protect your content. Block the second and you delete yourself from AI answers while believing you did something clever.
Every entry below links to the operator's own documentation, so you can check me. Verified against those sources on November 1, 2026. Tokens change and operators add bots without announcing them, so treat this as a snapshot and read your own server logs before acting on it.
The three jobs
Collects pages to train future models. Blocking it keeps your content out of training data and costs you nothing in citations, because this is not the bot that fetches you at answer time.
Builds the retrieval index the assistant searches when answering. Blocking it removes you from that product's answers. This is the one people block by accident.
Fetches one page because a human pasted a link or asked a question that needs it. Both OpenAI and Perplexity document that these fetchers may ignore robots.txt, on the reasoning that a person asked for the page. Worth knowing before you rely on a Disallow line here.
Almost every regrettable robots.txt line comes from treating an indexing bot like a training bot. If you take one thing from this page, take that. The full user agent string reference has the exact tokens, and a broader primer on how AI crawlers work covers the mechanics underneath.
The directory
GPTBot
OpenAITrainingUser-agent: GPTBot
The training crawler. OpenAI's own documentation says disallowing it indicates your content should not be used to train their foundation models. It says nothing about removing you from ChatGPT's answers, because that is a different bot. This is the single most expensive mix-up in AI robots.txt advice.
OAI-SearchBot
OpenAIIndexingUser-agent: OAI-SearchBot
Surfaces sites in ChatGPT's search features. OpenAI states plainly that blocking it means your site will not appear in ChatGPT search answers. If you want to be citable in ChatGPT, leave this one open.
ChatGPT-User
OpenAIFetch on demandUser-agent: ChatGPT-User
Fires when someone pastes your URL into ChatGPT or asks it to open a page. OpenAI documents that it does not crawl automatically, and that because a user initiated the action, robots.txt rules may not apply.
OAI-AdsBot
OpenAIFetch on demandUser-agent: OAI-AdsBot
Checks the safety of pages submitted as ads on ChatGPT. It only visits pages that were submitted, so for most sites it never shows up at all. Included because it exists and almost no directory lists it.
ClaudeBot
AnthropicTrainingUser-agent: ClaudeBot
Anthropic's training crawler. Same logic as GPTBot: restricting it is a training-data decision, not a visibility one.
Claude-SearchBot
AnthropicIndexingUser-agent: Claude-SearchBot
Indexes content to improve Claude's search results. Anthropic's wording is direct: disabling it prevents your content being indexed for search. This is Anthropic's equivalent of OAI-SearchBot, and blocking ClaudeBot while leaving this open is the sane default for most publishers.
Claude-User
AnthropicFetch on demandUser-agent: Claude-User
Fetches a page when a Claude user asks a question that needs it.
Google-Extended
GoogleTrainingUser-agent: Google-Extended
Not a crawler. Google documents that it has no user agent string of its own: crawling happens with normal Google agents and this token is used purely as a control. It governs training of Gemini models AND grounding, which Google defines as supplying content from the Search index to the model at prompt time. Google states it does not affect inclusion in Google Search and is not a ranking signal. Google has also said it intends to support the AIPREF training preference signal in early 2027.
Googlebot
GoogleIndexingUser-agent: Googlebot
Serves Search and AI Overviews from the same index. There is no directive that keeps you in blue links while opting out of AI summaries. If someone tells you otherwise, ask them to name the directive.
PerplexityBot
PerplexityIndexingUser-agent: PerplexityBot
Surfaces and links sites in Perplexity results. Perplexity documents explicitly that it is not used to crawl content for foundation models, and publishes no training crawler at all. So unlike OpenAI, Google and Anthropic, there is no training-versus-indexing trade to make here: blocking anything Perplexity runs only costs you citations.
Perplexity-User
PerplexityFetch on demandUser-agent: Perplexity-User
Fetches a page when a user's question requires it. Perplexity's docs say that since a user requested the fetch, this one generally ignores robots.txt.
Bingbot
MicrosoftIndexingUser-agent: bingbot
Copilot answers are built on Bing's index, which means Bing SEO is Copilot AEO. The least glamorous entry here and one of the more useful ones.
Applebot
AppleIndexingUser-agent: Applebot
Crawls for Apple search surfaces including Spotlight, Siri and Safari.
Applebot-Extended
AppleTrainingUser-agent: Applebot-Extended
The training opt-out, and a clean example of the pattern this page is about. Apple documents that Applebot-Extended does not crawl anything: it only decides how data already collected by Applebot may be used. Opt out and your content stays discoverable through Spotlight, Siri and Safari, and the rule is not considered in Search ranking.
Also in your logs, not checked here
These show up often enough to be worth naming, but they were not verified against operator documentation when this page was last checked, so they are listed separately rather than quietly mixed in with the sourced entries.
CCBot Common Crawl
Public corpus that many models ingest second-hand.
Bytespider ByteDance
Widely reported as a heavy crawler. Check your origin logs.
Meta-ExternalAgent Meta
Separate from the fetcher that builds link previews.
Amazonbot Amazon
Feeds Alexa and Amazon assistant surfaces.
Questions people actually ask
Does blocking GPTBot remove me from ChatGPT?+
No, and this is the costliest misunderstanding in AI SEO. GPTBot collects training data. ChatGPT search retrieves through OAI-SearchBot. Block GPTBot and you stay out of future training runs while remaining citable. Block OAI-SearchBot and you disappear from the answers, which is almost never what anyone intended.
Writing robots.txt rules for AI crawlers →Can I stay in Google Search but opt out of AI Overviews?+
No. AI Overviews are generated from the same Search index Googlebot builds. Google-Extended governs Gemini training and grounding, and Google states it does not affect Search inclusion or ranking. The only exit from AI Overviews is an exit from Search, which is not a trade worth taking.
AI Overviews and shopping results →Does robots.txt actually stop a crawler?+
robots.txt is a request, not a fence. Well-behaved crawlers honor it, and the user-initiated fetchers documented above may ignore it by design. Anything determined to ignore it will, and your robots file will never tell you. Enforcement happens at the network layer.
Blocking AI crawlers at the edge with Cloudflare →Does llms.txt do anything?+
Very little so far. Most llms.txt files are never fetched, Google has said it ignores them, and Shopify now generates one for every store automatically, which tells you roughly how much weight it carries.
Does llms.txt do anything for Shopify? →How do I find out which bots are actually hitting my site?+
Read your server logs instead of trusting this or any other directory. Lists go stale, your logs do not. Group requests by user agent, then compare that against what your robots.txt currently allows. The gap between the two is usually where the surprise lives.
Auditing which AI bots you are blocking →How do I tell whether any of this is working?+
Allowing a crawler is not the same as being cited. The measurable version is checking how often assistants actually name you in answers to questions you care about, then watching that over time. Logs tell you who visited. Citation checks tell you whether it mattered.
How to measure AI visibility →If you sell things
Everything above is about access, which is the easy half. The harder half is that assistants read structured product data rather than prose, and most catalogs fail on attributes rather than on crawling. Worth reading next: answer engine optimization for Shopify, what ChatGPT Shopping requires of your product feed, and how Google AI Mode treats Shopify catalogs. If you would rather not audit a catalog by hand, that is what Entitled does. I built it, so read that recommendation accordingly.
More on how I think about this in the apps I build and over on the essays.