AI Crawlers in robots.txt: Which Ones Exist, What They Do and How to Decide

Three kinds of AI crawler

Talk of "AI crawlers" lumps together three very different things. The large providers now separate them, each with its own identifier (user agent token) in robots.txt:

So the basic question is not "AI, yes or no", but: do you want to appear as a source in AI answers, and do you want your text to go into models? Most providers let you control the two separately.

Important first: robots.txt is a request, not a barrier. Google itself writes that reputable crawlers obey it but others might not, and that it is not a mechanism for keeping a page out of Google (Google Search Central: Introduction to robots.txt, accessed 30 October 2026).

The crawlers in detail

All details come from the providers' documentation, accessed 30 October 2026.

OpenAI: OAI-SearchBot, ChatGPT-User, GPTBot

OpenAI writes: "Each setting is independent of the others." (OpenAI: Overview of OpenAI Crawlers, accessed 30 October 2026)

• OAI-SearchBot is responsible for search in ChatGPT. Blocked pages do not appear in search answers, at most as a plain navigation link. Changes take effect after about 24 hours.
• ChatGPT-User visits pages when users trigger something: "Because these actions are initiated by a user, robots.txt rules may not apply."
• GPTBot collects content that may be used for training. A block means: no training of generative foundation models on this content.

Anthropic: Claude-SearchBot, Claude-User, ClaudeBot

Anthropic also separates search, user fetches and training (Anthropic Help Center, accessed 30 October 2026):

• Claude-SearchBot: a block may reduce visibility and accuracy in search results.
• Claude-User: a block prevents fetches when users ask Claude something.
• ClaudeBot: a block signals that future content should not go into training.

According to Anthropic, all its bots respect robots.txt, the block must be set on each subdomain, and blocking by IP address is unreliable because the bots can then no longer read robots.txt.

Perplexity: PerplexityBot and Perplexity-User

PerplexityBot shows and links websites in search results and, according to Perplexity, is not used to train foundation models. Perplexity-User fetches on user request: "this fetcher generally ignores robots.txt rules." (Perplexity: Perplexity Crawlers, accessed 30 October 2026)

Google: Google-Extended is not Google Search

Google-Extended is not a separate crawler but a control token: it governs whether content is used to train future Gemini models and for grounding in Gemini Apps and on Vertex AI. "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." (Google Search Central: Google's common crawlers, accessed 30 October 2026)

AI Overviews and AI Mode in Google Search, by contrast, are governed by the Googlebot rules and the snippet controls nosnippet, data-nosnippet, max-snippet or noindex (Google Search Central: AI features and your website, accessed 30 October 2026). Since 2026 there has also been a Search Console setting that takes the website out of these AI features; according to Google, it is not a ranking signal for regular search (Google Blog, accessed 30 October 2026).

Apple: Applebot-Extended

"Applebot-Extended does not crawl webpages." It only governs whether content that Applebot fetches anyway may be used to train Apple's foundation models; Apple's search is not affected. If your robots.txt does not mention Applebot, it follows the rules for Googlebot (Apple Support: About Applebot, accessed 30 October 2026).

Microsoft Bing and Copilot: no separate AI token

Microsoft uses robots meta tags instead of its own token: noarchive keeps a page out of chat answers and training, nocache allows only the URL, title and snippet (Bing Webmaster Blog, September 2023, accessed 30 October 2026). The post still refers to "Bing Chat"; whether today's guidelines say the same for Copilot could not be checked against the original. If you block bingbot, you block Bing search as well.

Common Crawl and Meta

CCBot builds Common Crawl's freely accessible archive (Common Crawl: CCBot, accessed 30 October 2026). Common Crawl itself does not mention AI training, but anyone can reuse the archive; we therefore class CCBot with the training crawlers. According to Meta, meta-externalagent crawls "for use cases such as training foundation AI models or improving products"; the fetcher meta-externalfetcher, Meta says, may bypass robots.txt (Meta for Developers: Meta Web Crawlers, accessed 30 October 2026).

Overview: 13 identifiers and what a block does

Helferlain's SEO audit checks these same 13 identifiers (list as of 28 October 2026).

Three stances with example entries

The examples assume a website that blocks an internal area under /internal/ for all crawlers. Replace the paths and domain with your own.

Stance 1: allow everything

You do not need separate groups for AI crawlers. Any crawler without its own group follows the group with User-agent: *.

User-agent: *
Disallow: /internal/

Sitemap: https://www.example.com/sitemap.xml

Stance 2: block training, allow search and user fetches

The training identifiers go into one shared group. Under RFC 9309, several User-agent lines above the same rules are permitted. Important: the group must also contain all the blocks that would otherwise sit under *, here /internal/. With Disallow: /, that happens automatically.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

User-agent: *
Disallow: /internal/

Sitemap: https://www.example.com/sitemap.xml

With Google-Extended, note that the block affects grounding in Gemini Apps as well as training. If visibility in Gemini matters to you, allow Google-Extended.

Stance 3: block all AI crawlers, keep classic search

This adds the AI services' search and user identifiers too. Googlebot and bingbot stay allowed, otherwise you disappear from Google and Bing.

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

User-agent: *
Disallow: /internal/

Sitemap: https://www.example.com/sitemap.xml

This stance has three limits. According to their providers, ChatGPT-User and Perplexity-User do not necessarily follow robots.txt. AI Overviews and AI Mode in Google cannot be switched off via robots.txt without also losing Google Search; for that, there are the Search Console setting and the snippet rules. The same applies to Copilot with noarchive in the robots meta tag. Really blocking everything, with User-agent: * and Disallow: /, removes you from every search engine that follows the rules.

Common mistakes

A group of its own overrides the star group

This mistake is hard to spot when reading the file. Under RFC 9309, a crawler looks for the group whose User-agent matches its identifier, ignoring case. If several groups match, they are combined. The group with * only applies if no matching group exists (RFC 9309, section 2.2.1, accessed 30 October 2026).

A common example:

User-agent: *
Disallow: /internal/
Disallow: /enquiries/

User-agent: GPTBot
Allow: /

The intent: "GPTBot is explicitly welcome." In fact, GPTBot may now also read /internal/ and /enquiries/, because only its own group applies to it. The fix: repeat the blocks from the star group in every separate group, or drop the separate group if it sets nothing beyond *.

Outdated or missing identifiers

Many robots.txt files still contain anthropic-ai or Claude-Web. Anthropic's current help page names only ClaudeBot, Claude-SearchBot and Claude-User; the two older identifiers no longer appear there. If you once blocked Anthropic, you should switch the rules to the three current identifiers. Conversely, older files often lack the newer search and user identifiers entirely.

Confusing Google-Extended with Google Search

Blocking Google-Extended will not get you out of AI Overviews; according to Google, Google-Extended does not affect Google Search. Blocking Googlebot instead loses you Google Search completely.

Googlebot rules also apply to Apple

If you have a group of its own for Googlebot but none for Applebot, Applebot adopts the Googlebot rules. That is usually intended, but sometimes a surprise.

Blocking by firewall instead of robots.txt

Turning bots away by IP address or firewall stops them reading robots.txt at all. Anthropic therefore explicitly advises against this. Conversely, a crawler that is "allowed" in robots.txt but turned away by the server or bot protection is effectively blocked anyway.

Rules you did not write yourself

Cloudflare can generate a managed robots.txt. When it is enabled, Cloudflare puts its entries ahead of your own file and serves both together; in the example in the documentation, GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent are among those blocked (Cloudflare Docs: Managed robots.txt, accessed 30 October 2026). Your file in the CMS shows none of this. What counts is always what is actually served under /robots.txt.

Impatience

Changes do not take effect immediately. Under RFC 9309, crawlers should not use a cached robots.txt for longer than 24 hours; OpenAI states about 24 hours for search, Perplexity up to 24 hours and Meta also up to 24 hours.

How to check it

1. Look at the file that is served. Open https://your-domain.com/robots.txt in the browser, not the file in the CMS. Only this shows entries added by a CDN or plugin. Repeat this for every subdomain.
2. Work out the applicable group for each identifier. Is there a group with its name? Then only that applies. Otherwise * applies. Check whether any blocks get lost.
3. Compare identifiers. Compare the names in your file with the overview above and remove outdated entries.
4. Check the Google side. The robots.txt report in Search Console shows which robots.txt files Google has found, when they were last crawled and which errors there are (Search Console Help: robots.txt report, accessed 30 October 2026). It tells you nothing about other providers.
5. Read the server logs. They show which identifiers really visit. Because any bot can pretend to be GPTBot, several providers publish lists of their IP ranges that let you tell genuine fetches from fake ones, for example openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, claude.com/crawling/bots.json, www.perplexity.ai/perplexitybot.json, www.bing.com/toolbox/bingbot.json and index.commoncrawl.org/ccbot.json (all reachable on 30 October 2026).

How the llms.txt file fits in, and why it has nothing to do with robots.txt, is explained in llms.txt: what the file can do, what it cannot, and whether it is worth it.

Where Helferlain stands

In the AI visibility area, Helferlain's SEO audit checks whether 13 AI and search crawlers are allowed to read your website according to robots.txt, including the identifiers of OpenAI, Anthropic, Perplexity, Google-Extended, Applebot-Extended, Bing, Common Crawl and Meta. It also checks whether there is an llms.txt and whether content only appears via JavaScript. More on the SEO audit page and in the free website check.