WebMCP Agency
Check a site

Guide · AI crawlers

AI crawlers: a reference for site owners

Updated · By the Convertlyft research team

On this page
  1. Three kinds of AI bot
  2. AI crawler reference table
  3. What each vendor says
  4. How to block AI crawlers in robots.txt
  5. User-triggered fetchers are different
  6. Why “allowed in robots.txt” does not mean “reachable”
  7. How to check what AI crawlers can actually reach

AI crawlers are bots run by AI companies that fetch web pages. Some collect pages to train models, some build an index for AI search, and some fetch one page because a user asked. Each has its own robots.txt token, so you can allow or block each one on its own.

The main AI crawlers, in short:

  • GPTBot (OpenAI): crawls for model training.
  • OAI-SearchBot (OpenAI): crawls for ChatGPT search.
  • ChatGPT-User (OpenAI): fetches a page when a user asks ChatGPT to.
  • ClaudeBot (Anthropic): collects content for model training.
  • Claude-SearchBot and Claude-User (Anthropic): improve search results, and fetch pages when people ask Claude questions.
  • PerplexityBot (Perplexity): “designed to surface and link websites in search results on Perplexity.”

Three kinds of AI bot

Most AI companies now run more than one bot. They do different jobs, and blocking one does not block the others.

  • Training crawlers collect pages that may be used to train AI models. Examples: GPTBot, ClaudeBot.
  • Search crawlers build an index so an AI assistant can find and cite your pages in answers. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
  • User-triggered fetchers visit one page because a person asked the assistant to read it, right now. Examples: ChatGPT-User, Claude-User, Perplexity-User.

There is also a fourth kind of control: a robots.txt token that is not a crawler. Google-Extended and Applebot-Extended never fetch pages. They are names you use in robots.txt to say how content collected by the normal crawler may be used for AI.

This split matters. A site might block training but still want to appear and be cited in AI search. That is possible, because the vendors keep these bots separate. OpenAI puts it plainly: “Each setting is independent of the others.”

AI crawler reference table

Every row comes from the vendor’s own documentation, read on 24 October 2026. The robots.txt column is a summary; quotes are in the sections below. User-agent strings change, so we list only the robots.txt token. Check the vendor page for the full string.

Vendor robots.txt token Purpose Follows robots.txt? (vendor’s own words)
OpenAI GPTBot Training Yes. Disallowing it signals content should not be used for training
OpenAI OAI-SearchBot Search (ChatGPT search) Yes. Opted-out sites are not shown in ChatGPT search answers
OpenAI ChatGPT-User User-triggered fetch “robots.txt rules may not apply”
Anthropic ClaudeBot Training Yes. Anthropic says its bots honor robots.txt
Anthropic Claude-SearchBot Search Yes. Same statement
Anthropic Claude-User User-triggered fetch Yes. Same statement; blocking it stops retrieval for user queries
Perplexity PerplexityBot Search (not for model training) Perplexity recommends allowing it in robots.txt to appear in results
Perplexity Perplexity-User User-triggered fetch “generally ignores robots.txt rules”
Google Googlebot Google Search Yes, when crawling automatically
Google Google-Extended Token only: Gemini training and grounding Not a crawler. Controls use of content Google crawls
Apple Applebot Search features (Siri, Spotlight, Safari); data may also train Apple’s models Yes, for crawls targeted at Applebot
Apple Applebot-Extended Token only: use of data for Apple’s generative AI Not a crawler. “Applebot-Extended does not crawl webpages”
Common Crawl CCBot Open web crawl dataset Common Crawl documents robots.txt as the way to block it
Meta meta-externalagent Training AI models, or indexing for Meta products Meta documents a robots.txt disallow for it
Meta meta-webindexer Meta AI search results Meta says allowing it helps Meta AI cite and link to you
Meta meta-externalfetcher User-triggered fetch Meta says it may bypass robots.txt
Amazon Amazonbot Improves Amazon products; may train Amazon AI models Yes. “respects the Robots Exclusion Protocol”
Amazon Amzn-SearchBot Search (for example, Alexa); not for AI training Yes; follows rules for other search bots if not named
Amazon Amzn-User User-triggered fetch “may not follow all robots.txt directives”

What each vendor says

What is GPTBot?

GPTBot is OpenAI’s training crawler. OpenAI says it “is used to crawl content that may be used in training our generative AI foundation models.” OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features.” OpenAI says sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links.” ChatGPT-User visits pages when users ask ChatGPT or a Custom GPT a question. OpenAI says it “is not used for crawling the web in an automatic fashion” and that “because these actions are initiated by a user, robots.txt rules may not apply.” OpenAI also says robots.txt changes can take about 24 hours to affect search results.

What is ClaudeBot?

ClaudeBot collects content that may be used to train Anthropic’s models. Claude-SearchBot “navigates the web to improve search result quality for users.” Claude-User fetches pages “when individuals ask questions to Claude.” For all three, Anthropic says: “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt.” It also supports the non-standard Crawl-delay rule. Anthropic adds that blocking its IP addresses may not work as a reliable opt-out, and that robots.txt is the supported method. Do this on every subdomain you want to opt out.

What is PerplexityBot?

PerplexityBot “is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” Perplexity-User supports user actions. Perplexity says: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”

What are Google-Extended and Applebot-Extended?

Google-Extended is “a standalone product token” that controls whether content Google crawls “may be used for training future generations of Gemini models” and for grounding in Gemini Apps and Vertex AI. Google says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” It has no separate HTTP user-agent string. Googlebot is Google’s search crawler and obeys robots.txt when crawling automatically. Note that Google’s AI Overviews and AI Mode are part of Google Search, Google says AI is built into Search, so Search controls (Googlebot rules, nosnippet, noindex) apply to them.

Applebot powers features such as Spotlight, Siri and Safari, and Apple says its data may also be used to train Apple foundation models. “Applebot-Extended does not crawl webpages.” It only decides how data collected by Applebot is used. Apple adds that pages which disallow Applebot-Extended “can still be included in search results.” If your robots.txt does not mention Applebot but does mention Googlebot, Apple says Applebot follows the Googlebot rules.

What is CCBot?

CCBot builds the Common Crawl open dataset. Common Crawl gives a robots.txt block as the way to stop it, and warns that some crawlers falsely identify themselves as CCBot.

What are Meta’s AI crawlers?

Meta lists meta-externalagent for training AI models or indexing content for its products, and meta-webindexer for Meta AI search. It says meta-externalfetcher may bypass robots.txt because the fetch was requested by a user. Its link-preview crawler, facebookexternalhit, may also bypass robots.txt for security or integrity checks. Meta notes crawlers may cache robots.txt for up to 24 hours.

What is Amazonbot?

Amazonbot is used “to improve our products and services” and “may be used to train Amazon AI models.” Amzn-SearchBot is for search experiences such as Alexa and does not crawl for AI training. Amazon says automated crawling from its listed agents “respects the Robots Exclusion Protocol.” Amzn-User “may not follow all robots.txt directives” because a user started the action.

How to block AI crawlers in robots.txt

robots.txt lives at the root of each host, for example https://example.com/robots.txt. Each group starts with one or more User-agent: lines, then Disallow: or Allow: rules. A bot uses the group that names it most specifically. If none names it, it uses the User-agent: * group.

Block GPTBot (OpenAI training) only:

User-agent: GPTBot
Disallow: /

Block training, keep AI search. This blocks the training crawlers and tokens listed above but leaves the search bots alone:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

Check each vendor’s notes before you copy this. Google-Extended also controls grounding in Gemini Apps and Vertex AI, and meta-externalagent also covers indexing for Meta’s products. Amazonbot, for example, covers both product improvement and possible model training in one agent.

Allow AI search bots explicitly:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Block one folder for selected AI bots:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Disallow: /members/

Add a group for each bot you want to cover.

Block everything from every bot that honors robots.txt:

User-agent: *
Disallow: /

This last one also blocks Googlebot and other search engines. Use it only if you really want the site out of search.

A few rules of thumb:

  • Name each bot. There is no single “all AI” token.
  • One file per host. shop.example.com needs its own robots.txt.
  • Allow time. OpenAI (for search), Meta and Amazon each mention about 24 hours before changes take effect.
  • robots.txt is public and voluntary. It tells well-behaved bots what you want. It does not stop a bot that ignores it, and it does not hide pages.

User-triggered fetchers are different

The table shows a pattern. Automatic crawlers follow robots.txt. Several user-triggered fetchers say they may not.

The reasoning given by vendors is that a person asked for that page, much like a person opening it in a browser. OpenAI, Perplexity, Meta and Amazon all say this in their docs. Anthropic’s article says its bots, including Claude-User, honor robots.txt.

So if you block ChatGPT-User or Perplexity-User in robots.txt, do not assume that is enforced. If you truly must keep a page from these tools, it needs a real access control, such as a login.

Why “allowed in robots.txt” does not mean “reachable”

robots.txt is only half the picture. It says what you want. Your infrastructure decides what happens.

Between a bot and your web server there are often several layers:

  • a CDN that caches and filters traffic,
  • a web application firewall with bot rules,
  • a bot-management or anti-scraping service that scores each visitor,
  • rate limits on the server or the host.

Many of these tools have their own settings for AI bots, separate from robots.txt. When they block a bot, they may return a 403 error, a challenge page, a CAPTCHA, or an empty response. They act on the request itself, not on your robots.txt.

The result is a common mismatch. robots.txt says “welcome, OAI-SearchBot”. The firewall says “403 Forbidden”. The bot cannot read the page, so the AI search tool cannot cite it. The opposite also happens: robots.txt blocks a bot, but nothing enforces it.

Some vendors publish IP ranges so you can verify their bots and allow-list them. OpenAI and Perplexity both recommend allowing requests from their published IP ranges if you want to appear in their search results. Vendors such as Common Crawl and Google warn that user-agent strings can be faked, so checking IP ranges or reverse DNS is the reliable way to confirm a bot is real.

How to check what AI crawlers can actually reach

Do two checks, not one.

  1. Read your robots.txt. Open https://yourdomain.com/robots.txt and look for groups that name the bots in the table. Remember the User-agent: * group applies to any bot not named.
  2. Test the real response. Ask your hosting, CDN or firewall provider which AI bot rules are on. Then look at your server logs for requests from these user agents and the status codes they got.

Our free agent-readiness check does both. It reads your robots.txt for each agent on its list and shows how many it allows. Then it sends a request as the allowed agents and shows how many your server still refuses, so you can spot a bot that is allowed on paper but blocked in practice. Some agents, such as control tokens like Google-Extended, are never sent as a user agent, so they are not probed.

AI crawlers decide whether agents can read your pages. To guide them once they arrive, see llms.txt. To let them take actions on your site, see what WebMCP is.

For a full review of how AI search and agents see your site, Convertlyft offers an agent-readiness report.

Questions people ask

How do I block GPTBot?

Add a group to your robots.txt with 'User-agent: GPTBot' and 'Disallow: /'. OpenAI says disallowing GPTBot indicates your content should not be used to train its generative AI foundation models. It does not remove you from ChatGPT search, which uses a separate bot, OAI-SearchBot.

Does blocking Google-Extended remove me from Google Search?

No. Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It only controls whether content may be used for Gemini model training and grounding.

Do AI crawlers obey robots.txt?

The automatic crawlers listed here are documented by their vendors as following robots.txt. Several user-triggered fetchers are not: OpenAI, Perplexity, Meta and Amazon all say their user-triggered agents may not follow robots.txt, because a person asked for the page.

What is the ChatGPT crawler called?

OpenAI uses three agents. GPTBot crawls for model training. OAI-SearchBot crawls for ChatGPT search. ChatGPT-User visits a page when a user asks ChatGPT to, and is not used for automatic crawling.

If robots.txt allows a bot, can it reach my site?

Not always. A CDN, firewall or bot-protection service can still block or challenge the request before your site answers. robots.txt is a set of instructions, not an access control.

Sources

  1. Overview of OpenAI crawlers (read 2026-10-24)
  2. Does Anthropic crawl data from the web, and how can site owners block the crawler? (Claude Help Center) (read 2026-10-24)
  3. Perplexity crawlers (read 2026-10-24)
  4. Google's common crawlers (Google-Extended, Googlebot) (read 2026-10-24)
  5. AI features and your website (Google Search Central) (read 2026-10-24)
  6. About Applebot (Apple Support) (read 2026-10-24)
  7. Common Crawl: CCBot (read 2026-10-24)
  8. Meta web crawlers (read 2026-10-24)
  9. About Amazonbot (read 2026-10-24)
  10. RFC 9309: Robots Exclusion Protocol (read 2026-10-24)