AI Crawlers: GPTBot, ClaudeBot, PerplexityBot and What to Allow

AI crawlers are automated clients that fetch webpages on behalf of AI companies, and they do three distinct jobs: collecting text that may train foundation models, building a search index that an assistant queries, and fetching a single page in real time because a user asked a question about it. Each job has its own user-agent token, its own vendor documentation and its own consequence when you block it. Reading a log line as an AI bot hides the only decision that matters: which of those three jobs you want to permit.

TL;DR

• Three jobs, not one. Training crawlers, search-index crawlers and user-triggered fetchers are separate agents with separate names. A blanket block treats them as one and usually removes visibility the business wanted.
• OpenAI documents four agents: GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (user-initiated actions) and OAI-AdsBot (checks on ad landing pages).
• Anthropic documents three (ClaudeBot, Claude-SearchBot, Claude-User) and Perplexity two (PerplexityBot for its index, Perplexity-User for user-initiated visits).
• Google-Extended is not a Search switch. Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search".
• Microsoft's public webmaster documentation does not list a separate training-crawler token. Bing documents page-level NOARCHIVE and NOCACHE controls and now reports citations through AI Performance in Bing Webmaster Tools.
• Applebot-Extended never fetches a page. Apple says it does not crawl, and pages that disallow it still appear in Spotlight, Siri and Safari.
• Providers document user-triggered fetchers differently. OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Meta says Meta-ExternalFetcher may bypass it. Anthropic publishes a robots.txt rule for Claude-User.
• robots.txt expresses rules to cooperating crawlers; it is not an access-control system. If access must be technically prevented, use an appropriate edge or application control and verify traffic using the provider methods that are actually available.

Why one log line has three possible meanings

A hit from an unfamiliar user-agent tells you little on its own. The same company can send three different agents in the same hour, and each one means something different for the business.

A training fetch collects content that may be used to develop or improve a model. It does not create a referral or guarantee that the model will remember, describe or recommend the source accurately. The decision is therefore about rights, confidentiality and business policy rather than measurable referral traffic.

A search-index fetch supports the retrieval layer an assistant can query for current information. Allowing the crawler can make a page eligible to be surfaced or cited; it does not guarantee inclusion. Blocking a search token can remove that route to visibility without necessarily changing a separate training preference.

A user-requested fetch happens because a person pasted your URL into an assistant, or asked something whose answer needed your page open. That behaves like traffic, not crawling, which is why most providers document that robots.txt may not apply to it.

Confusing the three can make policy contradict the business goal: a blanket rule may block retrieval visibility while leaving unrelated training, preview or user-triggered behaviour unchanged. Decisions should be made per documented token and verified against the delivery layer.

Glossary

• User-agent token - the short name a crawler matches on in robots.txt, such as GPTBot, not the full browser-style string it sends in the header.
• Training crawler - an agent whose documented purpose includes collecting content that may train or refine foundation models.
• Search crawler - an agent that builds an index used to answer live questions and link back to sources.
• User-triggered fetcher - an agent that retrieves one URL because a person asked for it in an assistant, rather than crawling on a schedule.
• Reverse DNS verification - confirming a crawler by resolving its IP to a hostname, then resolving that hostname back to the same IP.
• Content signal - a statement of how content may be used after access, separate from whether access is allowed.

The agent map: who is fetching, why, and what blocking costs

Every purpose below comes from the operator's own documentation. Every entry in the last column is a mechanical consequence the provider describes, not a traffic estimate.

Several specialised agents sit outside the training-versus-search choice. OAI-AdsBot checks landing pages submitted to OpenAI advertising products; facebookexternalhit renders link previews; Meta-ExternalAds supports Meta advertising products; and GoogleOther performs assorted Google product and research fetches. Allow each only where its documented function is useful to the site. A marketing landing page, private customer portal and licensed archive do not need the same rule.

Two patterns are worth naming. Search agents are normally allowed when citation and referral eligibility is the goal, but allowing them does not guarantee a link. Training access is a separate decision based on content rights, confidentiality, licensing strategy, server cost and the organisation's risk policy. Training access should not be justified with an unproven promise of better brand representation.

Where the table's own schema breaks

Microsoft does not fit the same public token map. Its webmaster documentation does not list a Bing equivalent of GPTBot. Microsoft instead documented page-level NOARCHIVE and NOCACHE controls for Bing's generative experiences: NOARCHIVE excludes the content from generated answers under the published policy, while NOCACHE permits a more limited use of URL, title and snippet. Because this guidance and product names can change, verify the current Bing documentation before deploying either directive at scale.

So the Microsoft control is per-page, not per-site, and lives in your templates:

<meta name="robots" content="noarchive">
<meta name="bingbot" content="nocache">

Apple breaks the pattern differently. Applebot-Extended never fetches anything, so you will never see it in logs. It exists only so a robots.txt rule can carry an opt-out signal for Apple's foundation-model training, and Apple states that using it does not remove pages from Apple's search surfaces.

Google-Extended: the block that costs nothing in Search

Google-Extended is a robots.txt product token that controls specified uses of content Google has already crawled for Gemini model training and grounding. It is not a separate HTTP user agent you can identify in access logs. Google's documentation states that Google-Extended does not affect inclusion or ranking in Google Search.

If a rights or editorial policy says the content should not be used for those Gemini purposes, disallowing Google-Extended is the documented signal and does not change Google Search eligibility. It does not control AI Overviews or AI Mode, which are Search features. Googlebot access and supported snippet controls, including nosnippet, max-snippet and data-nosnippet, govern how Search can use and present page content. How those features select sources is covered in our guide to AI SEO across search engines, AI Overviews and LLMs.

Which agents actually honour robots.txt

Anthropic's position differs from the OpenAI, Perplexity and Meta wording: it documents a User-agent: Claude-User / Disallow: / rule for its user-initiated agent. That is a published preference for a cooperating crawler, not a substitute for authentication on private content. Anthropic also warns that broad IP blocking can prevent its bots from reading robots.txt, so edge rules and the file must be designed together.

How to verify a bot is what it says it is

A user-agent string is self-reported text. Anyone can send ClaudeBot in a header, so it is not proof of identity. Many — but not all — providers publish a verification method, generally in one of two families.

Reverse DNS, then forward DNS. Google documents the canonical version: resolve the accessing IP to a hostname, confirm it ends in googlebot.com, google.com or googleusercontent.com, then resolve that hostname forward and confirm it returns the original IP. The forward step is the one people skip, and skipping it makes the check worthless, because reverse DNS records can be set by whoever controls the IP.

Published IP range files. OpenAI, Anthropic, Perplexity and Google publish machine-readable IP ranges for documented agents. Use the URLs in the provider's current documentation rather than copying a range into an article or firewall rule, because ranges and file locations can change.

The rule that follows is: use the documented user-agent token to express policy, and use the provider's supported verification method when identity matters. A WAF allow rule based only on a user-agent name can be spoofed. Where a provider publishes no reliable identity method, enforce on authentication and behaviour rather than pretending the header is proof.

You will also see agents with no public documentation. No vendor page, no published range and no verification path means no informed allow decision is possible. Treat those as unverifiable traffic and handle them on behaviour: request rate, path patterns and server cost.

robots.txt is a request, not a lock

The Robots Exclusion Protocol was formalised as RFC 9309 in 2022 and describes a voluntary convention. A well-behaved client reads your rules and complies. Nothing in the protocol prevents a non-compliant client from ignoring them.

If a resource must not be publicly accessible, protect it with authentication and authorisation. For public content where bot access itself must be limited, WAF or bot-management controls can use verified provider identity, request behaviour and rate limits. Cloudflare makes the distinction clearly in its Content Signals Policy: preference signals are not technical countermeasures against scraping.

The ecosystem is also splitting two questions robots.txt collapses into one. Robots.txt answers may you fetch this, never what may you do with it afterwards. The Content Signals Policy adds a line naming three post-access uses, each taking yes or no, with an absent signal meaning no stated preference:

Content-Signal: search=yes, ai-input=yes, ai-train=no

The IETF has chartered an AI Preferences working group to standardise a vocabulary for these preferences and mechanisms for attaching them to content, including robots.txt-like formats and HTTP headers. While that work develops, treat vendor-specific post-access signals as policy declarations whose legal effect must be assessed by counsel, and keep access enforcement in the layer that can actually refuse a request.

How we approach this at Space Ads

At Space Ads, we would start with raw server or CDN logs, the current robots.txt file and the edge configuration. The purpose is to separate a declared rule from the response a verified crawler actually receives. A WAF rule, bot-management setting, CDN challenge, authentication layer or rate limit can deny access even when robots.txt allows it.

We then classify each documented token by purpose and record an owner, decision and rationale. Search-retrieval agents are normally allowed when citation eligibility is the objective. Training access goes to the content-rights and risk owner rather than the last person who edited robots.txt. User-triggered fetchers are assessed as a separate class, with private resources protected by authentication rather than crawler etiquette. Monitoring combines verified log activity, crawl errors and platform reporting such as Bing Webmaster Tools AI Performance where available. The broader visibility and measurement scope sits on our AI SEO service page.

An eight-step plan for setting your crawler policy

1. Pull a representative period of raw access logs and group requests by user-agent token. Cover normal traffic and any known publishing peak; analytics tools often filter the bot traffic you need to inspect.
2. Verify the top agents with reverse DNS plus a forward check, or against the provider's IP range file. Agents that fail are impersonators, not policy problems.
3. Classify each verified agent as training, search index or user-triggered, using the operator's documentation rather than a third-party list.
4. Decide training exposure once, with the content owner, and write the reason next to the rule. If nobody can state why a training agent is blocked, it was inherited.
5. Allow documented search agents when retrieval visibility is the goal, unless rights, security or operational constraints require otherwise. Allowing creates eligibility, not a citation guarantee.
6. Reconcile robots.txt with the edge. Check responses recorded for verified provider traffic or use the provider's supported diagnostics. Merely spoofing its user-agent from your own IP does not reproduce provider identity or prove that the real crawler is allowed.
7. Add per-page controls where you need nuance. noarchive, nocache and data-nosnippet keep a page indexed while limiting what appears inside a generated answer.
8. Set a review trigger and owner. Re-check documentation after a provider change, a security-policy change and on a recurring schedule appropriate to the site's risk; crawler names and behaviour can change without a local code edit.

The file-level layer beside this - what belongs in llms.txt and what the file can and cannot do - is a separate topic, covered in llms.txt and AI crawler controls.

Common mistakes and what to do instead

FAQ

What are AI crawlers?

AI crawlers are automated clients that fetch webpages on behalf of AI companies. They serve three distinct purposes: gathering content that may train foundation models, building search indexes that assistants query for current information, and retrieving individual pages when a user's question requires it. Each purpose has its own documented user-agent token.

What is ClaudeBot and should I block it?

ClaudeBot is Anthropic's crawler, documented as collecting web content to help improve the utility and safety of its generative models. It is blocked with a User-agent: ClaudeBot / Disallow: / rule in robots.txt. Blocking it does not affect Claude's search behaviour, which uses the separate Claude-SearchBot agent.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot crawls content that may train OpenAI's generative foundation models. OAI-SearchBot surfaces websites in ChatGPT's search features. They are separate tokens with separate robots.txt rules, so a site can disallow training while staying eligible to appear in ChatGPT search results.

What is ChatGPT-User?

ChatGPT-User is the OpenAI agent handling user-initiated actions inside ChatGPT and custom GPTs, such as opening a page a person asked about. OpenAI states that because these actions are initiated by a user, robots.txt rules may not apply, and that this agent does not determine whether content appears in search.

Does blocking Google-Extended hurt Google rankings?

No. Google's crawler documentation states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal in Google Search. It controls whether crawled content may train future Gemini models and support grounding features.

Should I block AI crawlers?

There is no safe universal rule. Decide separately for training, search retrieval and user-triggered fetches. Training access depends on rights, confidentiality, licensing strategy, server cost and risk policy. Search agents are commonly allowed when citation and referral eligibility is valuable, but access does not guarantee inclusion. Private or paid content should be protected by authentication rather than robots.txt alone.

How do I verify that a crawler is genuine?

Run a reverse DNS lookup on the requesting IP, confirm the hostname belongs to the provider's documented domain, then run a forward lookup on that hostname and confirm it returns the same IP. Alternatively, check the IP against the CIDR ranges the provider publishes as JSON. User-agent strings alone are easily spoofed.

Why is a bot blocked even though robots.txt allows it?

Because robots.txt is not the only gate. WAF rules, bot-management settings, CDN challenges, authentication and rate limits can refuse a request robots.txt permits. Inspect responses for verified provider traffic or use supported diagnostics; a request from your own IP with a copied user-agent tests only part of the stack.

Sources and further reading

• OpenAI - Overview of OpenAI crawlers
• Anthropic - Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity - Perplexity crawlers
• Google Search Central - Google crawlers and fetchers overview
• Google Search Central - Verifying Googlebot and other Google crawlers
• Apple Support - About Applebot
• Meta for Developers - Web crawlers
• Bing Webmaster Blog - Options for webmasters to control usage of their content in Bing Chat
• Bing Webmaster Blog - AI Performance in Bing Webmaster Tools
• Common Crawl - CCBot
• IETF - RFC 9309: Robots Exclusion Protocol
• IETF - AI Preferences working group
• Cloudflare - Content Signals Policy

Agent names and behaviour above reflect vendor documentation as of July 2026. Providers add, rename and retire agents without notice, so re-check the source pages before changing a live configuration.

In short

• AI crawlers do three jobs - training, search indexing and user-requested fetching - and each has its own agent name and its own cost when blocked.
• Search agents support citation and referral eligibility. Blocking them can remove that route to visibility without necessarily changing a separate training policy.
• Google-Extended is a training and grounding control, and Google states it affects neither inclusion nor ranking in Search.
• Microsoft and Apple break the common token pattern: Bing documents page-level NOARCHIVE and NOCACHE controls, while Applebot-Extended is a control token that does not crawl.
• OpenAI, Perplexity and Meta qualify robots.txt handling for user-triggered fetchers; Anthropic publishes a Claude-User rule.
• Verify identity using supported provider methods and inspect the response delivered to genuine bot traffic; robots.txt, WAF, CDN and authentication rules must agree.

Continue reading

• AI SEO: How to Optimize for Search Engines, AI Overviews and LLMs
• llms.txt and AI Crawler Controls
• Generative Engine Optimization: Getting Brands Mentioned in AI Search Results
• AEO: Answer Engine Optimization vs GEO and SEO
• What Is an SEO Audit and How to Do It Properly
• AI SEO · Marketing audit · Contact Space Ads