GPTBot vs OAI-SearchBot: Training, Search, and User-Triggered AI Crawlers

By Rasmus Bastrup 2026-10-30 Parent: Crawl Budget & Indexing · Pillar: Technical SEO Infrastructure

One entity

OpenAI, Anthropic, and Perplexity do not run “one AI bot.” Each runs separate user-agents for training, search indexing, and user-triggered fetches. Block the wrong token and you disappear from ChatGPT / Claude / Perplexity answers while still feeding training — or the reverse. This page is the comparison entity: tokens, purpose, robots.txt behavior, published IP ranges, and how that maps to Cloudflare AI Crawl Control.

1. The three jobs (do not conflate them)

Training crawl — collect public pages that may enter foundation-model training (OpenAI GPTBot, Anthropic ClaudeBot).
Search / index crawl — build or refresh an index used to cite or surface your site in product search (OpenAI OAI-SearchBot, Anthropic Claude-SearchBot, Perplexity PerplexityBot).
User-triggered fetch — hit a URL because a person asked the product to look it up (OpenAI ChatGPT-User, Anthropic Claude-User, Perplexity Perplexity-User). These are not automatic sitewide crawls.

OpenAI’s own docs state the settings are independent: you can allow OAI-SearchBot for ChatGPT search while disallowing GPTBot for training. If both are allowed, OpenAI may reuse one crawl for both use cases to avoid duplicate crawling. Search opt-out via robots.txt can take ~24 hours to take effect.

2. Comparison table

User-agent token	Operator	Purpose (vendor claim)	Respects robots.txt?	Published IP ranges
`GPTBot`	OpenAI	Training generative AI foundation models	Yes — `Disallow` signals content should not be used for training	openai.com/gptbot.json
`OAI-SearchBot`	OpenAI	Surface sites in ChatGPT search features	Yes — opt-out sites are not shown in ChatGPT search answers (may still appear as navigational links)	openai.com/searchbot.json
`ChatGPT-User`	OpenAI	User actions in ChatGPT / Custom GPTs (not automatic crawl; not used for Search eligibility)	May not apply — user-initiated; use `OAI-SearchBot` for Search opt-out	openai.com/chatgpt-user.json
`ClaudeBot`	Anthropic	Collect web content that could contribute to model training	Yes — Anthropic states bots honor robots.txt (and Crawl-delay)	claude.com/crawling/bots.json (shared Anthropic list)
`Claude-SearchBot`	Anthropic	Improve search-result quality / indexing for Claude search	Yes — per Anthropic help center (same robots.txt policy)	claude.com/crawling/bots.json
`Claude-User`	Anthropic	Fetch pages when a Claude user asks a question that needs the web	Yes (vendor claim) — Anthropic documents robots.txt opt-out for Claude-User	claude.com/crawling/bots.json
`PerplexityBot`	Perplexity	Surface / link sites in Perplexity search; not for foundation-model training (Perplexity claim)	Yes	perplexity.ai/perplexitybot.json
`Perplexity-User`	Perplexity	User-triggered page fetch to answer a question + cite the URL	Generally ignores robots.txt (user-requested fetch)	perplexity.ai/perplexity-user.json

Example full User-Agent strings (versions change) are published by OpenAI and Perplexity. Anthropic’s help article documents the three tokens and robots.txt rules; it does not publish a full example UA string in the article body we fetched — treat exact Claude* UA formatting as “token-verified, full string not quoted by Anthropic on that page.”

3. robots.txt pattern: allow search, block training

A common publisher stance: stay eligible for AI search citations, refuse training crawls. Tokens are independent — write each block explicitly.

# Example only — verify against your policy and current vendor docs.
# Search / index bots (allow)
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Training bots (block)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# User-triggered fetchers: policy choice.
# OpenAI: robots.txt may not apply to ChatGPT-User.
# Perplexity: Perplexity-User generally ignores robots.txt.
# Anthropic: documents robots.txt control for Claude-User.
User-agent: Claude-User
Allow: /

robots.txt is preference, not enforcement. Vendors that honor it will usually stop; others may still fetch on user request. For hard blocks, combine robots.txt with edge controls (WAF / Cloudflare AI Crawl Control).

4. How this relates to Cloudflare AI Crawl Control

Cloudflare AI Crawl Control lists verified AI crawlers (including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) and lets you Allow or Block each one. Blocking is enforced via a WAF custom rule — that is the enforcement layer; robots.txt alone is not.

Cloudflare’s bot reference categorizes OpenAI’s three agents as AI Crawler (GPTBot), AI Search (OAI-SearchBot), and AI Assistant (ChatGPT-User) — the same three-job split as above.
Anthropic and Perplexity follow the same category pattern in that table.
Bot Management customers can match cf.bot_management.detection_ids; others typically match User-Agent (and should prefer published IP JSON for allowlists).
Cloudflare can also sync managed robots.txt / Content Signals from Search / Agent / Training policies; still use AI Crawl Control when you need a technical block.

On valerion.tech (state as of 2026-10-29): Bot Management ai_training disallow was disabled so legitimate AI crawlers are not zone-blocked wholesale; Bytespider remains on a custom WAF block. This page does not change that config — it documents the decision model.

5. Practical decision rule

Want citations in ChatGPT / Claude / Perplexity search → allow the Search tokens + their IP JSON on the WAF allow path.
Refuse training corpora → disallow GPTBot and ClaudeBot in robots.txt and Block them in AI Crawl Control if you need enforcement.
Treat *-User agents as product UX, not crawl budget: OpenAI and Perplexity say robots.txt may not stop them; Anthropic documents robots.txt for Claude-User.
Never equate “blocked AI” with “good SEO.” You can block training and still win search visibility.

6. Sources (fetched for this draft)

OpenAI — Overview of OpenAI Crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot; IP JSON links).
Anthropic — Claude Help Center: crawl / block crawler (ClaudeBot, Claude-User, Claude-SearchBot; robots.txt; links to IP list).
Anthropic IP list — claude.com/crawling/bots.json (HTTP 200 verified 2026-10-30).
Perplexity — Perplexity Crawlers (PerplexityBot, Perplexity-User; IP JSON; robots.txt note for User).
Cloudflare — AI Crawl Control bot reference and Manage AI crawlers.

7. Verification notes (claims we could not fully pin)

Anthropic full User-Agent strings: help center verifies the three tokens and robots.txt semantics; it does not quote a complete example UA string like OpenAI/Perplexity do. Cloudflare lists the same tokens as detection User-Agents.
Per-bot Anthropic IP splits: the published JSON is a shared Anthropic prefix list, not separate files per Claude* token (unlike OpenAI/Perplexity).
OAI-AdsBot: documented by OpenAI for ad landing-page checks; out of scope for this entity page’s primary comparison.
Exact Cloudflare plan / detection_id values on valerion.tech: not re-verified in this draft turn beyond the public bot reference table.