AI Crawler Access Checker

Paste a public URL. See which AI and search crawlers are allowed, what page-level robots tags say, whether /llms.txt exists, and copy fix snippets. Facts are sourced from official vendor docs.

Why check AI crawler access?

Training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) and search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) use different robots.txt tokens. Blocking the wrong one can hide you from ChatGPT search while still allowing training — or the reverse.

This checker applies longest-match Allow/Disallow rules (Allow wins on equal length, Google-style), reads <meta name="robots">, bot-specific meta, X-Robots-Tag, Bing’s ai-training / ai-grounding preferences, and probes /llms.txt.

Bing ai-grounding (verified Oct 30 2026)

From the Bing blog: grounding is page-level only.

<meta name="Bingbot" content="ai-grounding:disallow">
X-Robots-Tag: Bingbot: ai-grounding:disallow
<!-- or -->
<meta name="Bingbot" content="ai-grounding:disallowextended">

Site/path AI Training uses robots.txt User-Agent: __usage_ai-training.

FAQ

What does this AI crawler checker test?

It fetches robots.txt, the target page (meta robots, Bingbot preferences, X-Robots-Tag), and /llms.txt, then evaluates allow/disallow per bot using longest-match rules.

Which bots are covered?

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Bingbot, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot.

What is Bing ai-grounding?

Per Bing’s 30 Oct 2026 blog, ai-grounding is a page-level preference via meta name=Bingbot content=ai-grounding:disallow|disallowextended|allow or X-Robots-Tag. It is not a robots.txt directive.

Is localhost supported?

No. The API blocks private, loopback, link-local, and metadata targets (SSRF protection).