No account neededWe read your robots.txt once and keep nothing

AI crawler checker

See whether ChatGPT, Claude, Perplexity and Google's AI features may read a page under your robots.txt, and which crawlers only collect training data.

Your reading appears here

Enter a page of your site to see what its robots.txt tells each AI crawler. Or see a finished reading first.

Three kinds of AI crawler

AI search: Index the pages AI answers cite: OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot, and Googlebot, whose index feeds Google’s AI Overviews.
Live reads: Fetch one page when a person asks about it: ChatGPT-User, Claude-User, Perplexity-User. OpenAI and Perplexity say robots.txt may not apply to theirs.
Training: Collect pages to train models: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent and Common Crawl’s CCBot.

Stay in AI answers, stay out of training

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Crawlers named together share one group. All six read the Disallow under them.
Every crawler not named reads the * group. The search crawlers are not named, so they may fetch everything.
A crawler with its own group ignores the * group. Give a named crawler every rule it needs, the private paths included.

Questions

Does blocking GPTBot take my site out of ChatGPT?

No. OpenAI documents GPTBot, which collects training data, and OAI-SearchBot, which powers ChatGPT search, as separate crawlers with separate robots.txt rules.

Does Cloudflare block AI crawlers on my site?

It can, in two ways. It can add rules to your robots.txt, which this check reads and labels, and it can refuse crawlers at its firewall, which no outside check can see. Your Cloudflare dashboard shows both.

Do I need an llms.txt file?

Not for crawler access: llms.txt is not a robots.txt rule, and it allows or blocks nothing. Google says its Search does not use it.

Do you keep the sites I check?

No. Our server fetches the robots.txt once, sends you the reading, and stores neither the address nor the result.

Awiser is the builder network where founders show their skills, find collaborators, and grow their projects with an AI copilot. More mini-apps