How to Unblock GPTBot, ClaudeBot and PerplexityBot

Blocking an AI crawler in robots.txt removes your site from that engine's answers — GPTBot blocked means ChatGPT cannot browse your pages, ClaudeBot blocked means Claude can't, and so on. Many sites did this by accident: a blanket anti-scraper rule added in 2023, a security plugin's default, a CDN template. This guide shows you how to check, and exactly what to change.

How do I check if my site blocks AI crawlers?

Open yoursite.com/robots.txt in a browser and look for these five user-agents: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Google's AI features) and CCBot (Common Crawl, whose corpus feeds many models). Two patterns block them:

• A named block — User-agent: GPTBot followed by Disallow: /. Deliberate or copied from a template; either way, that engine is locked out.
• A wildcard block — User-agent: * with Disallow: / and no explicit groups allowing the AI bots. The wildcard catches every crawler you haven't named, AI crawlers included.

Or skip the manual reading: the free scan parses your robots.txt with proper longest-match precedence and reports each of the five bots individually.

What should my robots.txt say instead?

An explicit welcome, one group per crawler:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

Explicit groups matter even if nothing currently blocks the bots: they make your intent unambiguous, and they keep a future blanket Disallow from silently removing you from AI answers. That's why our scoring gives full crawler-access credit only to explicit allows.

Should any business keep AI crawlers blocked?

There are legitimate reasons to block — paywalled publishers licensing their archives, sites whose content is their entire product. Those are deliberate commercial decisions. For a business that wants customers to find it, the calculus is different: the crawlers you block can't recommend you. Decide on purpose, not by template.

How long until changes take effect?

Crawlers re-fetch robots.txt frequently — typically within hours to a day. Being crawled is the prerequisite, not the payoff: once readable, your odds of citation depend on the content itself — see how AI engines choose which businesses to cite.