Matan · GEO · AI Readability
AI crawlers allowed
If GPTBot, PerplexityBot, ClaudeBot or Google-Extended are disallowed, generative engines cannot cite you.
Pass condition in our engine: No known AI bot is blocked by its own robots.txt group.
1. Problem description
robots.txt groups that Disallow: / for a named AI bot are an explicit “do not train / do not quote” signal. Unlike a missing file, this actively prevents ChatGPT, Perplexity and similar systems from fetching the page. It is weighted critical in the GEO score.
2. Most common causes
- A privacy-default robots.txt copied from a generator that blocks GPTBot.
- Legal asked to “opt out of AI” without distinguishing training vs live search crawlers.
- A plugin added Google-Extended Disallow while leaving Googlebot allowed.
3. How to fix it
- Remove Disallow: / from named AI user-agents you want to be cited by.
- Keep blocks only if you truly do not want those engines to read the site.
- Allow GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended separately if you need a split policy.
Test this check
We fetch only this URL (plus robots.txt / llms.txt at the domain root when the check needs them). We do not crawl the rest of the site like a full report.
Scoring rules for the full site crawl are documented in methodology. Want every category at once? Generate a free report.
Get your free SEO & GEO report
Enter your URL and receive a full audit with scoring and recommendations. No sign-up required.