Matan · GEO · AI Readability
robots.txt not blocking all (GEO)
User-agent: * combined with Disallow: / hides the entire site from every well-behaved crawler.
Pass condition in our engine: No User-agent: * / Disallow: / rule is present.
This GEO check is part of SEO-GEO scoring 2.2.1, last updated 2026-09-18. About 90 percent of the evaluation runs on the HTML we crawl, so you can verify the same pass condition with a free report.
1. Problem description
This is the nuclear option: Google, Bing and AI bots that honour robots.txt will not fetch any path. It is common on staging hosts that were pointed at a production domain, or after a botched “launch mode” plugin.
2. Most common causes
- Staging robots.txt deployed to production.
- A maintenance plugin enabled “block search engines”.
- Mis-indented robots.txt so Disallow: / applies to * unexpectedly.
3. How to fix it
- Delete Disallow: / under User-agent: * or replace it with specific sensitive paths only.
- Verify with Google’s robots.txt tester.
- Keep a separate robots.txt on staging hostnames, not on the live apex.
Test this check
We fetch only this URL (plus robots.txt / llms.txt at the domain root when the check needs them). We do not crawl the rest of the site like a full report.
Scoring rules for the full site crawl are documented in methodology. Want every category at once? Generate a free report.
Get your free website audit
Enter your URL and receive a full audit with scoring and recommendations. No sign-up required.