Matan · GEO · AI Readability

robots.txt not blocking all

User-agent: * combined with Disallow: / hides the entire site from every well-behaved crawler.

Pass condition in our engine: No User-agent: * / Disallow: / rule is present.

1. Problem description

This is the nuclear option: Google, Bing and AI bots that honour robots.txt will not fetch any path. It is common on staging hosts that were pointed at a production domain, or after a botched “launch mode” plugin.

2. Most common causes

  • Staging robots.txt deployed to production.
  • A maintenance plugin enabled “block search engines”.
  • Mis-indented robots.txt so Disallow: / applies to * unexpectedly.

3. How to fix it

  1. Delete Disallow: / under User-agent: * or replace it with specific sensitive paths only.
  2. Verify with Google’s robots.txt tester.
  3. Keep a separate robots.txt on staging hostnames, not on the live apex.

Test this check

We fetch only this URL (plus robots.txt / llms.txt at the domain root when the check needs them). We do not crawl the rest of the site like a full report.

Scoring rules for the full site crawl are documented in methodology. Want every category at once? Generate a free report.

Get your free SEO & GEO report

Enter your URL and receive a full audit with scoring and recommendations. No sign-up required.

Generate a free reportView sample report