Matan · SEO · Technical

robots.txt

robots.txt is the first file crawlers request; it should exist, be valid, and not accidentally block the site.

Pass condition in our engine: A non-empty robots.txt exists at the domain root.

1. Problem description

A missing file is not a disaster (crawlers assume they may fetch), but you lose sitemap discovery and fine-grained allow/disallow control. A Disallow: / on User-agent: * is a disaster: nothing gets crawled.

2. Most common causes

  • The file was never deployed to the domain root.
  • www and apex serve different robots.txt files.
  • A default “block all” file was copied from staging.

3. How to fix it

  1. Serve https://example.com/robots.txt with User-agent: * and Allow: / (or specific Disallows).
  2. Reference your XML sitemap.
  3. Do not Disallow CSS/JS that Google needs to render the page.

Test this check

We fetch only this URL (plus robots.txt / llms.txt at the domain root when the check needs them). We do not crawl the rest of the site like a full report.

Scoring rules for the full site crawl are documented in methodology. Want every category at once? Generate a free report.

Get your free SEO & GEO report

Enter your URL and receive a full audit with scoring and recommendations. No sign-up required.

Generate a free reportView sample report