Matan · SEO · Technical

robots.txt (SEO)

robots.txt is the first file crawlers request; it should exist, be valid, and not accidentally block the site.

Pass condition in our engine: A non-empty robots.txt exists at the domain root.

This SEO check is part of SEO-GEO scoring 3.0.0, last updated 2026-10-25. The number on this page is the developer scoring version. It is not an announcement that public release 3.0.0 is live. A free report shows the same pass condition for the pages it fetched.

1. Problem description

A missing file is not a disaster (crawlers assume they may fetch), but you lose sitemap discovery and fine-grained allow/disallow control. A Disallow: / on User-agent: * is a disaster: nothing gets crawled.

2. Most common causes

  • The file was never deployed to the domain root.
  • www and apex serve different robots.txt files.
  • A default “block all” file was copied from staging.

Test this check

We fetch only this URL (plus robots.txt / llms.txt at the domain root when the check needs them). We do not crawl the rest of the site like a full report.

3. How to fix it

  1. Serve https://example.com/robots.txt with User-agent: * and Allow: / (or specific Disallows).
  2. Reference your XML sitemap.
  3. Do not Disallow CSS/JS that Google needs to render the page.

Scoring rules for the scan of fetched pages are documented in methodology. Want every category at once? Generate a free report.

Get your free website audit

Enter a public URL. The scan reads up to 300 pages and returns an HTML report with scores for measured pillars and a recommendation for each finding. No sign-up required.