Matan · SEO · Technical
robots.txt (SEO)
robots.txt is the first file crawlers request; it should exist, be valid, and not accidentally block the site.
Pass condition in our engine: A non-empty robots.txt exists at the domain root.
This SEO check is part of SEO-GEO scoring 3.0.0, last updated 2026-10-25. The number on this page is the developer scoring version. It is not an announcement that public release 3.0.0 is live. A free report shows the same pass condition for the pages it fetched.
1. Problem description
A missing file is not a disaster (crawlers assume they may fetch), but you lose sitemap discovery and fine-grained allow/disallow control. A Disallow: / on User-agent: * is a disaster: nothing gets crawled.
2. Most common causes
- The file was never deployed to the domain root.
- www and apex serve different robots.txt files.
- A default “block all” file was copied from staging.
Test this check
We fetch only this URL (plus robots.txt / llms.txt at the domain root when the check needs them). We do not crawl the rest of the site like a full report.
3. How to fix it
- Serve https://example.com/robots.txt with User-agent: * and Allow: / (or specific Disallows).
- Reference your XML sitemap.
- Do not Disallow CSS/JS that Google needs to render the page.
Scoring rules for the scan of fetched pages are documented in methodology. Want every category at once? Generate a free report.
Get your free website audit
Enter a public URL. The scan reads up to 300 pages and returns an HTML report with scores for measured pillars and a recommendation for each finding. No sign-up required.