Matan · SEO · Structure

Crawl completeness (SEO)

If we cannot fetch this URL, a site-wide crawl will not analyse it either.

Pass condition in our engine: The URL returns HTTP 200 with HTML.

This SEO check is part of SEO-GEO scoring 3.0.0, last updated 2026-10-25. The number on this page is the developer scoring version. It is not an announcement that public release 3.0.0 is live. A free report shows the same pass condition for the pages it fetched.

1. Problem description

In a full report this check asks whether the crawler reached the discoverable set (up to the page cap). On a single URL it reduces to: did this address return HTML? Failures here are fetch, TLS, or bot-block problems — not “the sitemap is incomplete”.

2. Most common causes

  • Bot blocking, geo fences, or IP reputation filters.
  • The URL requires cookies or a login.
  • Timeouts on a slow origin.

Test this check

We fetch only this URL (plus robots.txt / llms.txt at the domain root when the check needs them). We do not crawl the rest of the site like a full report.

3. How to fix it

  1. Make the URL publicly fetchable without a session.
  2. Allow our user agent if you filter bots.
  3. Use a full report to see how much of the site is actually discovered.

Scoring rules for the scan of fetched pages are documented in methodology. Want every category at once? Generate a free report.

Get your free website audit

Enter a public URL. The scan reads up to 300 pages and returns an HTML report with scores for measured pillars and a recommendation for each finding. No sign-up required.