Matan · SEO · Structure
Crawl completeness
If we cannot fetch this URL, a site-wide crawl will not analyse it either.
Pass condition in our engine: The URL returns HTTP 200 with HTML.
1. Problem description
In a full report this check asks whether the crawler reached the discoverable set (up to the page cap). On a single URL it reduces to: did this address return HTML? Failures here are fetch, TLS, or bot-block problems — not “the sitemap is incomplete”.
2. Most common causes
- Bot blocking, geo fences, or IP reputation filters.
- The URL requires cookies or a login.
- Timeouts on a slow origin.
3. How to fix it
- Make the URL publicly fetchable without a session.
- Allow our user agent if you filter bots.
- Use a full report to see how much of the site is actually discovered.
Test this check
We fetch only this URL (plus robots.txt / llms.txt at the domain root when the check needs them). We do not crawl the rest of the site like a full report.
Scoring rules for the full site crawl are documented in methodology. Want every category at once? Generate a free report.
Get your free SEO & GEO report
Enter your URL and receive a full audit with scoring and recommendations. No sign-up required.