Matan · SEO · Technical

Page loads (HTTP 200)

Crawlers can only index a URL that returns a successful HTML document.

Pass condition in our engine: The URL returns HTTP status 200.

1. Problem description

A 4xx/5xx, timeout or empty response means Googlebot and AI crawlers never see your content. Soft 404s (200 with an error template) are worse: they waste crawl budget and dilute quality signals.

2. Most common causes

  • The slug was renamed without a redirect.
  • The origin is down, overloaded, or geo-blocked.
  • WAF / bot protection blocks unknown user agents (including SEO-GEO-Bot).
  • The CMS returns 200 for missing pages with a “not found” template.

3. How to fix it

  1. Confirm the URL returns 200 in an incognito window and in server logs.
  2. Allow SEO-GEO-Bot/1.0 and Googlebot in the WAF if you whitelist crawlers.
  3. Add 301s for moved URLs; use a real 404/410 for gone pages.
  4. Fix application errors that produce 500s on otherwise valid routes.

Test this check

We fetch only this URL (plus robots.txt / llms.txt at the domain root when the check needs them). We do not crawl the rest of the site like a full report.

Scoring rules for the full site crawl are documented in methodology. Want every category at once? Generate a free report.

Get your free SEO & GEO report

Enter your URL and receive a full audit with scoring and recommendations. No sign-up required.

Generate a free reportView sample report