Matan · GEO · AI Readability
CDN/WAF allows crawlers (GEO)
A Cloudflare JS challenge or WAF 403 hides the HTML from unknown crawlers even when robots.txt is open.
Pass condition in our engine: The HTML document is returned without a CDN/WAF challenge or bot-block status.
This GEO check is part of SEO-GEO scoring 2.2.1, last updated 2026-09-18. About 90 percent of the evaluation runs on the HTML we crawl, so you can verify the same pass condition with a free report.
1. Problem description
Generative engines that are not on the CDN’s verified-bot list never see the document. The page looks fine in a browser after the challenge, but GPTBot, ClaudeBot and similar fetchers get “Just a moment…” or a 403. That is a GEO-critical miss: robots.txt can allow AI bots while the WAF still blocks them.
2. Most common causes
- Cloudflare Bot Fight / Super Bot Fight / Managed Challenge on HTML.
- WAF rule blocking non-browser user agents, datacenter IPs, or missing cookies.
- Country or ASN allowlists that exclude crawler networks.
3. How to fix it
- Allow Cloudflare Verified Bots (Google, Bing, and listed AI crawlers).
- Add a WAF skip or allowlist for SEO-GEO-Bot/1.0 if you want this audit to match AI fetchers.
- Do not put a JS challenge in front of HTML you want cited in AI answers.
Test this check
We fetch only this URL (plus robots.txt / llms.txt at the domain root when the check needs them). We do not crawl the rest of the site like a full report.
Scoring rules for the full site crawl are documented in methodology. Want every category at once? Generate a free report.
Get your free website audit
Enter your URL and receive a full audit with scoring and recommendations. No sign-up required.