Every domain here was asked for its homepage once as a browser and once as each of the crawlers behind ChatGPT, Claude, Perplexity, Gemini, Meta AI, Apple Intelligence and Doubao, and the answers compared. Not a prediction, not a crawl of someone else's dataset: the requests are made, and what came back is what you see. Google-Extended and Applebot-Extended never crawl — they are robots.txt opt-out tokens — so for those only robots.txt is reported, and a dash means there is nothing a request could have measured.
| Crawler | Measured | Served | robots.txt says no | Server says no anyway |
|---|---|---|---|---|
| ClaudeBot (Claude) | 46,907 | 84% | 2,066 | 6,503 |
| GPTBot (ChatGPT) | 46,907 | 85% | 2,583 | 6,190 |
| OAI-SearchBot (ChatGPT Search) | 46,907 | 90% | 823 | 4,490 |
| PerplexityBot (Perplexity) | 46,907 | 90% | 1,229 | 4,366 |
| Google-Extended (Gemini, AI Overviews) a robots.txt token, not a crawler: it controls how already-crawled pages may be used, and never makes a request of its own | 46,907 | — | 1,765 | — |
| Meta-ExternalAgent (Meta AI) | 18,535 | 83% | 826 | 2,931 |
| Amazonbot (Alexa, Rufus) | 18,535 | 79% | 897 | 3,652 |
| Bytespider (Doubao, Lark) | 245 | 74% | 25 | 54 |
| Applebot (Siri, Apple Intelligence) | 245 | 88% | 5 | 30 |
| Applebot-Extended (Apple Intelligence training) a robots.txt token, not a crawler: it controls how already-crawled pages may be used, and never makes a request of its own | 245 | — | 16 | — |
The last column is the number that exists nowhere else: robots.txt lets the crawler in and the server refuses it regardless. It is almost never a decision anyone made — it is an edge rule nobody checked.
| Edge in front of the site | Domains | Requests robots.txt allows | Refused anyway |
|---|---|---|---|
| 391 | 1,751 | 60% | |
| Akamai | 122 | 513 | 36% |
| Cloudflare | 3,197 | 12,910 | 21% |
| AWS CloudFront | 950 | 3,880 | 16% |
| Azure Front Door | 58 | 240 | 15% |
| no known edge | 2,952 | 12,023 | 11% |
| Fastly | 452 | 1,747 | 7% |
| Vercel | 133 | 584 | 6% |
| Qrator | 61 | 283 | 5% |
| Varnish | 64 | 269 | 5% |
| DDoS-Guard | 63 | 287 | 5% |
| Netlify | 67 | 302 | 4% |
Unit: one domain × one crawler. Counted only where that site's robots.txt allows that crawler, so every refusal here contradicts the site's own stated policy. The edge is read from the response headers of the same request (cf-ray, akamai-grn, x-amz-cf-id, x-fastly-request-id…); sites with no recognisable signature are grouped as "no known edge", and domains measured before the header was recorded are left out of this table entirely rather than guessed into it. For part of the domains the edge was read in a later pass than the crawler verdicts (one request, headers only), so a site that changed CDN in between is shown under its current one until a full re-measurement replaces both. The two robots.txt-only tokens make no requests and are excluded.
| Platform | Stores | Mean score | Open to all |
|---|---|---|---|
| shopify | 14,010 | 72 | 99% |
| woocommerce | 1,690 | 83.7 | 80% |
| other | 626 | 86.2 | 73% |
| bigcommerce | 435 | 69.6 | 96% |
| magento | 149 | 74.9 | 36% |