How much of the web can AI assistants actually read?

Every domain here was asked for its homepage once as a browser and once as each of the crawlers behind ChatGPT, Claude, Perplexity, Gemini, Meta AI, Apple Intelligence and Doubao, and the answers compared. Not a prediction, not a crawl of someone else's dataset: the requests are made, and what came back is what you see. Google-Extended and Applebot-Extended never crawl — they are robots.txt opt-out tokens — so for those only robots.txt is reported, and a dash means there is nothing a request could have measured.

Who changed their mind — the last 30 days

No crawler-level change recorded yet. A change is only reportable between two passes that both carry a per-crawler fingerprint, and the index started storing those on 2026-09-19: the score history before that says a site moved, not which crawler moved. This is an empty result, not a claim that nothing changed — and nobody can reconstruct it afterwards, which is why it is recorded from now on.

Per crawler

The last column is the number that exists nowhere else: robots.txt lets the crawler in and the server refuses it regardless. It is almost never a decision anyone made — it is an edge rule nobody checked.

Who is actually doing the blocking (47,174 domains with the edge identified)

Unit: one domain × one crawler. Counted only where that site's robots.txt allows that crawler, so every refusal here contradicts the site's own stated policy. The edge is read from the response headers of the same request (cf-ray, akamai-grn, x-amz-cf-id, x-fastly-request-id…); sites with no recognisable signature are grouped as "no known edge", and domains measured before the header was recorded are left out of this table entirely rather than guessed into it. Read "no known edge" as an upper bound, not as a vendor: headers are not kept, so a domain read before a signature was added to the table stays in that bucket until it is re-requested — a sample of 250 of them re-requested on 19 Sep 2026 found 14% already carrying a signature the current table recognises. A weekly pass re-reads them, so the bucket shrinks on its own; the named vendors below are therefore undercounts, never overcounts. For part of the domains the edge was read in a later pass than the crawler verdicts (one request, headers only), so a site that changed CDN in between is shown under its current one until a full re-measurement replaces both. The two robots.txt-only tokens make no requests and are excluded.

Same vendors, one column per crawler

Each cell: of the domain × crawler pairs behind that vendor whose robots.txt allows that crawler, the share the server refused anyway. A vendor that refuses every crawler at the same rate is a wall nobody aimed; a vendor whose rate swings between crawlers is a managed list that names some user-agents and not others — which is the vendor's policy, not the site's. The small figure under each rate is that cell divided by the same cell for domains with no known edge: sites refuse AI crawlers for their own reasons everywhere, and this ratio is what being behind that vendor adds. 1.00× means the vendor changes nothing for that crawler. On Cloudflare (26,708 domains), the widest such gap is Amazonbot, refused on 23% of its 9,818 allowed pairs, against Applebot at 4% of 3,086 — 6.4×. Cells with fewer than 50 robots-allowed pairs are left empty rather than estimated; a crawler most sites block in robots.txt (Bytespider) reaches that floor on the largest vendors only. Columns do not share a denominator: an agent added to the registry later has only been asked on the domains measured since, so its column is a more recent slice of the same list — the pair count behind every cell is in its tooltip, and comparing two columns compares two samples, not two moments of one.

By shop platform (17,619 confirmed stores)