The Agent Web Index — how much of the web can AI assistants actually read?

—

Who gets let in, and who stops them

For each assistant's crawler, the split that only a live request can reveal: sites that serve it, sites that ban it in robots.txt, and — the number that exists nowhere else — sites whose robots.txt says yes while the server or CDN says no. That last column is almost always an accident: the owner allowed the crawler and a bot rule blocks it anyway.

served blocked in robots.txt (a deliberate choice) blocked by the server or CDN despite robots.txt

Grades across the measured web

The same weighted audit used on a single site, applied to every domain in the index.

Look up any domain

If it is already in the index you get its page and its history. If it is not, you can measure it live right now.

The series

Method, in full. Domains come from the Tranco research list (a 30-day average of five rankings, built to be reproducible and citable). For each domain we request the homepage over HTTPS six times — once with a Chrome user-agent and once with each crawler's own published user-agent — plus /robots.txt, /llms.txt, /llms-full.txt, /sitemap.xml and /.well-known/security.txt, with a 12-second timeout each. A crawler counts as served when it receives at least 60% of the text a browser received. Scores are the weighted audit published on every domain page. The raw record of every domain is at /api/laboratorio/agentindex.

One vantage point, one page. All requests leave from a single server in Europe and hit the homepage only. Edge networks answer differently from different places, and a site can be open on its homepage and closed on its product pages.

Blocking is a choice, not a failure. A publisher that shuts out AI crawlers is trading reach for control, and several of the biggest have done it deliberately. This index records what is true, and never scores whether it should be.

Infrastructure is separated, not hidden. Domains that answer without a real HTML page — telemetry endpoints, CDNs, resolvers — are measured but kept out of the site statistics, because ranking them against websites would be meaningless. The count of those is shown with every figure.

A snapshot becomes a series. A single sweep says how the web is today; the point of repeating it is the trend, and a trend needs more than one sweep. Until there are several, this page says so instead of drawing a line through one dot.