{"openapi":"3.1.0","info":{"title":"Agent Web Index","version":"1.0.0","description":"How much of the most-visited web AI assistants can actually read, measured with real HTTP requests from each AI crawler user-agent — not inferred from robots.txt alone. The index separates what a site owner wrote in robots.txt from what the CDN or WAF refuses on top of it, tracks day-to-day changes that cannot be reconstructed after the fact, and can measure any domain on demand. No key, no signup.","contact":{"url":"https://shop.lumnika.com/ai-readiness/"},"license":{"name":"Free to use with attribution to the Agent Web Index"}},"servers":[{"url":"https://shop.lumnika.com"}],"paths":{"/ai-readiness/api/v1/domain_readiness":{"get":{"operationId":"domain_readiness","summary":"Can AI assistants actually read this domain? Returns the measured readiness score, the per-crawler verdict (served / blocked in robots","description":"Can AI assistants actually read this domain? Returns the measured readiness score, the per-crawler verdict (served / blocked in robots.txt / refused by the edge despite robots.txt allowing it), and which edge vendor answers for the host. Measured with real HTTP requests, not guessed from robots.txt alone. Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers: they are flagged robotsOnly, carry no server verdict, and are excluded from every count about what the server did. Also returns `fixes`: the ordered, domain-specific list of what to change to let the blocked crawlers in, separating what is written in the site own robots.txt from what the CDN/WAF applies on top of it.","tags":["Agent Web Index"],"parameters":[{"name":"host","in":"query","required":true,"description":"A bare domain, e.g. \"wikipedia.org\" (no scheme, no path).","schema":{"type":"string"}}],"responses":{"200":{"description":"The measurement or aggregate, plus `summary`: the same answer as one English sentence.","content":{"application/json":{"schema":{"type":"object","properties":{"endpoint":{"type":"string"},"summary":{"type":"string"},"source":{"type":"string"},"attribution":{"type":"string"}},"additionalProperties":true}}}},"400":{"description":"A parameter is missing or malformed; `error` says which one."}}}},"/ai-readiness/api/v1/web_openness":{"get":{"operationId":"web_openness","summary":"How much of the most-visited web is readable by AI assistants right now: the share of measured domains that block at least one AI crawler, and the served / robots-blocked / edge-blocked breakdown per crawler","description":"How much of the most-visited web is readable by AI assistants right now: the share of measured domains that block at least one AI crawler, and the served / robots-blocked / edge-blocked breakdown per crawler.","tags":["Agent Web Index"],"parameters":[],"responses":{"200":{"description":"The measurement or aggregate, plus `summary`: the same answer as one English sentence.","content":{"application/json":{"schema":{"type":"object","properties":{"endpoint":{"type":"string"},"summary":{"type":"string"},"source":{"type":"string"},"attribution":{"type":"string"}},"additionalProperties":true}}}},"400":{"description":"A parameter is missing or malformed; `error` says which one."}}}},"/ai-readiness/api/v1/edge_blocking":{"get":{"operationId":"edge_blocking","summary":"Who is actually doing the blocking: for each CDN/WAF vendor, the share of (domain x crawler) pairs that robots","description":"Who is actually doing the blocking: for each CDN/WAF vendor, the share of (domain x crawler) pairs that robots.txt ALLOWS and the server refuses anyway — i.e. how much of the blocking is an infrastructure default rather than a decision the site owner made. Each vendor also comes broken down per crawler, which separates a blanket wall (same rate for every crawler) from a managed block list that names some AI user-agents and not others.","tags":["Agent Web Index"],"parameters":[{"name":"limit","in":"query","required":false,"description":"How many vendors to return, highest contradiction rate first (default 12). Vendors with fewer than 200 allowed pairs are left out of the table rather than reported on thin evidence.","schema":{"type":"integer"}}],"responses":{"200":{"description":"The measurement or aggregate, plus `summary`: the same answer as one English sentence.","content":{"application/json":{"schema":{"type":"object","properties":{"endpoint":{"type":"string"},"summary":{"type":"string"},"source":{"type":"string"},"attribution":{"type":"string"}},"additionalProperties":true}}}},"400":{"description":"A parameter is missing or malformed; `error` says which one."}}}},"/ai-readiness/api/v1/recent_changes":{"get":{"operationId":"recent_changes","summary":"What CHANGED: the domains that recently started or stopped blocking a specific AI crawler, with the day the flip was observed and whether it happened in robots","description":"What CHANGED: the domains that recently started or stopped blocking a specific AI crawler, with the day the flip was observed and whether it happened in robots.txt or at the edge. This cannot be reconstructed after the fact from any public source — it exists only because the index made the same requests the day before and the day after. Use it to answer 'who just blocked/unblocked ChatGPT, Claude, Perplexity...' or to watch one domain over time.","tags":["Agent Web Index"],"parameters":[{"name":"days","in":"query","required":false,"description":"Look-back window in days (default 30, max 365).","schema":{"type":"integer"}},{"name":"host","in":"query","required":false,"description":"Restrict to one domain, e.g. \"nytimes.com\" (optional).","schema":{"type":"string"}},{"name":"limit","in":"query","required":false,"description":"How many changes to return, most recent first (default 50, max 500).","schema":{"type":"integer"}}],"responses":{"200":{"description":"The measurement or aggregate, plus `summary`: the same answer as one English sentence.","content":{"application/json":{"schema":{"type":"object","properties":{"endpoint":{"type":"string"},"summary":{"type":"string"},"source":{"type":"string"},"attribution":{"type":"string"}},"additionalProperties":true}}}},"400":{"description":"A parameter is missing or malformed; `error` says which one."}}}},"/ai-readiness/api/v1/measure_domain":{"get":{"operationId":"measure_domain","summary":"Measure a domain RIGHT NOW instead of reading the archive: 9 real HTTP requests, one from a browser user-agent and one per AI crawler user-agent, plus robots","description":"Measure a domain RIGHT NOW instead of reading the archive: 9 real HTTP requests, one from a browser user-agent and one per AI crawler user-agent, plus robots.txt / llms.txt / sitemap. Use it for any site the index has not reached yet, or when the caller wants a fresh verdict after changing robots.txt or a WAF rule. Returns the same `fixes` list as domain_readiness, derived from the fresh measurement.","tags":["Agent Web Index"],"parameters":[{"name":"host","in":"query","required":true,"description":"A bare domain, e.g. \"wikipedia.org\" (no scheme, no path).","schema":{"type":"string"}},{"name":"force","in":"query","required":false,"description":"Measure again even if the archive already has a verdict from the last 24 hours (default false).","schema":{"type":"boolean"}}],"responses":{"200":{"description":"The measurement or aggregate, plus `summary`: the same answer as one English sentence.","content":{"application/json":{"schema":{"type":"object","properties":{"endpoint":{"type":"string"},"summary":{"type":"string"},"source":{"type":"string"},"attribution":{"type":"string"}},"additionalProperties":true}}}},"400":{"description":"A parameter is missing or malformed; `error` says which one."}}}}}}