Monthly report · October 2026
The Agent-Ready Web: October 2026
What AI agents find on 10,000 popular US web origins from the Chrome UX Report: WebMCP tools, AI-crawler access, llms.txt, and SEO. Measured by a real crawl in Chrome, with every figure shown against the sites it was measured on.
4.2%
had WebMCP tools active in Chrome
213 of 5,050 (4.2%) sites a browser loaded
12.2%
publish an llms.txt
585 of 4,805 (12.2%) sites where the file could be checked
32
more sites registered WebMCP tools that stock Chrome did not activate yet
of 5,050 sites a browser loaded
12%
refused a request naming an AI search agent their robots.txt allows
594 of 4,947 (12%) sites where one was probed
How we measured
- Sample: Chrome UX Report top origins, United States, September 2026 (rank ≤ 10,000) (source): 10,000 origins, ranked in popularity bands (top 1,000, 5,000 and 10,000) by Chrome page loads from US users who share usage data. Origins are addresses, not companies: the list includes sign-in, school-portal and adult sites, and some brands appear more than once.
- Coverage: 10,000 crawled (9,619 completed); 5,119 served our crawler at least one page, and 5,050 of those were loaded in a browser. The rest served no pages: many large sites block automated crawlers, others redirect to a sign-in host. So these figures describe sites that let a crawler in, not the whole list.
- Crawl: 2026-10-23 to 2026-10-25 by ConvertlyftBot, the crawler behind Convertlyft, build cefe2e2. robots.txt obeyed; no tool called, no form submitted.
- Depth: the homepage and at most one more page per origin, loaded in Chrome. Findings are about those pages, not whole sites.
- AI crawlers: we asked for a page sending each AI agent's name from our own server, not from the agent's own network. A refusal shows a rule keyed on the agent's name; some sites refuse anyone claiming to be a known bot and may still serve the real one.
- What counts: SEO and tool findings use only issue types the crawler's own test harness verifies. Each share's denominator is the sites where that check could look.
- Names: no site is named for a problem. The directory names only sites with WebMCP tools that work.
WebMCP adoption
213 of 5,050 (4.2%) sites that a browser loaded had at least one WebMCP tool active in stock Chrome on the pages we loaded. 1 of 5,119 (<0.1%) sites declared a tool form in the HTML they served. Across sites with active tools we counted 2,549 tools: 0 declared on forms and 2,549 registered from script.
Levels are over the 5,119 sites that served pages. L3 and L4 were not measured this month: they depend on an agent-accessibility check that did not run in this crawl, so no site is shown above L2.
The tools sites register
212 of 213 (99.5%) sites with active tools register the identical set of 12 tools: add_to_cart, browse_store, cancel_cart, get_cart, get_product, get_ucp_profile, manage_orders, proceed_to_checkout, search_catalog, search_shop_policies_and_faqs, show_variant, update_cart_lines. An identical set across many sites means the tools come from one shared platform, not from each site's own code.
- 1 of 213 (0.5%): 5 tools (list_app_categories, list_app_guides, list_popular_apps, list_top_searched_apps, …)
Built, not yet switched on
On 32 of 5,050 (0.6%) sites, Chrome's tool registry held WebMCP tools but none was active in stock Chrome, which usually means the tools wait for Chrome's testing flag or an origin trial: amplify.com, chatgpt.com, detroitaxle.com, helpinghandsact.com, hentaihaven.xxx, herrschners.com, higgsfield.ai, musicgoround.com, na.store.square-enix-games.com, padlet.com, pickdawgz.com, playitagainsports.com, rog.asus.com, simplicity.com, smkw.com, us.jellycat.com, vercel.com, aboutamazon.com, asus.com, bestmoney.com, cloudflare.com, coldwatercreek.com, fishusa.com, gamenerdz.com, geappliances.com, legalzoom.com, nvidia.com, on.com, softsurroundings.com, spicylingerie.com, thumbtack.com, verizon.com.
AI crawler access
4,248 of 5,114 (83.1%) sites served a robots.txt. Where an allowed agent was probed, sites refused a request naming one: training crawlers on 792 of 4,727 (16.8%) sites, AI search agents on 594 of 4,947 (12%), agents that fetch a page when a user asks on 312 of 4,930 (6.3%). We sent the agent's name from our own server, so a refusal shows a rule keyed on the name; the real crawler, coming from its own network, may be treated differently.
| Agent | Purpose | Allowed by robots.txt | Refused a request in its name, though allowed |
|---|---|---|---|
| Amazonbot | Search | 4,781 of 5,119 (93.4%) | 446 of 4,511 (9.9%) |
| Amzn-SearchBot | Search | 5,081 of 5,119 (99.3%) | 95 of 4,795 (2%) |
| Amzn-User | Fetch when a user asks | 5,091 of 5,119 (99.5%) | 91 of 4,777 (1.9%) |
| anthropic-ai | Training | 4,743 of 5,119 (92.7%) | not measured |
| Applebot | Search | 5,053 of 5,119 (98.7%) | 150 of 4,784 (3.1%) |
| Applebot-Extended | Training | 4,743 of 5,119 (92.7%) | not measured |
| Bytespider | Training | 4,636 of 5,119 (90.6%) | 595 of 4,338 (13.7%) |
| CCBot | Training | 4,591 of 5,119 (89.7%) | 172 of 4,374 (3.9%) |
| ChatGPT-User | Fetch when a user asks | 4,885 of 5,119 (95.4%) | 208 of 4,627 (4.5%) |
| Claude-SearchBot | Search | 5,026 of 5,119 (98.2%) | 152 of 4,760 (3.2%) |
| Claude-User | Fetch when a user asks | 5,022 of 5,119 (98.1%) | 109 of 4,803 (2.3%) |
| ClaudeBot | Training | 4,650 of 5,119 (90.8%) | 446 of 4,408 (10.1%) |
| cohere-ai | Training | 4,762 of 5,119 (93%) | 315 of 4,515 (7%) |
| DuckAssistBot | Search | 5,023 of 5,119 (98.1%) | 142 of 4,694 (3%) |
| Google-CloudVertexBot | Search | 4,953 of 5,119 (96.8%) | 129 of 4,652 (2.8%) |
| Google-Extended | Training | 4,706 of 5,119 (91.9%) | not measured |
| GPTBot | Training | 4,595 of 5,119 (89.8%) | 252 of 4,361 (5.8%) |
| meta-externalagent | Training | 4,748 of 5,119 (92.8%) | 215 of 4,507 (4.8%) |
| meta-externalfetcher | Fetch when a user asks | 5,015 of 5,119 (98%) | 150 of 4,732 (3.2%) |
| meta-webindexer | Search | 5,047 of 5,119 (98.6%) | 143 of 4,753 (3%) |
| MistralAI-User | Fetch when a user asks | 5,030 of 5,119 (98.3%) | 156 of 4,695 (3.3%) |
| OAI-AdsBot | Search | 5,116 of 5,119 (>99.9%) | 48 of 4,877 (1%) |
| OAI-SearchBot | Search | 5,006 of 5,119 (97.8%) | 162 of 4,762 (3.4%) |
| Perplexity-User | Fetch when a user asks | 5,023 of 5,119 (98.1%) | 174 of 4,797 (3.6%) |
| PerplexityBot | Search | 4,861 of 5,119 (95%) | 213 of 4,651 (4.6%) |
AI crawlers: what each one is for and how to allow or block it
Files for agents
SEO on the pages we read
Across the 5,024 sites the crawler could score (homepage plus at most one more page), the median of our own 0–100 crawl score was 80. The most common verified issues, counted when they appear on at least one of those pages:
Data and citation
Download the figures: CSV.
Download the facts for every site in the sample (10,000 rows): per-site CSV. Each row holds only what anyone can see on the site itself: its robots.txt rules, its llms.txt and other agent files, and the WebMCP tools Chrome found active, with the crawl date and crawler build. Visit any origin to check its row. Per-site SEO scores and AI-agent refusals are left out: they depend on our crawler and our network, so they are published only as totals.
Fingerprints (SHA-256): per-site CSV df808d437eec81f7a49c7b46f65bd144ef685b5d3afda7fb547d82876dc50185; raw crawl rows e29ed334a4d5f0e4d9da5b4d1ce8efde7467cc6ea6b84b47a9331c8fee02f0e0. The raw rows are archived unchanged, so this report can always be traced back to them.
Cite as: Convertlyft, “The Agent-Ready Web: October 2026”, WebMCP Agency, https://webmcpagency.com/report/2026-10. Licensed CC BY 4.0.