日本語EN
Parallel World
Scan your site free

Not only "is the site open to AI" but "is it appropriately closed": how we measure it, and what we have seen (October 2026)

Every tool that checks a site for AI measures how open it is to AI. Parallel World also measures whether it is appropriately closed. This page sets out the idea and the numbers we have recorded so far, each with its source and a line on what it does not prove.

Recorded 22 October 2026. Every number is one that is already written in our research record (six claims in knowledge/claims in the repository); nothing new was counted for this page. We do not name other companies' sites; we give counts only.

日本語

What this page argues

  • For AI bots, more open is not always better. An open door is open to AI bots and to everyone else, and what comes back through it differs from site to site.
  • A good site is open where it should be and closed where it should be. "Closed" does not mean refusing everything. It means telling apart those who only borrow a name, not writing paths into robots.txt that you would rather not publish, and not handing an agent the actions it should not take.
  • As far as we could find, no document defines "too open" or "appropriately closed", and no tool, in Japan or elsewhere, scores that balance (C1). So for each axis we show the site's number next to an official reference value, and we give a verdict only where there is a specification.

Words used on this page

  • We / our: migurei (a subsidiary of HBLAB), which runs Parallel World.
  • UA: User-Agent, the name a visitor gives itself. It is a claim, not proof of identity.
  • Feed (IP list): the list of IP addresses of their own bots that OpenAI, Google and others publish.
  • A partner site we measure: one site whose access log we hold. Not named.
  • Other sites: Japanese sites checked by our scanner (read-only). Not named; counts only.

C1 There is no official standard for "too open"

robots.txt, Content Signals, aipref, IP lists, reverse DNS, Web Bot Auth and WebMCP each give a mechanism for opening or closing. How far to open is left to the site owner.

Numbers

  • Of the official documents we checked on 22 October 2026, 0 define when a site is "too open" or "appropriately closed".
  • We checked RFC 9309; the IETF aipref-vocab draft (version 08, 13 September 2026, marked "NOT REFLECT CONSENSUS"); the Web Bot Auth drafts; Content Signals; Cloudflare's verified bots policy (8 October 2026); the bot pages of OpenAI, Anthropic, Google, Perplexity and Apple; and the Agency for Cultural Affairs paper, among others.
  • Among Japanese AI-visibility tools, ミエルカGEO, AKARUMI and LLMOチェキ publish nothing on how closed a site is.

Source

What this does not prove: "We did not find it" is not "it does not exist". We did not read the 2026 Intellectual Property Strategic Program or the ACP and AP2 specifications directly. Tools such as Cloudflare AI Crawl Control already show crawl-to-referral per bot for sites on Cloudflare; our only difference is that we can look at sites that do not use Cloudflare.

C2 A name is not an identity

A UA is only a claim. What identifies a bot is the IP list its operator publishes, reverse DNS, or a signature. A site that lets a visitor in by name lets in anyone who borrows the name.

Numbers

  • 18 of 21 other sites answered a request that called itself ChatGPT-User, Claude-User, Perplexity-User and the like, from an IP that is not theirs, the same way they answered our scanner. One request to the top page per UA (540 rows, 90 scans, 3 to 21 October 2026).
  • The exceptions are 3 sites. Two returned 403 to ChatGPT-User (all 3 scans); one returned 403 to MistralAI-User (all 8 scans).
  • Our own pw.migurei.ai (1 to 22 October 2026): 69 requests called themselves Claude-User from IPs outside Anthropic's list; 68 of them were one IP probing for .env and .git, all answered 404. Every OpenAI, Google, Bing and Perplexity name matched the operators' lists.
  • Our own migurei.ai (7 to 21 October 2026): 12,801 probing requests, 0 of them 2xx; more than 3,000 carried a name from the bot registry (about 30 names). One IP sent 200 requests under 13 AI bot names in about 2 minutes; 175 got 404.
  • A partner site we measure (19 June to 2 October 2026): 85,290 probing requests carried a bot UA. 46,188 came from an IP that does not match the list, 39,101 used a name that has no list, and 1 came from a verified bot.

Source

  • Our ClickHouse pw.ua_probe and check_result (criterion internal.fetcher_ua_response), the probe class of our own access logs, and the IP lists in the bot registry of 20 October 2026. Read on 22 October 2026.
  • Claim files: sites-answer-fake-ai-ua-like-anyone.md and ai-bot-names-used-to-mask-probes.md (level: our observation).

What this does not prove: We do not know how each site treats a real fetcher (IP verified). A CDN may let the real IP in and still block a borrowed name. We asked for the top page once per UA and did not retry. A 200 for a borrowed name does not mean the site was harmed. An IP outside the list is not proof of a fake (the partner site used an older list, and most of its web-server log holds only Cloudflare edge IPs). Of the 81,151 probing requests the partner site answered 200, what came back was the size of an app shell (183 to 188 KB); we have not seen a file leak.

C3 What we give and what we get back

On the partner site we measure, AI bots fetched from tens to over a thousand pages for each person the app sent back. Opening to AI bots today looks like a trade that gives more than it receives.

Numbers

  • For a worldwide comparison, see Cloudflare Radar.
  • A partner site we measure (19 June to 2 October 2026): OpenAI about 88 to 1, Perplexity about 407 to 1, Anthropic about 1,240 to 1. For OpenAI that is 317,851 requests from OAI-SearchBot and ChatGPT-User against 3,605 human requests with a ChatGPT referrer.
  • Our own pw.migurei.ai and migurei.ai: crawling, and 0 people from AI apps in the window we read. The ratio is infinite, and the "get back" side is not yet measured.
Cloudflare Radar chart "Crawl-to-refer ratio": for each AI company and other search engine, the ratio of HTML page crawl requests to HTML page referrals from that platform, with the share of each bot. Period: 16 October 2026 00:00 UTC to 23 October 2026 01:45 UTC.
Source: Cloudflare Radar (CC BY 4.0), as of 23 October 2026 (period: 16 to 23 October 2026). These are worldwide figures, not the figures of any single site. This ratio is not a verdict. Restricting crawlers can also change how the site appears in AI search; if you change it, measure before and after.

Source

  • Definition of the crawl-to-referral ratio: HTML requests from a platform's crawlers divided by HTML referrals from that platform (Cloudflare blog, 28 August 2025).
  • The partner site: our bot request log and the table of human-request totals.
  • Claim file: ai-crawl-to-refer-hundreds-to-one.md (level: our observation for the partner site).

What this does not prove: The ratio is inflated where an app sends no referrer. Training bots by nature bring no visitor. ChatGPT-User fetches a page for one answer; that is not a human visit. We do not say which ratio is "reasonable". And the only way an owner can change the ratio is to block, which may lower how often the site appears in search or in answers.

C4 A file that invites AI is also a map

robots.txt, sitemaps and llms.txt can be read by anyone, even when written for AI bots. A path written there is a published path.

Numbers

  • About 10 of 14 other sites list /admin under Disallow in robots.txt. Many of them use Shopify's default list.
  • One site lists a few internal CMS paths under Disallow. The sitemap of another holds a test URL that answers 400. None of the sitemaps we looked at contained a staging host.
  • Our own pw.migurei.ai has 0 Disallow lines in robots.txt (open on purpose, by design).

Source

  • RFC 9309, section 1, "These rules are not a form of access authorization", and section 3, "Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable".
  • OWASP WSTG-INFO-03 (checking robots.txt, sitemaps and .well-known for leakage; it does not mention llms.txt and has no list of "sensitive" paths).
  • The raw robots.txt our scanner fetched in its latest run, read on 22 October 2026.
  • Claim file: robots-disallow-lists-paths-publicly.md (level: outside source for the principle; our observation for the counts).

What this does not prove: A path being named in robots.txt does not mean it opens. Our scanner does not request a disallowed path (it obeys robots.txt), so we do not know. We also do not score which path is "sensitive": there is no official list; we only list the paths. Most come from platform defaults, so some would say the weight is small.

C5 If an agent can act, it needs limits

When an agent calls tools or buys on a person's behalf, the question moves from "can it read" to "what may it do": switch it off on pages that do not need it, and mark actions that cannot be undone.

Numbers

  • WebMCP (W3C Community Group Draft Report, 20 October 2026; not on the W3C Standards Track) has Permissions-Policy: tools=() to turn tool calls off. The default is the page itself ('self').
  • An action with a significant real-world effect, or one that cannot be undone, can carry consequentialHint, so the agent asks the person first.
  • Of the sites we looked at, 0 had WebMCP on. Our own pw.migurei.ai has it off too; its MCP server is read-only and returns only what is already on the pages.

Source

  • WebMCP (Web Machine Learning Community Group): the tools feature of Permissions-Policy, section 6.4.1 "Disable access to WebMCP APIs in documents where WebMCP is not an intended capability", consequentialHint, untrustedContentHint, exposedTo. Prompt injection is "primarily the responsibility of the agent".
  • Claim file: webmcp-site-can-turn-tools-off.md (level: outside source).

What this does not prove: No site in our sample has WebMCP on, so we cannot say how it is used in practice. The draft will change.

What Parallel World measures, and what it does not

  • For each axis we show the site's number next to an official reference value (RFC 9309, the WebMCP draft and so on).
  • We give a verdict only on axes that have an official specification (how robots.txt is written, WebMCP headers). On axes without one (C1, and a "reasonable ratio" in C3) we show the numbers and do not score them.
  • We use no scoring criteria we made up. We store the result of the official API or specification as it is.

What this page as a whole does not prove

  • The counts and periods are small. 21 other sites (14 for robots.txt), one partner site we measure and two sites of our own. This is not the state of the whole industry.
  • We do not say that closing is better. Closing may lower how often a site appears in AI search and answers. That is why we measure, per axis, the effect of opening and the effect of closing separately.
  • We wrote this and we measured it. No independent party has checked it.

Official sources

The claims this page uses

  • knowledge/claims/no-official-verdict-for-too-open.md
  • knowledge/claims/sites-answer-fake-ai-ua-like-anyone.md
  • knowledge/claims/ai-bot-names-used-to-mask-probes.md
  • knowledge/claims/ai-crawl-to-refer-hundreds-to-one.md
  • knowledge/claims/robots-disallow-lists-paths-publicly.md
  • knowledge/claims/webmcp-site-can-turn-tools-off.md