---
title: "C. Open and closed | Parallel World - migurei"
description: "C. Open and closed: each principle of the Parallel World lab and the 15 claims under it, with evidence level, sources, what it does not prove and what a site owner can do."
lang: "en"
url: "https://pw.migurei.ai/en/handbook/open-and-closed"
---

# C. Open and closed

The other side of group A. Group A asks whether a site exists for AI; this group asks who the site is open to and what it gets in return. Each door opened for AI is open to others too; a good site is open in the right places and closed in the right places.

Built from our knowledge base at every deploy. 31 principles, 200 claims. Newest note: 2026-11-12.

[日本語](https://pw.migurei.ai/ja/handbook/open-and-closed) · [Handbook index](https://pw.migurei.ai/en/handbook)

Each principle is followed by the claims it rests on. Evidence levels are explained on the handbook index.

## C1. Opening has a price

Every door opened to AI is also open to others and may bring nothing back; a good site is open in the right places and closed in the right places, not open everywhere.

**Evidence for the principle:** hypothesis

### [external source] No official source defines when a site is "too open" or "too closed" to AI bots; standards only give mechanisms. On 2026-10-30 Google, Microsoft and OpenAI announced support for a new robots.txt group for AI training, still with no threshold.

- **Evidence:** external source. Recorded 2026-10-22, re-check by 2026-12-22
- **Basis:** Standards documents, AI vendors' bot pages and blogs, Japanese government guidance and tool vendors' pages, read 2026-10-22 to 2026-11-02.
- **Sources:** [contentsignals.org](https://contentsignals.org/) (2026-10-08)
- **Does not prove:** "Not found" is not "does not exist". Some Japanese policy documents and agent-commerce specs were not read directly. Our scan of .jp sites shows what they declare in robots.txt, not how bots behave.
- **What a site owner can do:** Decide which bots to allow in robots.txt per purpose (search, AI answers, training); no official threshold will decide for you. If you only want to opt out of training, watch the new __usage_ai-training group (Microsoft from 2027-02-01).

### [our lab observation] After migurei.ai unblocked GPTBot (blocked 28.6 hours, 2026-10-20 to 2026-10-21), OAI-SearchBot fetched 26 pages within 5 hours and GPTBot returned about 25 hours later; on pw.migurei.ai, where GPTBot is allowed, OAI-SearchBot fetched 0 pages in 30 days to 2026-11-04.

- **Evidence:** our lab observation. Recorded 2026-11-04, re-check by 2026-12-04
- **Basis:** Our own server logs for migurei.ai and pw.migurei.ai with OpenAI bot IPs verified, 2026-10-21 to 2026-11-04, and OpenAI's crawler documentation.
- **Sources:** [developers.openai.com](https://developers.openai.com/api/docs/bots)
- **Does not prove:** Verified logs cover only the last 35 minutes of the block, so we do not know whether GPTBot stopped. The 26-page burst may be chance, not caused by the block. One site, one time. It says nothing about ChatGPT citing the site.
- **What a site owner can do:** Do not judge OpenAI's access to your site by OAI-SearchBot page fetches alone: if both bots are allowed, OpenAI may use one crawl for both. Count GPTBot and OAI-SearchBot together, and allow about 24 hours for robots.txt changes to apply.

## C2. A name is not an identity

A User-Agent is only a self-declaration; a bot's identity is its IP address in the vendor's published feed, reverse DNS, or a signature (Web Bot Auth). A site that lets bots in by name lets in everyone who borrows the name.

**Evidence for the principle:** external source (partly: our lab observation)

### [our lab observation] 18 of 21 sites we scanned (2026-10-03 to 2026-10-21) answered a request that claimed to be an AI fetcher, such as ChatGPT-User or Claude-User, sent from a non-AI IP address, exactly as they answered our own scanner.

- **Evidence:** our lab observation. Recorded 2026-10-22, re-check by 2027-01-22
- **Basis:** Our scanner's checks of 21 sites, 90 scans with AI fetcher user agents, 2026-10-03 to 2026-10-21; a follow-up on 61 .jp sites, 2026-11-02.
- **Sources:** our own data, no outside link
- **Does not prove:** It does not show how real fetchers from verified IPs are treated; a CDN may admit real IPs and still block fake names. One request per user agent, home page only. A 200 for a fake name is not harm in itself: public content may need no protection.
- **What a site owner can do:** Fetch your home page with the ChatGPT-User and Claude-User user agents and compare with a normal request. If you need to tell real bots from fakes, check the IP against the AI company's published list, not the name.

### [our lab observation] On our sites many requests hunting for secret files (.env, .git) use AI bot names: on migurei.ai, 2026-10-24 to 2026-11-07, only 105 of 499 requests (21.0%) under six AI and search bot names could be verified.

- **Evidence:** our lab observation. Recorded 2026-10-22, re-check by 2027-01-22
- **Basis:** Our own logs on pw.migurei.ai and migurei.ai (October to November 2026) and a large Vietnamese site's logs, checked against vendors' IP lists and signatures.
- **Sources:** our own data, no outside link
- **Does not prove:** An IP outside a vendor list is not always fake: lists change, and some logs hold only proxy IPs. No probe obtained a secret: on our two sites, 0 probes got a success response. It says nothing about other sites.
- **What a site owner can do:** Never trust a user agent alone. Verify AI bots by the vendor's published IP list or Web Bot Auth signature, return 404 for .env, .git and similar paths, and count only verified requests as bot visits.

### [external source] AI vendors change their published bot IP lists without notice: between our 2026-10-21 copy and 2026-11-04, 3 of 16 lists changed (PerplexityBot 8 to 13 ranges, Perplexity-User 4 to 10, ChatGPT-User 234 to 235), so checks using our old copy ran on outdated lists.

- **Evidence:** external source (partly: our lab observation). Recorded 2026-11-04, re-check by 2026-11-18
- **Basis:** Perplexity's and OpenAI's published bot IP lists, downloaded 2026-11-04 and compared with our 2026-10-21 copy; our own server logs.
- **Sources:** [www.perplexity.ai](https://www.perplexity.ai/perplexitybot.json) (2025-02-07), [www.perplexity.ai](https://www.perplexity.ai/perplexity-user.json) (2025-10-17), [openai.com](https://openai.com/chatgpt-user.json) (2026-10-20)
- **Does not prove:** We found no real request judged wrongly because of the old lists: every PerplexityBot IP we logged fits both old and new ranges. Requests carrying a bot name but no recorded IP could not be rechecked, and may be fake or real.
- **What a site owner can do:** If you verify AI bots by IP, re-download each vendor's IP list regularly (ours went stale within two weeks) rather than bundling a fixed copy, and keep identifying bots by IP range, not by user-agent name alone.

### [external source] On 2026-11-04 Cloudflare made "Personal Assistants" (AI agents acting for a person) a separate group in Bot Analysis, with request counts per agent, and lets site owners choose "Allow (do not block)", "Block on pages with ads" or "Block".

- **Evidence:** external source. Recorded 2026-11-05, re-check by 2026-12-05
- **Basis:** Cloudflare's blog post of 2026-11-04, read 2026-11-05, plus our own server logs on migurei.ai and pw.migurei.ai, 2026-10-28 to 2026-11-05.
- **Sources:** [blog.cloudflare.com](https://blog.cloudflare.com/personal-agents-visibility-control/) (2026-11-04)
- **Does not prove:** Which plans get it, the default setting, and which bots Cloudflare puts in the group. In our logs only 6 requests over 9 days came from these agents' published IP ranges; most others using their names were fakes probing for secret files, so no effect is visible yet.
- **What a site owner can do:** Open Bot Analysis in your Cloudflare dashboard, see which personal agents reach you, and choose the setting on purpose. Do not judge agents by name alone: check requests against each vendor's published IP list.

### [our lab observation] On migurei.ai, in the 14 days to 2026-11-06, 100 requests used the DuckAssistBot name: the 20 signed with Web Bot Auth all had valid signatures and fetched normal pages, while 79 unsigned ones probed for secret files such as .env.

- **Evidence:** our lab observation (partly: external source). Recorded 2026-11-06, re-check by 2026-12-06
- **Basis:** Our own server logs on migurei.ai, 2026-10-23 to 2026-11-06, signatures checked against the published key directory; DuckDuckGo's DuckAssistBot help page.
- **Sources:** [assistbot.duckduckgo.com](https://assistbot.duckduckgo.com/) (2026-11-03), [pw.migurei.ai](https://pw.migurei.ai/), [duckduckgo.com](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) (2026-11-06), [duckduckgo.com](http://duckduckgo.com/duckassistbot.html), [duckduckgo.com](https://duckduckgo.com/duckassistbot.json) (2026-11-04), [assistbot.duckduckgo.com](https://assistbot.duckduckgo.com/.well-known/http-message-signatures-directory)
- **Does not prove:** A valid signature proves the signer holds a key listed at assistbot.duckduckgo.com, not that DuckDuckGo owns it; we have not matched IPs with DuckDuckGo's published list. Whether DuckAssist has users in Japan or sends visitors is unknown.
- **What a site owner can do:** Do not trust a bot by its name. Verify DuckAssistBot by its Web Bot Auth signature or DuckDuckGo's published IP list, and block requests for paths like /.env, which a real crawler never needs.

### [external source] On 2026-11-05 Cloudflare launched a Trusted Agent Delegation coalition and named Web Bot Auth (signed requests) as the way to recognise personal AI agents; site owners can already see personal-agent traffic and set allow or block rules per agent and per site section.

- **Evidence:** external source. Recorded 2026-11-06, re-check by 2026-12-06
- **Basis:** Cloudflare's blog post of 2026-11-05, read 2026-11-06 through a web reading tool's summary.
- **Sources:** [blog.cloudflare.com](https://blog.cloudflare.com/trusted-agent-delegation/) (2026-11-05)
- **Does not prove:** We read a tool's summary, not every sentence of the original. It does not show that OpenAI, Anthropic, Google or Perplexity will sign their requests: in 14 days of our logs, none of their requests were signed.
- **What a site owner can do:** If you use Cloudflare, look at the personal-assistants view in Security Analytics and set per-agent policies only for the site sections you care about. Prefer signed identity over user-agent names when deciding.

**Also relevant here:**

- [[external source] Cloudflare continuously scores Verified bots on whether they respect robots.txt and HTTP 429 (slow down) responses, plans to publish trust levels on Radar and let site owners pick a threshold; bots apply through the BotBase form, with no minimum traffic.](https://pw.migurei.ai/en/handbook/bots-and-apps#c-c8773272)

## C3. Giving and getting are out of balance

AI bots fetch hundreds to thousands of pages for each human visit the app sends back; apart from Google, opening to AI bots today gives more than it gets.

**Evidence for the principle:** external source (partly: our lab observation)

### [external source] AI bots fetch hundreds to thousands of pages per visitor their app sends back: Cloudflare Radar for 2026-10-16 to 2026-10-23 shows Perplexity 3.3K:1, OpenAI 590:1, Anthropic 495:1, against Google 4.5:1.

- **Evidence:** external source (partly: our lab observation). Recorded 2026-10-22, re-check by 2026-12-22
- **Basis:** Cloudflare Radar's global crawl-to-refer chart, 2026-10-16 to 2026-10-23; plus our own logs on pw.migurei.ai and migurei.ai and a large Vietnamese site's logs.
- **Sources:** [radar.cloudflare.com](https://radar.cloudflare.com/ai-insights) (2026-10-23), [radar.cloudflare.com](https://radar.cloudflare.com/about), [blog.cloudflare.com](https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/) (2025-08-28)
- **Does not prove:** Ratios are inflated when apps drop the referrer; on our sites 64–88% of human visits had no referrer, so our ratios are upper bounds. Training bots send no visitors by design. No Japan slice. No ratio is 'fair', and this does not say which bot to block.
- **What a site owner can do:** Compute your own ratio per vendor over 30 days: verified AI bot page fetches versus human visits with an AI referrer or utm_source. Read it beside Cloudflare Radar, and remember that blocking lowers the ratio but also your AI visibility.
- **Also under:** [D5](https://pw.migurei.ai/en/handbook/agent-world#d5)

**Also relevant here:**

- [[external source] OpenAI states that if a site allows both OAI-SearchBot and GPTBot, it may use one crawl for both search and training, so seeing no OAI-SearchBot page fetches does not mean your pages are missing from OpenAI's search data.](https://pw.migurei.ai/en/handbook/bots-and-apps#c-1ba1aaeb)
- [[external source] Many visits from AI answers arrive without a referrer (vendor figures: 70.6% and 35.7%); ChatGPT on the web adds a utm_source tag naming ChatGPT to cited links, while the ChatGPT and Claude mobile apps drop the referrer (April 2026 snapshot).](https://pw.migurei.ai/en/handbook/japanese-buyers#c-1d10413e)

## C4. Files that invite AI are also a map

robots.txt, the sitemap and llms.txt written for AI bots can be read by anyone; every path written there is a public path.

**Evidence for the principle:** external source

### [our lab observation] On 2026-11-04, two of our own app hosts that the llms.txt of migurei.ai pointed AI to had no usable robots.txt: m-trans.migurei.ai returned its app's HTML page with status 200 for robots.txt and llms.txt, and m-rag.migurei.ai redirected robots.txt to a login page on hub.migurei.ai.

- **Evidence:** our lab observation. Recorded 2026-11-05, re-check by 2026-12-05
- **Basis:** One plain fetch of each URL by our scanner on 2026-11-04, compared with RFC 9309 (the robots.txt standard).
- **Sources:** [migurei.ai](https://migurei.ai/llms.txt) (2026-11-04), [migurei.ai](https://migurei.ai/en/applications/ai-translation) (2026-11-04)
- **Does not prove:** We saw no bot read this HTML as robots.txt and do not know what any bot made of it. One fetch per URL. We removed the m-trans.migurei.ai line from llms.txt on 2026-11-05; the host itself is unchanged.
- **What a site owner can do:** Fetch /robots.txt and /llms.txt on every host you point AI to. Each should return plain text or a real 404, not an HTML app page with status 200 or a redirect to a login page. Do not list hosts in llms.txt that fail this check.

**Also relevant here:**

- [[external source] Listing paths in robots.txt makes them public: RFC 9309 says the rules are not access control, and about 10 of 14 sites in our latest scan listed /admin or CMS paths under Disallow, mostly platform defaults.](https://pw.migurei.ai/en/handbook/bots-and-apps#c-8e76e4db)

## C5. An agent that can act needs limits

When an agent calls tools or buys on a person's behalf, the question moves from "can it read" to "what may it do": turn tools off on pages that do not need them, and mark actions that cannot be undone.

**Evidence for the principle:** external source

### [external source] The WebMCP draft (dated 2026-10-20) lets a site switch off AI agents calling tools on its pages with the header Permissions-Policy: tools=(), and mark irreversible actions with consequentialHint so the agent asks the user first.

- **Evidence:** external source. Recorded 2026-10-22, re-check by 2026-12-22
- **Basis:** The WebMCP Community Group Draft Report dated 2026-10-20 (not a W3C standard), read 2026-10-22.
- **Sources:** [webmachinelearning.github.io](https://webmachinelearning.github.io/webmcp/) (2026-10-20)
- **Does not prove:** No site in our sample has turned on WebMCP yet, and the draft can still change. The draft puts defence against prompt injection mainly on the agent, not on the site.
- **What a site owner can do:** If your pages offer no WebMCP tools, consider sending Permissions-Policy: tools=(). If you add tools, mark payments, bookings and other irreversible actions with consequentialHint.

## C6. One main door

The same page served through several doors (http://, extra ports, extra hosts) is several copies that others can read, alter on the way, or cite instead of the main one. Open to AI at the declared door (canonical, https, port 443) and close the others with redirects or HSTS.

**Evidence for the principle:** external source (partly: our lab observation)

### [our lab observation] On 2026-11-04, pw.migurei.ai and migurei.ai served full pages over plain http:// without redirecting to https:// and sent no HSTS header. migurei.ai also answered on 11 extra Cloudflare ports. Only https on port 443 is the declared (canonical) address.

- **Evidence:** our lab observation (partly: external source). Recorded 2026-11-04, re-check by 2026-12-04
- **Basis:** Direct requests on 2026-11-04, our server logs (30 days; migurei.ai since 2026-10-21), Cloudflare's network ports documentation and Mozilla HTTP Observatory.
- **Sources:** [pw.migurei.ai](http://pw.migurei.ai/), [pw.migurei.ai](https://pw.migurei.ai/), [migurei.ai](http://migurei.ai/en), [migurei.ai](https://migurei.ai/en), [migurei.ai](http://migurei.ai:8080/en), [migurei.ai](https://migurei.ai:8443/en), [pw.migurei.ai](http://pw.migurei.ai:8080/), [pw.migurei.ai](https://pw.migurei.ai:8443/), [developers.cloudflare.com](https://developers.cloudflare.com/fundamentals/reference/network-ports/) (2026-11-04)
- **Does not prove:** That anyone altered content in transit (no sign of it), or that Google or AI apps chose the http version as the main one (canonical tags point to https). Requests on the extra ports were mostly scanners, and we saw no AI bots there.
- **What a site owner can do:** Request your pages over http:// and on ports such as 8080 and 8443. Redirect http to https with a 301, add a Strict-Transport-Security header, and block non-standard ports with a WAF rule if your plan allows it.

### [external source] On 2026-11-04, Mozilla HTTP Observatory graded pw.migurei.ai F (5/100) and migurei.ai F (0/100): both lack an HTTPS redirect, HSTS, X-Frame-Options, nosniff and a Content Security Policy, and migurei.ai also sets a cookie without the Secure flag.

- **Evidence:** external source. Recorded 2026-11-04, re-check by 2026-12-04
- **Basis:** Mozilla HTTP Observatory API scans on 2026-11-04 and Mozilla's published scoring table.
- **Sources:** [observatory-api.mdn.mozilla.net](https://observatory-api.mdn.mozilla.net/api/v2/scan?host=), [developer.mozilla.org](https://developer.mozilla.org/en-US/observatory/docs/tests_and_scoring)
- **Does not prove:** Observatory grades browser security, not what AI bots read; none of these headers changes what AI bots can fetch. No data shows Japanese buyers check a vendor's headers. The score after fixes is known only after a new scan.
- **What a site owner can do:** Scan your site with Mozilla HTTP Observatory and add an HTTPS redirect, HSTS, X-Frame-Options, X-Content-Type-Options: nosniff and a Content Security Policy; mark cookies Secure. Do not expect this to change AI visibility.

## C7. Say who is speaking

A mention of you that you wrote yourself, commissioned or paid for must show that to readers (and to the apps that copy it); naming the author closes the door on covert promotion that apps cannot tell apart from a real third party.

**Evidence for the principle:** external source (partly: our lab observation)

### [external source] In Japan, since 2023-10-01, advertising that consumers cannot recognise as advertising (stealth marketing) violates the Act against Unjustifiable Premiums and Misleading Representations; posts a company writes, or has others write while deciding the content, count as the company's own and must be labelled clearly (広告, PR).

- **Evidence:** external source. Recorded 2026-11-05, re-check by 2027-05-05
- **Basis:** The Consumer Affairs Agency's stealth marketing page, read 2026-11-05; the 'decides the content' reading comes from summaries of its operating standards.
- **Sources:** [www.caa.go.jp](https://www.caa.go.jp/policies/policy/representation/fair_labeling/stealth_marketing/) (2026-11-05)
- **Does not prove:** This is not legal advice. We have not read the original operating standards. It does not settle whether the rule covers English-language posts on overseas platforms, or whether a given existing post breaks it.
- **What a site owner can do:** When you write about your own product on third-party platforms (note, guest articles, listicles), or pay others and shape what they write, label it clearly as 広告 or PR and state that the author is the operator; ask a lawyer when unsure.

### [our lab observation] In one ChatGPT answer (2026-11-03), a tool marked as a paid (PR) entry in a third-party comparison article was listed first, with that article cited, and the answer carried no PR or ad label; in 18 later answers (2026-11-06 to 2026-11-08) it did not recur.

- **Evidence:** our lab observation. Recorded 2026-11-05, re-check by 2026-11-26
- **Basis:** Test questions we asked ChatGPT and Gemini, 2026-11-03 to 2026-11-08, and a reading of the cited comparison article.
- **Sources:** our own data, no outside link
- **Does not prove:** That ChatGPT named the tool because of the paid entry: it may appear in other articles or in the model's memory. One answer. That the PR label was on that entry at the time of the answer (we read the page 15 to 19 hours later).
- **What a site owner can do:** If you buy a listing in a comparison article, check how AI answers present it; do not count on the PR label being kept, and keep your own disclosure clear.

```json
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "C. Open and closed",
  "inLanguage": "en",
  "dateModified": "2026-11-12",
  "author": {
    "@type": "Organization",
    "name": "migurei"
  },
  "mainEntityOfPage": "https://pw.migurei.ai/en/handbook/open-and-closed",
  "isPartOf": {
    "@type": "CreativeWork",
    "name": "Handbook for AI agents: how a website exists for AI apps",
    "url": "https://pw.migurei.ai/en/handbook"
  }
}
```