日本語EN
Parallel World
Scan your site free

C. Open and closed

The other side of group A. Group A asks whether a site exists for AI; this group asks who the site is open to and what it gets in return. Each door opened for AI is open to others too; a good site is open in the right places and closed in the right places.

Built from our knowledge base at every deploy. 31 principles, 200 claims. Newest note: 2026-11-12.

日本語Handbook index

Each principle is followed by the claims it rests on. Evidence levels are explained on the handbook index.

Show claims by evidence level

C1. Opening has a price

Every door opened to AI is also open to others and may bring nothing back; a good site is open in the right places and closed in the right places, not open everywhere.

Evidence for the principleHypothesis

External sourceRecorded 2026-10-22, re-check by 2026-12-22

No official source defines when a site is "too open" or "too closed" to AI bots; standards only give mechanisms. On 2026-10-30 Google, Microsoft and OpenAI announced support for a new robots.txt group for AI training, still with no threshold.

Basis
Standards documents, AI vendors' bot pages and blogs, Japanese government guidance and tool vendors' pages, read 2026-10-22 to 2026-11-02.
Does not prove
"Not found" is not "does not exist". Some Japanese policy documents and agent-commerce specs were not read directly. Our scan of .jp sites shows what they declare in robots.txt, not how bots behave.
Sources
What a site owner can do

Decide which bots to allow in robots.txt per purpose (search, AI answers, training); no official threshold will decide for you. If you only want to opt out of training, watch the new __usage_ai-training group (Microsoft from 2027-02-01).

Our lab observationRecorded 2026-11-04, re-check by 2026-12-04

After migurei.ai unblocked GPTBot (blocked 28.6 hours, 2026-10-20 to 2026-10-21), OAI-SearchBot fetched 26 pages within 5 hours and GPTBot returned about 25 hours later; on pw.migurei.ai, where GPTBot is allowed, OAI-SearchBot fetched 0 pages in 30 days to 2026-11-04.

Basis
Our own server logs for migurei.ai and pw.migurei.ai with OpenAI bot IPs verified, 2026-10-21 to 2026-11-04, and OpenAI's crawler documentation.
Does not prove
Verified logs cover only the last 35 minutes of the block, so we do not know whether GPTBot stopped. The 26-page burst may be chance, not caused by the block. One site, one time. It says nothing about ChatGPT citing the site.
What a site owner can do

Do not judge OpenAI's access to your site by OAI-SearchBot page fetches alone: if both bots are allowed, OpenAI may use one crawl for both. Count GPTBot and OAI-SearchBot together, and allow about 24 hours for robots.txt changes to apply.

C2. A name is not an identity

A User-Agent is only a self-declaration; a bot's identity is its IP address in the vendor's published feed, reverse DNS, or a signature (Web Bot Auth). A site that lets bots in by name lets in everyone who borrows the name.

Evidence for the principleExternal source(partly: our lab observation)

Our lab observationRecorded 2026-10-22, re-check by 2027-01-22

18 of 21 sites we scanned (2026-10-03 to 2026-10-21) answered a request that claimed to be an AI fetcher, such as ChatGPT-User or Claude-User, sent from a non-AI IP address, exactly as they answered our own scanner.

Basis
Our scanner's checks of 21 sites, 90 scans with AI fetcher user agents, 2026-10-03 to 2026-10-21; a follow-up on 61 .jp sites, 2026-11-02.
Does not prove
It does not show how real fetchers from verified IPs are treated; a CDN may admit real IPs and still block fake names. One request per user agent, home page only. A 200 for a fake name is not harm in itself: public content may need no protection.
Sources
our own data, no outside link
What a site owner can do

Fetch your home page with the ChatGPT-User and Claude-User user agents and compare with a normal request. If you need to tell real bots from fakes, check the IP against the AI company's published list, not the name.

Our lab observationRecorded 2026-10-22, re-check by 2027-01-22

On our sites many requests hunting for secret files (.env, .git) use AI bot names: on migurei.ai, 2026-10-24 to 2026-11-07, only 105 of 499 requests (21.0%) under six AI and search bot names could be verified.

Basis
Our own logs on pw.migurei.ai and migurei.ai (October to November 2026) and a large Vietnamese site's logs, checked against vendors' IP lists and signatures.
Does not prove
An IP outside a vendor list is not always fake: lists change, and some logs hold only proxy IPs. No probe obtained a secret: on our two sites, 0 probes got a success response. It says nothing about other sites.
Sources
our own data, no outside link
What a site owner can do

Never trust a user agent alone. Verify AI bots by the vendor's published IP list or Web Bot Auth signature, return 404 for .env, .git and similar paths, and count only verified requests as bot visits.

External source(partly: our lab observation)Recorded 2026-11-04, re-check by 2026-11-18

AI vendors change their published bot IP lists without notice: between our 2026-10-21 copy and 2026-11-04, 3 of 16 lists changed (PerplexityBot 8 to 13 ranges, Perplexity-User 4 to 10, ChatGPT-User 234 to 235), so checks using our old copy ran on outdated lists.

Basis
Perplexity's and OpenAI's published bot IP lists, downloaded 2026-11-04 and compared with our 2026-10-21 copy; our own server logs.
Does not prove
We found no real request judged wrongly because of the old lists: every PerplexityBot IP we logged fits both old and new ranges. Requests carrying a bot name but no recorded IP could not be rechecked, and may be fake or real.
Sources
What a site owner can do

If you verify AI bots by IP, re-download each vendor's IP list regularly (ours went stale within two weeks) rather than bundling a fixed copy, and keep identifying bots by IP range, not by user-agent name alone.

External sourceRecorded 2026-11-05, re-check by 2026-12-05

On 2026-11-04 Cloudflare made "Personal Assistants" (AI agents acting for a person) a separate group in Bot Analysis, with request counts per agent, and lets site owners choose "Allow (do not block)", "Block on pages with ads" or "Block".

Basis
Cloudflare's blog post of 2026-11-04, read 2026-11-05, plus our own server logs on migurei.ai and pw.migurei.ai, 2026-10-28 to 2026-11-05.
Does not prove
Which plans get it, the default setting, and which bots Cloudflare puts in the group. In our logs only 6 requests over 9 days came from these agents' published IP ranges; most others using their names were fakes probing for secret files, so no effect is visible yet.
Sources
What a site owner can do

Open Bot Analysis in your Cloudflare dashboard, see which personal agents reach you, and choose the setting on purpose. Do not judge agents by name alone: check requests against each vendor's published IP list.

Our lab observation(partly: external source)Recorded 2026-11-06, re-check by 2026-12-06

On migurei.ai, in the 14 days to 2026-11-06, 100 requests used the DuckAssistBot name: the 20 signed with Web Bot Auth all had valid signatures and fetched normal pages, while 79 unsigned ones probed for secret files such as .env.

Basis
Our own server logs on migurei.ai, 2026-10-23 to 2026-11-06, signatures checked against the published key directory; DuckDuckGo's DuckAssistBot help page.
Does not prove
A valid signature proves the signer holds a key listed at assistbot.duckduckgo.com, not that DuckDuckGo owns it; we have not matched IPs with DuckDuckGo's published list. Whether DuckAssist has users in Japan or sends visitors is unknown.
What a site owner can do

Do not trust a bot by its name. Verify DuckAssistBot by its Web Bot Auth signature or DuckDuckGo's published IP list, and block requests for paths like /.env, which a real crawler never needs.

External sourceRecorded 2026-11-06, re-check by 2026-12-06

On 2026-11-05 Cloudflare launched a Trusted Agent Delegation coalition and named Web Bot Auth (signed requests) as the way to recognise personal AI agents; site owners can already see personal-agent traffic and set allow or block rules per agent and per site section.

Basis
Cloudflare's blog post of 2026-11-05, read 2026-11-06 through a web reading tool's summary.
Does not prove
We read a tool's summary, not every sentence of the original. It does not show that OpenAI, Anthropic, Google or Perplexity will sign their requests: in 14 days of our logs, none of their requests were signed.
Sources
What a site owner can do

If you use Cloudflare, look at the personal-assistants view in Security Analytics and set per-agent policies only for the site sections you care about. Prefer signed identity over user-agent names when deciding.

Also relevant here

C3. Giving and getting are out of balance

AI bots fetch hundreds to thousands of pages for each human visit the app sends back; apart from Google, opening to AI bots today gives more than it gets.

Evidence for the principleExternal source(partly: our lab observation)

External source(partly: our lab observation)Recorded 2026-10-22, re-check by 2026-12-22

AI bots fetch hundreds to thousands of pages per visitor their app sends back: Cloudflare Radar for 2026-10-16 to 2026-10-23 shows Perplexity 3.3K:1, OpenAI 590:1, Anthropic 495:1, against Google 4.5:1.

Basis
Cloudflare Radar's global crawl-to-refer chart, 2026-10-16 to 2026-10-23; plus our own logs on pw.migurei.ai and migurei.ai and a large Vietnamese site's logs.
Does not prove
Ratios are inflated when apps drop the referrer; on our sites 64–88% of human visits had no referrer, so our ratios are upper bounds. Training bots send no visitors by design. No Japan slice. No ratio is 'fair', and this does not say which bot to block.
What a site owner can do

Compute your own ratio per vendor over 30 days: verified AI bot page fetches versus human visits with an AI referrer or utm_source. Read it beside Cloudflare Radar, and remember that blocking lowers the ratio but also your AI visibility.

Also underD5

Also relevant here

C4. Files that invite AI are also a map

robots.txt, the sitemap and llms.txt written for AI bots can be read by anyone; every path written there is a public path.

Evidence for the principleExternal source

Our lab observationRecorded 2026-11-05, re-check by 2026-12-05

On 2026-11-04, two of our own app hosts that the llms.txt of migurei.ai pointed AI to had no usable robots.txt: m-trans.migurei.ai returned its app's HTML page with status 200 for robots.txt and llms.txt, and m-rag.migurei.ai redirected robots.txt to a login page on hub.migurei.ai.

Basis
One plain fetch of each URL by our scanner on 2026-11-04, compared with RFC 9309 (the robots.txt standard).
Does not prove
We saw no bot read this HTML as robots.txt and do not know what any bot made of it. One fetch per URL. We removed the m-trans.migurei.ai line from llms.txt on 2026-11-05; the host itself is unchanged.
Sources
What a site owner can do

Fetch /robots.txt and /llms.txt on every host you point AI to. Each should return plain text or a real 404, not an HTML app page with status 200 or a redirect to a login page. Do not list hosts in llms.txt that fail this check.

Also relevant here

C5. An agent that can act needs limits

When an agent calls tools or buys on a person's behalf, the question moves from "can it read" to "what may it do": turn tools off on pages that do not need them, and mark actions that cannot be undone.

Evidence for the principleExternal source

External sourceRecorded 2026-10-22, re-check by 2026-12-22

The WebMCP draft (dated 2026-10-20) lets a site switch off AI agents calling tools on its pages with the header Permissions-Policy: tools=(), and mark irreversible actions with consequentialHint so the agent asks the user first.

Basis
The WebMCP Community Group Draft Report dated 2026-10-20 (not a W3C standard), read 2026-10-22.
Does not prove
No site in our sample has turned on WebMCP yet, and the draft can still change. The draft puts defence against prompt injection mainly on the agent, not on the site.
What a site owner can do

If your pages offer no WebMCP tools, consider sending Permissions-Policy: tools=(). If you add tools, mark payments, bookings and other irreversible actions with consequentialHint.

C6. One main door

The same page served through several doors (http://, extra ports, extra hosts) is several copies that others can read, alter on the way, or cite instead of the main one. Open to AI at the declared door (canonical, https, port 443) and close the others with redirects or HSTS.

Evidence for the principleExternal source(partly: our lab observation)

Our lab observation(partly: external source)Recorded 2026-11-04, re-check by 2026-12-04

On 2026-11-04, pw.migurei.ai and migurei.ai served full pages over plain http:// without redirecting to https:// and sent no HSTS header. migurei.ai also answered on 11 extra Cloudflare ports. Only https on port 443 is the declared (canonical) address.

Basis
Direct requests on 2026-11-04, our server logs (30 days; migurei.ai since 2026-10-21), Cloudflare's network ports documentation and Mozilla HTTP Observatory.
Does not prove
That anyone altered content in transit (no sign of it), or that Google or AI apps chose the http version as the main one (canonical tags point to https). Requests on the extra ports were mostly scanners, and we saw no AI bots there.
What a site owner can do

Request your pages over http:// and on ports such as 8080 and 8443. Redirect http to https with a 301, add a Strict-Transport-Security header, and block non-standard ports with a WAF rule if your plan allows it.

External sourceRecorded 2026-11-04, re-check by 2026-12-04

On 2026-11-04, Mozilla HTTP Observatory graded pw.migurei.ai F (5/100) and migurei.ai F (0/100): both lack an HTTPS redirect, HSTS, X-Frame-Options, nosniff and a Content Security Policy, and migurei.ai also sets a cookie without the Secure flag.

Basis
Mozilla HTTP Observatory API scans on 2026-11-04 and Mozilla's published scoring table.
Does not prove
Observatory grades browser security, not what AI bots read; none of these headers changes what AI bots can fetch. No data shows Japanese buyers check a vendor's headers. The score after fixes is known only after a new scan.
What a site owner can do

Scan your site with Mozilla HTTP Observatory and add an HTTPS redirect, HSTS, X-Frame-Options, X-Content-Type-Options: nosniff and a Content Security Policy; mark cookies Secure. Do not expect this to change AI visibility.

C7. Say who is speaking

A mention of you that you wrote yourself, commissioned or paid for must show that to readers (and to the apps that copy it); naming the author closes the door on covert promotion that apps cannot tell apart from a real third party.

Evidence for the principleExternal source(partly: our lab observation)

External sourceRecorded 2026-11-05, re-check by 2027-05-05

In Japan, since 2023-10-01, advertising that consumers cannot recognise as advertising (stealth marketing) violates the Act against Unjustifiable Premiums and Misleading Representations; posts a company writes, or has others write while deciding the content, count as the company's own and must be labelled clearly (広告, PR).

Basis
The Consumer Affairs Agency's stealth marketing page, read 2026-11-05; the 'decides the content' reading comes from summaries of its operating standards.
Does not prove
This is not legal advice. We have not read the original operating standards. It does not settle whether the rule covers English-language posts on overseas platforms, or whether a given existing post breaks it.
Sources
What a site owner can do

When you write about your own product on third-party platforms (note, guest articles, listicles), or pay others and shape what they write, label it clearly as 広告 or PR and state that the author is the operator; ask a lawyer when unsure.

Our lab observationRecorded 2026-11-05, re-check by 2026-11-26

In one ChatGPT answer (2026-11-03), a tool marked as a paid (PR) entry in a third-party comparison article was listed first, with that article cited, and the answer carried no PR or ad label; in 18 later answers (2026-11-06 to 2026-11-08) it did not recur.

Basis
Test questions we asked ChatGPT and Gemini, 2026-11-03 to 2026-11-08, and a reading of the cited comparison article.
Does not prove
That ChatGPT named the tool because of the paid entry: it may appear in other articles or in the model's memory. One answer. That the PR label was on that entry at the time of the answer (we read the page 15 to 19 hours later).
Sources
our own data, no outside link
What a site owner can do

If you buy a listing in a comparison article, check how AI answers present it; do not count on the PR label being kept, and keep your own disclosure clear.