How Japanese sites treat AI bots (robots.txt and llms.txt, tracked weekly)

"Sites with a verdict (719)" are sites where the robots.txt content, or the absence of a file, decides each bot's access. "Sites with a robots.txt file (518)" are those among them that returned a robots.txt with actual rules.

Purposes are those in our bot registry, taken from each operator's own documentation. Google-Extended is not a crawler but a robots.txt name that tells Google whether content may be used for Gemini and similar uses. The numbers for the other 33 bots (Amazonbot, Applebot-Extended, Bytespider, meta-externalagent, ChatGPT-User, Claude-User and more) are in the CSV.

Only 0.8% of sites disallow Googlebot. Meanwhile about 1 site in 7 (15.0%) allows Googlebot and disallows at least one AI training bot: a share of sites treat search and training differently.

Record of 2 November 2026 (lab batch #337, and a recount of the robots.txt data of week 45)

Short answer: in ChatGPT search, this experiment did not see sites that disallow GPTBot named less often than sites that do not. That does not show that disallowing GPTBot is harmless. Read it together with "What this does not show" below.

Lab result (ChatGPT search only)

• Method: from the first 1,000 .jp domains in Tranco rank order (ranks 459 to 57,386 on the full Tranco list), we paired sites that disallow GPTBot in robots.txt (and allow Googlebot) with sites that do not, matched by sector label and close rank. We asked ChatGPT (free account, Japanese, temporary chat, Japanese IP) a question about the sector without naming the site, twice per site, and counted whether the answer named the site.
• Every number here is from 49 pairs and 195 valid Runs (28 October 2026 to 31 October 2026). Of those, the 193 Runs where both sides of a pair were present (96 disallowing, 97 open) are compared.
• The rule for "no gap" was fixed before the experiment: the mean of the per-pair difference (disallowing minus open) is above −0.05.

The -0.031 for links comes from two pairs on the disallowing side where the company's current main site sits on a registrable domain other than the one in the list (the pre-set rule does not count another domain), so the gap comes from those two pairs alone.

By the way of disallowing (description only, not tested), the Runs that named the site:

What this does not show

• "Refuted" means this batch saw no gap. It does not prove that disallowing GPTBot has no effect.
• An observation outside the pre-registration (a hint, not a test). For sites that disallow both GPTBot and OAI-SearchBot, the site appeared as a citation chip (the small source card on an answer) in 7 / 42 Runs, against 22 / 43 for the paired open sites. The site was named about as often (38 / 42 and 39 / 43) and was linked. We noticed it after looking at the data, so we do not use it as a conclusion until a new, pre-registered batch checks it.
• Ceiling effect. The site was named in 90/96 and 91/97 Runs, near the top. In 36 of the 49 pairs both sides were named in both Runs. The questions were written for each site's sector and point almost directly at it. Small sites, more general questions, or questions where the site is not the obvious answer may behave differently.
• Sites chose their own setting. Disallowing was each site's decision, not a random assignment. This does not measure the effect of disallowing; it compares sites with the same sector label and close rank. A reader who re-read the pairs found 20 or more pairs that were clearly different in sector. Brand strength was not matched.
• Question wording. About 70 of the 100 questions contain 「日本の」 ("Japan's") or 「日本語で」 ("in Japanese"), which people in Japan rarely say, so the wording differs from how people really ask.
• Grading. A model graded the answers. A Japanese-reading model re-read 64 of the 195 Runs; no human has re-read them yet. The code_sha of the code that ran the Runs is unknown (#374).
• Conditions. One free ChatGPT account, one machine, one Japanese exit, ChatGPT search only. Other AI search was not measured. robots.txt is what a site declares; it does not show what OpenAI's bots actually do.
• Some Runs are not graded. The numbers above are from 49 pairs graded on 31 October 2026. 5 replacement Runs for the 5 Runs with an empty answer (2 November 2026, 11:41 to 12:20 UTC) are not included.

The robots.txt side: GPTBot and OAI-SearchBot together

We recounted the stored data of week 45, measured 2 November 2026. No new request was sent. Split by GPTBot and OAI-SearchBot together, the 719 sites whose robots.txt gave a verdict are:

All 38 sites that disallow OAI-SearchBot also disallow GPTBot. The combination that disallows OpenAI's training bot (GPTBot) but allows its search bot (OAI-SearchBot) is 38 / 719 (5.3%).

• The 63 sites that returned 403 answered our signed request; that is not how they answer GPTBot or OAI-SearchBot. Their robots.txt could not be read, so they are not in the 719 above and we do not know whether they allow or disallow OpenAI's bots. The Server-header breakdown is in "Sites that returned 403 for robots.txt" above.
• We counted only the stored robots.txt text and sent no request to any disallowed path.
• Small numbers read "<5" (a cell under 5 sites is not published as a number).

For site owners: if appearing in ChatGPT search matters to you, check OAI-SearchBot as well as GPTBot. In the lab experiment above, sites that disallow both bots were still named in 38 / 42 Runs but shown as a citation chip less often (7 / 42 vs 22 / 43; outside the pre-registration, a hint only). OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links (https://developers.openai.com/api/docs/bots). This experiment did not measure that directly.

From 1,000 domains to the denominators

Not answered: 87 names did not resolve, 42 timed out, 49 other network errors, 7 too many redirects. No verdict: 63 returned 403, 10 returned 5xx.

How we fetched

• One GET for robots.txt per site, over https, following up to 3 redirects. On a network failure only, one retry on the www host.
• User agent migurei-ParallelWeb/0.1 (+https://migurei.ai/ja/scanner; research@migurei.ai). The 725 sites whose requests were recorded (2,277 requests) were all sent with a Web Bot Auth signature (none unsigned). For the other 275 sites the measurement was restarted midway and neither the number of requests nor whether they were signed was recorded. The public key is at https://pw.migurei.ai/.well-known/http-message-signatures-directory.
• Sent from a Japanese IP address.
• /llms.txt was fetched once, only on sites whose robots.txt does not disallow our user agent (6 of 719 sites disallow it).
• /.well-known/api-catalog (RFC 9727) was also fetched once, only on sites whose robots.txt does not disallow our user agent for that path. No redirect was followed and there was no retry.
• Read-only: no form was submitted, no login, no purchase.

How we evaluated

• Under RFC 9309 (Robots Exclusion Protocol), for each bot: may it fetch the path "/"?
• A version in a User-agent line (for example GPTBot/1.1) is stripped before matching, as Google's open-source robots.txt parser does.
• A missing robots.txt (404 and similar) means everything is allowed, as RFC 9309 says. A 200 answer with an HTML page or a file without directives is read as the rules it contains, that is, none.
• Sites whose robots.txt redirected to another registrable domain (23) are left out of the denominators: those rules are not clearly the site's own.

• robots.txt is a request, not a barrier. This survey says nothing about which bots actually visit a site, or whether a disallowed bot obeys.
• Sites that returned 403 or 5xx are outside the shares. If all 63 sites that returned 403 allowed GPTBot, the share would be 9.7%; if all disallowed it, 17.8% (the 10 sites with 5xx are left out of this range too). The true share may lie anywhere in that range.
• This is not "the Japanese web". It is 1,000 .jp domains from one day's Tranco list. Japanese sites on .com and other domains are not in it.
• We sent only our own user agent. We do not know how sites answer a request that calls itself GPTBot or any other bot.
• We evaluated the top page (/) only. A site that disallows only some directories is not counted.
• Some of the 87 name-resolution failures may be transient.
• We do not judge whether disallowing is good or bad. Which bot may do what is each site's decision.

• Aggregate data (CSV): 2 November 2026, 2026-W45, all weeks (series.csv)
• Aggregates only. We never publish data on an individual site or domain. Cells under 5 are shown as "<5". "Minus our own and excluded sites" in the CSV column denominator_definition is the selection rule; in this edition 0 sites were excluded.
• License: CC BY 4.0
• When you cite or reuse the data, credit "Parallel World (pw.migurei.ai)". For example: Parallel World (pw.migurei.ai), "How Japanese sites treat AI bots", measured 2 November 2026.

Parallel World (migurei) measures how AI finds, reads and cites websites. We ran this survey with our own scanner and counted the results ourselves. No independent party has checked it.

To check your own robots.txt and llms.txt, use the free scan on the top page (no sign-up). About the scanner used for this survey: migurei.ai/en/scanner.