How Japanese sites treat AI bots (robots.txt and llms.txt, tracked weekly)
Latest: measured 26 October 2026 (edition 1) / updated weekly
What do popular Japanese .jp sites tell AI bots such as those of ChatGPT, Claude and Gemini in their robots.txt? Every week Parallel World fetches robots.txt and llms.txt from the same 1,000 top .jp domains of the Tranco ranking and counts the results. We name no company and no domain; we publish counts and shares only. The aggregate data (CSV) is free to use under CC BY 4.0.
- Measured: 26 October 2026, 08:38 to 10:00 UTC (17:38 to 19:00 Japan time)
- Sample: the first 1,000 .jp registrable domains, in rank order, of Tranco list W3ZV9 (25 October 2026). No site was excluded (0). Our own sites (migurei.ai, kiddihub.com) are not .jp domains, so they were never in the sample
- Bots evaluated: the 39 bots in our bot registry, version bots-2026-10-25
- Next: the same 1,000 domains are due to be measured again in the week of 2 November 2026
Key findings
- Of the 719 sites whose robots.txt gave a verdict, 74 sites (10.3%) do not allow GPTBot to crawl the top page (/). For Googlebot it is 7 sites (1.0%).
- 111 sites (15.4%) disallow at least one AI training bot. 105 sites (14.6%) disallow a training bot while still allowing Googlebot.
- Of the 788 sites that answered from their own domain, 61 sites (7.7%) returned 403 to our signed request for robots.txt.
- 29 sites (2.9%) of the 1,000 have an llms.txt on their own domain. Of the 777 sites checked, 69 sites (8.9%) answer /llms.txt with 200 and an HTML page or other non-llms.txt content (a soft 404): more than twice as many as have the file.
- Fewer than 5 sites use Content-Signal in robots.txt.
By bot: sites that do not allow the top page (/)
"Sites with a verdict (719)" are sites where the robots.txt content, or the absence of a file, decides each bot's access. "Sites with a robots.txt file (515)" are those among them that returned a robots.txt with actual rules.
| Bot | Operator | Purpose | Of 719 sites with a verdict | Of 515 sites with a robots.txt file |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | 74 (10.3%) | 74 (14.4%) |
| ClaudeBot | Anthropic | Training | 73 (10.2%) | 73 (14.2%) |
| CCBot | Common Crawl | Training | 65 (9.0%) | 65 (12.6%) |
| Google-Extended | Training | 63 (8.8%) | 63 (12.2%) | |
| PerplexityBot | Perplexity | Search index | 49 (6.8%) | 49 (9.5%) |
| OAI-SearchBot | OpenAI | Search index | 39 (5.4%) | 39 (7.6%) |
| Googlebot | Search index | 7 (1.0%) | 7 (1.4%) |
Purposes are those in our bot registry, taken from each operator's own documentation. Google-Extended is not a crawler but a robots.txt name that tells Google whether content may be used for Gemini and similar uses. The numbers for the other 32 bots (Amazonbot, Applebot-Extended, Bytespider, meta-externalagent, ChatGPT-User, Claude-User and more) are in the CSV.
Only 1.0% of sites disallow Googlebot. Meanwhile about 1 site in 7 (14.6%) allows Googlebot and disallows at least one AI training bot: a share of sites treat search and training differently.
Sites that returned 403 for robots.txt
Of the 788 sites that answered from their own domain, 61 sites (7.7%) returned 403 (forbidden) to our signed request instead of a robots.txt. By Server header, 19 were AkamaiGHost and 13 AmazonS3. An AmazonS3 403 can simply mean the file is missing, so not every 403 is an access restriction.
Because no per-bot verdict can be read for these 61 sites, they are not in the 719 denominator above.
llms.txt and Content-Signal
- 29 sites (2.9%) of 1,000 have an llms.txt at /llms.txt on their own domain. Among the 777 sites where we could check it, the share is 3.7%.
- Of the 777 sites checked, 69 sites (8.9%) answer /llms.txt with 200 and an HTML page or other content that is not an llms.txt. This is a soft 404: it looks as if the file exists, so some ways of checking would count these sites as having an llms.txt.
- Fewer than 5 sites use Content-Signal (a robots.txt line stating whether content may be used for search, AI input and AI training). We do not publish the exact number of so small a cell.
Sample and method
From 1,000 domains to the denominators
| Step | Sites |
|---|---|
| Sample (top .jp registrable domains on Tranco) | 1,000 |
| Name resolved (apex or www) | 909 |
| Answered the robots.txt request with any HTTP status | 813 |
| Answered from its own registrable domain (25 redirected to another domain excluded) | 788 |
| robots.txt gave a per-bot verdict | 719 |
| Of which returned a robots.txt file with rules | 515 |
Not answered: 91 names did not resolve, 41 timed out, 48 other network errors, 7 too many redirects. No verdict: 61 returned 403, 8 returned 5xx.
How we fetched
- One GET for robots.txt per site, over https, following up to 3 redirects. On a network failure only, one retry on the www host.
- User agent
migurei-ParallelWeb/0.1 (+https://migurei.ai/ja/scanner; research@migurei.ai). Every request (2,396 in all) carried a Web Bot Auth signature; the public key is at https://pw.migurei.ai/.well-known/http-message-signatures-directory. - Sent from a Japanese IP address.
- /llms.txt was fetched once, only on sites whose robots.txt does not disallow our user agent (7 of 719 sites disallow it).
- Read-only: no form was submitted, no login, no purchase.
How we evaluated
- Under RFC 9309 (Robots Exclusion Protocol), for each bot: may it fetch the path "/"?
- A version in a User-agent line (for example
GPTBot/1.1) is stripped before matching, as Google's open-source robots.txt parser does. - A missing robots.txt (404 and similar) means everything is allowed, as RFC 9309 says. A 200 answer with an HTML page or a file without directives is read as the rules it contains, that is, none.
- Sites whose robots.txt redirected to another registrable domain (25) are left out of the denominators: those rules are not clearly the site's own.
Week by week
We measure the same frozen list of 1,000 domains every week. Because the list does not change, a difference from one week to the next reflects a change on the sites (or in whether we could reach them). The latest week is on top.
| Week | Measured | GPTBot not allowed | At least one training bot disallowed | 403 for robots.txt | llms.txt present |
|---|---|---|---|---|---|
| 2026-W44 | 26 October 2026 | 74 / 719 (10.3%) | 111 / 719 (15.4%) | 61 / 788 (7.7%) | 29 / 1,000 (2.9%) |
There is only the first edition so far. From the second on, one row is added to this table every week.
What these numbers do not prove
- robots.txt is a request, not a barrier. This survey says nothing about which bots actually visit a site, or whether a disallowed bot obeys.
- Sites that returned 403 or 5xx are outside the shares. If all 61 sites that returned 403 allowed GPTBot, the share would be 9.5%; if all disallowed it, 17.3% (the 8 sites with 5xx are left out of this range too). The true share may lie anywhere in that range.
- This is not "the Japanese web". It is 1,000 .jp domains from one day's Tranco list. Japanese sites on .com and other domains are not in it.
- We sent only our own user agent. We do not know how sites answer a request that calls itself GPTBot or any other bot.
- We evaluated the top page (/) only. A site that disallows only some directories is not counted.
- Some of the 91 name-resolution failures may be transient. On a recheck on 26 October 2026, 5 of the 91 resolved.
- We do not judge whether disallowing is good or bad. Which bot may do what is each site's decision.
Frequently asked questions
What share of Japanese sites disallow GPTBot in robots.txt?
In Parallel World's survey of the top 1,000 .jp domains on the Tranco list (26 October 2026), 74 sites (10.3%) of the 719 sites whose robots.txt gave a verdict did not allow GPTBot to crawl the top page (14.4% of the 515 sites with a robots.txt file).
What share of Japanese sites disallow ClaudeBot in robots.txt?
In Parallel World's survey of the top 1,000 .jp domains on the Tranco list (26 October 2026), 73 sites (10.2%) of the 719 sites whose robots.txt gave a verdict did not allow ClaudeBot to crawl the top page.
What share of Japanese sites disallow CCBot in robots.txt?
In Parallel World's survey of the top 1,000 .jp domains on the Tranco list (26 October 2026), 65 sites (9.0%) of the 719 sites whose robots.txt gave a verdict did not allow CCBot to crawl the top page.
What share of Japanese sites disallow Google-Extended in robots.txt?
In Parallel World's survey of the top 1,000 .jp domains on the Tranco list (26 October 2026), 63 sites (8.8%) of the 719 sites whose robots.txt gave a verdict did not allow Google-Extended to crawl the top page.
What share of Japanese sites disallow PerplexityBot in robots.txt?
In Parallel World's survey of the top 1,000 .jp domains on the Tranco list (26 October 2026), 49 sites (6.8%) of the 719 sites whose robots.txt gave a verdict did not allow PerplexityBot to crawl the top page.
What share of Japanese sites disallow OAI-SearchBot (ChatGPT search) in robots.txt?
In Parallel World's survey of the top 1,000 .jp domains on the Tranco list (26 October 2026), 39 sites (5.4%) of the 719 sites whose robots.txt gave a verdict did not allow OAI-SearchBot to crawl the top page.
What share of Japanese sites disallow Googlebot in robots.txt?
In Parallel World's survey of the top 1,000 .jp domains on the Tranco list (26 October 2026), 7 sites (1.0%) of the 719 sites whose robots.txt gave a verdict did not allow Googlebot to crawl the top page.
How many Japanese sites disallow AI training bots in robots.txt?
In Parallel World's survey of the top 1,000 .jp domains (26 October 2026), 111 sites (15.4%) of the 719 sites with a verdict disallowed at least one AI training bot in robots.txt.
How many Japanese sites allow Googlebot but disallow an AI training bot in robots.txt?
In Parallel World's survey of the top 1,000 .jp domains (26 October 2026), 105 sites (14.6%) of the 719 sites with a verdict allowed Googlebot while disallowing at least one AI training bot.
How many Japanese sites answer 403 to our signed request for robots.txt?
In Parallel World's survey of the top 1,000 .jp domains (26 October 2026), 61 sites (7.7%) of the 788 sites that answered from their own domain returned 403 to a signed bot request for robots.txt sent from a Japanese IP.
How many Japanese sites have an llms.txt?
In Parallel World's survey of the top 1,000 .jp domains on the Tranco list (26 October 2026), 29 sites (2.9%) had an llms.txt on their own domain.
How many Japanese sites answer /llms.txt with an HTML page?
In Parallel World's survey of the top 1,000 .jp domains (26 October 2026), 69 sites (8.9%) of the 777 sites checked answered /llms.txt with 200 and an HTML page or other non-llms.txt content (a soft 404), more than twice the 29 sites that have the file.
How many Japanese sites use Content-Signal?
In Parallel World's survey of the top 1,000 .jp domains (26 October 2026), fewer than 5 sites used Content-Signal in robots.txt.
Data and license
- Aggregate data (CSV): 26 October 2026, 2026-W44, all weeks (series.csv)
- GitHub: https://github.com/theta-claude-dot/jp-ai-bot-census
- Aggregates only. We never publish data on an individual site or domain. Cells under 5 are shown as "<5". "Minus our own and excluded sites" in the CSV column denominator_definition is the selection rule; in this edition 0 sites were excluded.
- License: CC BY 4.0
- When you cite or reuse the data, credit "Parallel World (pw.migurei.ai)". For example: Parallel World (pw.migurei.ai), "How Japanese sites treat AI bots", measured 26 October 2026.
Sources
- Tranco list W3ZV9 (25 October 2026): https://tranco-list.eu/list/W3ZV9 ; Victor Le Pochat et al., "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation" (NDSS 2019)
- Bot list: the Parallel World AI bot registry (bots-2026-10-25, CC BY 4.0); every bot links to its operator's own page: https://github.com/theta-claude-dot/ai-bot-registry
- RFC 9309 Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309.html
- Google's robots.txt parser: https://github.com/google/robotstxt
- Web Bot Auth (IETF working group): https://datatracker.ietf.org/wg/webbotauth/about/
- The llms.txt proposal: https://llmstxt.org/
- Content Signals: https://contentsignals.org/
About this survey
Parallel World (migurei) measures how AI finds, reads and cites websites. We ran this survey with our own scanner and counted the results ourselves. No independent party has checked it.
To check your own robots.txt and llms.txt, use the free scan on the top page (no sign-up). About the scanner used for this survey: migurei.ai/en/scanner.