For three weeks in a row, top .jp sites answering /llms.txt with 200 and an HTML page outnumbered those with a real file about two to one
The answer first: in the weekly scan of 26 October 2026, of 777 top .jp sites we asked for /llms.txt, 29 (3.7%) had the file, and 69 (8.9%) answered "200 OK" with something that was not the file. All 69 answered with text/html. Counted by status code alone, 98 sites (12.6%) would look as if they had it: 3.4 times the real number.
Measured on 26 October, 2 November and 9 November 2026 (the scan runs every Monday). The same Tranco list (a public ranking of the most visited domains) each time; each site was asked once for /llms.txt, under our scanner's identifying User-Agent, from a Japanese IP. No site is named.
The numbers (three weeks)
| Scan day | Sites checked | Real file | 200 but not the file | Times the real count |
|---|---|---|---|---|
| 2026-10-26 | 777 | 29 (3.7%) | 69 (8.9%) | 2.4 |
| 2026-11-02 | 782 | 30 (3.8%) | 66 (8.4%) | 2.2 |
| 2026-11-09 | 775 | 30 (3.9%) | 63 (8.1%) | 2.1 |
- Breakdown of the 777 sites of 2026-10-26: a real file 29, 200 but not the file 69 (all
text/html), everything else 679 (most of them 404). - We call an answer "not the file" when its
Content-Typeis not plain text or Markdown, its body is empty, or it opens with an HTML tag. A server that answers 200 for an address that has nothing at it is giving what is called a soft 404. - "Sites checked" leaves out sites whose robots.txt was served by another domain and sites whose /llms.txt request ended on another domain: 775 to 782 sites per week.
- The real count hardly moved (29, 30, 30), and the sites answering with a page outnumbered the real files by about two to one every week.
Basis and limits
- How other surveys count. We do not know whether they count by status code. The gap between our two figures shows that the way of counting can triple the result; it does not show what another survey did.
- Why the sites answer 200. We have not looked at their servers. A catch-all page is the likely cause, not a finding.
- Whether any AI app reads llms.txt, or treats an HTML page as one, is a separate question. llms.txt is a proposal, not a standard.
- Our filter does not check syntax. A file served as plain text with wrong contents counts as present; a real file served with an odd
Content-Typecounts as "not the file". - Top .jp sites on one list only, about 780 of the first 1,000, as seen from our scanner. Sites whose answer ended on another domain are left out. Other sites may differ.
- Sources:
knowledge/claims/jp-sites-answer-llms-txt-with-html-soft-404.md(week of 26 October 2026: 29, 69 and 98 of 777, list W3ZV9 of 2026-10-25, matched against the published CSV row and a recount from the raw record). Weeks 2 and 3 are the same rows of each week's published CSV. Level: observed in our lab. - Next: the scan continues every Monday; the next numbers are those of 16 November 2026.