日本語EN
Parallel World
Scan your site free

A. Bots and AI apps

How a page gets from your server into an AI answer: which bot fetches it, which index the app reads, how a bot learns a URL exists, and what makes a page readable and trusted.

Built from our knowledge base at every deploy. 31 principles, 200 claims. Newest note: 2026-11-12.

日本語Handbook index

Each principle is followed by the claims it rests on. Evidence levels are explained on the handbook index.

Show claims by evidence level

A1. The chain of existence

An AI app names a page only after the page has passed the whole chain: found, fetched, indexed, retrieved, trusted, cited. Each step blocks the next, so being fetched comes before being cited.

Evidence for the principleExternal source

HypothesisRecorded 2026-10-10, re-check by 2027-01-10

When answering discovery questions, AI apps probably draw on a search index built earlier rather than visiting your site at answer time; this is still a hypothesis.

Basis
One test of ours on a large Vietnamese site, 2026-10-02: apps named it with no request reaching its server; six questions per app.
Does not prove
One site, in Vietnamese, one round, six questions per app. How far ChatGPT relies on Bing is unconfirmed by OpenAI. A follow-up test on two of our own tool pages has no result yet.
Sources
our own data, no outside link
What a site owner can do

Nothing yet: this is a hypothesis.

External sourceRecorded 2026-10-10, re-check by 2027-01-10

OpenAI says sites that block OAI-SearchBot will not be shown in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot may reduce visibility in Claude's search results. Neither offers a way to submit new URLs.

Basis
OpenAI's bot documentation and Anthropic's support article on its crawlers, read 2026-10-10.
Does not prove
Allowing these bots is necessary, not enough, and says nothing about speed. When we added explicit Allow groups for each bot to our robots.txt, they fetched 0 pages in the next 72 to 96 hours (2026-10-21 to 2026-10-26). One site only.
Sources
What a site owner can do

Do not block OAI-SearchBot or Claude-SearchBot in robots.txt; OpenAI applies changes after about 24 hours. Do not expect explicit Allow lines to bring them; get links to your pages from pages they already fetch.

Also underA2A3

External sourceRecorded 2026-10-12, re-check by 2027-01-12

Google finds new URLs only through links from known pages or submitted sitemaps; our new subdomain pw.migurei.ai got no Googlebot visit for 10 days, then Googlebot fetched 12 of 13 sitemap URLs the same day we submitted the sitemap in Search Console (2026-10-12).

Basis
Google Search Central and Search Console help pages, read 2026-10-11, plus our own server logs on pw.migurei.ai, 2026-10-01 to 2026-10-13.
Does not prove
Google promises no timing and no crawl; 10 days may just be waiting. A link from migurei.ai was not tested, so links are not shown useless. Being fetched is not being indexed: URL Inspection still showed a page as not indexed the next day.
What a site owner can do

For a new subdomain, verify it in Google Search Console (a domain property covers all subdomains), submit its sitemap there, and add a normal link from a page Google already knows.

External sourceRecorded 2026-10-15, re-check by 2027-01-15

Google says a page can be a source link in AI Overviews and AI Mode only if it is indexed and eligible to show with a snippet, with no other technical requirements; noindex, nosnippet, max-snippet and data-nosnippet limit this (read 2026-10-15).

Basis
Google Search Central page on AI features and your website, updated 2025-12-10, read 2026-10-15.
Does not prove
These are necessary, not sufficient, conditions; Google says indexing and serving are not guaranteed. The page does not say the Gemini app outside Search uses the same rules.
Sources
What a site owner can do

Remove noindex and nosnippet from pages you want cited in Google's AI answers, unless you set them on purpose, and confirm in Search Console that those pages are indexed.

A2. Each app has its own index

ChatGPT reads the index of OAI-SearchBot (and possibly Bing), Claude uses Brave, Gemini and AI Overviews use Google, Copilot uses Bing, Perplexity uses PerplexityBot. Being in one index does not put you in another.

Evidence for the principleExternal source

External sourceRecorded 2026-10-13, re-check by 2026-11-21

Anthropic lists Brave Search, along with TurboPuffer, as a subprocessor for Web Search in all its products, but does not say how much of Claude's answers comes from Brave's index.

Basis
Anthropic's subprocessor list, read 2026-10-13. Its help pages and API documentation name no provider. Secondary reports and our small Claude tests add context.
Does not prove
The list gives no shares and does not say which provider serves which question. TurboPuffer is a vector database, and whether it holds Claude's index is speculation. Our Claude tests (0 of 9, twice, with citations often not stored) neither confirm nor rule out Brave.
Sources
What a site owner can do

Search for your site on Brave Search to check that your key pages are indexed. If they are not, request a recrawl through Brave's form, because Claude's web search may rely on Brave.

Also underD3

External sourceRecorded 2026-10-14, re-check by 2027-01-14

According to Ahrefs (Japan, 2026-09-29 to 2026-09-30), the ten domains AI apps cite most in Japan are Japanese Wikipedia, Google, note, Yahoo! Chiebukuro, YouTube, Ameba blog, PR TIMES, my-best, Weblio and Kotobank; PR TIMES ranks 2nd for ChatGPT, and note 1st for Gemini.

Basis
An Ahrefs Brand Radar ranking published 2026-09-30 (read 2026-10-14), also summarised by a GEO tool vendor's blog.
Does not prove
Data from a vendor that sells the measuring tool; general questions across all industries, not B2B tool questions; ranks, not shares. It does not show that posting on note or PR TIMES makes a new brand get named.
Sources
What a site owner can do

If you publish press releases or articles, consider PR TIMES (strong for ChatGPT) and note (strong for Gemini, Perplexity, Copilot, AI Mode). Do not post your own company as a user on Q&A sites like Chiebukuro; that is impersonation.

Also underA6B4B7

Our lab observationRecorded 2026-10-29, re-check by 2026-11-29

Asked who studies how Japanese sites handle AI bots, Gemini named our survey page in 3 of 3 answers on 2026-10-28, about 38 hours after Google indexed it; on 2026-11-05 it named us 0 of 3 times, though the page stayed indexed.

Basis
Test questions we asked Gemini, ChatGPT, Claude and DeepSeek on 2026-10-28, 2026-10-29 and 2026-11-05, graded blind, plus Search Console index checks.
Does not prove
We have no answers from before the page was indexed, so Google indexing is not shown to be the cause. Only 3 answers per question per day, one account, one machine. Why Gemini stopped naming us is unknown. Naming migurei.ai is not naming Parallel World.
Sources
our own data, no outside link
What a site owner can do

Get your key data page indexed by Google and confirm it in URL Inspection, but do not treat indexing as a lasting place in Gemini's answers; ask Gemini the same question on several different days.

Also underA6

Our lab observationRecorded 2026-10-30, re-check by 2026-11-30

On 2026-10-29, after reading our GitHub repository, Claude named our weekly survey of Japanese sites as the most direct source and linked our page on pw.migurei.ai. This happened in 1 of 12 runs, and that repository has been offline since 2026-11-03.

Basis
Blind-graded test questions to Claude (Haiku 5.5 Low, normal chat, Japanese IP) on 2026-10-28 and 2026-10-29, checked against our server logs.
Does not prove
It happened once in 12 runs. In normal chat with account history, memory effects cannot be ruled out. The search engine used is unknown. The answer contained errors and failed our strict grade. Being named is not the same as a buyer visiting.
Sources
What a site owner can do

Check which page Claude actually reads about you. It is often a page on a third-party host such as GitHub, not your own site. Keep that page accurate and complete.

Also underA6

Our lab observationRecorded 2026-10-30, re-check by 2026-11-30

On 2026-10-29 Brave ranked a GitHub repository of ours first for our page titles while our own site was not on page 1; by 2026-11-03, after that GitHub account was suspended, our site ranked first for the same kind of query.

Basis
Single readings of Brave search results from our server, 2026-10-29 to 2026-11-03, plus Claude's search results in our test questions.
Does not prove
One reading per query, from a server IP outside Japan. The queries were our own page titles, not words buyers type. Claude may use search providers other than Brave. Whether Claude then opens our page was not tested.
Sources
our own data, no outside link
What a site owner can do

Search Brave for your page titles and key questions. If a copy on another host outranks your site, make sure your own page carries the exact words people ask. Do not depend on one third-party platform account to hold your ranking.

Our lab observationRecorded 2026-10-30, re-check by 2026-11-30

For two Japanese statistics questions, Claude (Haiku 5.5 Low) searched the web in English first in 9 of 12 runs (2026-10-28 and 2026-10-29). Our repository appeared only in a run whose first query contained Japanese words.

Basis
Search queries and results Claude displayed in 12 test runs in normal chat on 2026-10-28 and 2026-10-29. One later round pointed the same way.
Does not prove
One model, two questions, two days. Account history may shape the queries. We do not know whether English comes from the English technical terms in the question or from Claude always translating. The search engine behind it was not recorded.
Sources
our own data, no outside link
What a site owner can do

On Japanese pages about technical topics, add one English line in the title, description or first paragraph, using the words Claude searches for: bot names, 'block rate', 'Japan', 'survey'.

Also underB2

Our lab observationRecorded 2026-11-03, re-check by 2026-11-17

Free ChatGPT linked our statistics page on pw.migurei.ai in 3 of 6 runs on 2026-11-02 and again on 2026-11-03. No OpenAI or Bing bot had ever fetched that page, and the source chip ChatGPT showed for us was always our note article.

Basis
Blind-graded test questions to free ChatGPT (temporary chat, Japanese IP), 2026-11-02 to 2026-11-04, checked against our own server logs.
Does not prove
Why results rose from 0 of 12 in earlier runs: several things changed at once. That ChatGPT copied the URL from our note article is an inference. We ran three runs per question on one account, without ruling out account memory. This says nothing about paid ChatGPT.
Sources
our own data, no outside link
What a site owner can do

Check whether AI answers reach your pages through a third-party platform article such as note. Keep that article accurate: answer the exact question there and link to your page.

Also underB7

External sourceRecorded 2026-11-03, re-check by 2027-01-03

For the same questions, ChatGPT's default model (GPT-5.3 Instant) sent 8% of its citations to brand websites, while the paid GPT-5.4 Thinking sent 56%. The two models shared only 7% of their sources, and the default model mostly cited articles and third-party reviews.

Basis
A Search Engine Journal article (2026-03-12, read 2026-11-03) summarising a Writesonic study of 50 questions. We did not read the original study.
Does not prove
These numbers come from one GEO tool vendor and cover 50 English shopping questions. They may not hold for Japanese questions, for logged-out use, or for the default model after March 2026. Our free-ChatGPT tests leaned the same way but could not test 8% against 56%.
Sources
What a site owner can do

Most users see the default model, so get your key facts onto credible third-party articles and review pages. Keep your own pages strong as well, because the paid model cites brand sites far more often.

Also relevant here

A3. There must be a way in

Bots find new URLs through links on pages they already fetched, a sitemap sent through the site owner's tools, or a notification channel (IndexNow, a re-fetch form). Each bot has its own door, and some bots have no door a site owner can use.

Evidence for the principleExternal source

HypothesisRecorded 2026-10-13, re-check by 2026-11-13

Hypothesis: on a site it already crawls, OAI-SearchBot finds most new pages through links on pages it just fetched; on a large site we studied, 92% of its first fetches of new pages came right after fetching a linking page.

Basis
Server logs of a large Vietnamese site, 2026-06-19 to 2026-10-02, with bot IPs checked against OpenAI's list; OpenAI's bot documentation, read 2026-10-13.
Does not prove
One established site, observation only. All links followed stayed on the same host; following links to another host is untested, and our cross-host link tests got no bot visits. 35% of first fetches had no known source. A fetch is not indexing.
Sources
What a site owner can do

Nothing yet: this is a hypothesis.

External sourceRecorded 2026-10-10, re-check by 2026-11-21

IndexNow (a protocol to tell search engines about changed URLs) reaches Bing and other participants but not Google, needs no account, and guarantees no crawl: on our site Bingbot came within seconds after early pings (2026-10-14, 2026-10-15) but ignored four later ones (2026-10-17 to 2026-10-21).

Basis
IndexNow and Bing documentation, read 2026-10-10 to 2026-10-21, plus our own server logs on pw.migurei.ai with verified Bingbot IPs.
Does not prove
The documentation gives no timing. Each of our observations is one deploy, not a controlled measurement; Bing URL submissions sent minutes apart cannot be separated from IndexNow. It does not reach OpenAI, Anthropic, Perplexity, Google or Brave crawlers.
Sources
What a site owner can do

Serve the IndexNow key file at your site root and check it still returns 200 after every deploy (ours returned 404 once). Ping IndexNow when pages change, but keep your sitemap current and confirm in Bing Webmaster Tools whether Bing knows the URL.

External sourceRecorded 2026-10-13, re-check by 2027-01-13

Brave Search has no webmaster tools and does not accept IndexNow; its only channel for site owners is a form to request a re-fetch of one URL, which Brave describes for updated or removed pages.

Basis
Brave's help pages and submission form, and IndexNow's published list of search engines, all read 2026-10-13.
Does not prove
Brave's documents do not say the form works for new URLs, nor how long a re-fetch takes.
Sources
What a site owner can do

Use Brave's re-fetch form on search.brave.com when you update a page; you may also try it for new pages. Do not expect IndexNow notifications to reach Brave.

Our lab observationRecorded 2026-10-14, re-check by 2026-11-09

From 2026-10-01 to 2026-10-29, OpenAI's search crawler OAI-SearchBot visited pw.migurei.ai 58 times and fetched only robots.txt, never a page, while OpenAI's training crawler GPTBot fetched 18 HTML pages on 2026-10-26.

Basis
Our own server logs on pw.migurei.ai, 2026-09-14 to 2026-10-29, bot IPs checked against OpenAI's lists, plus Cloudflare settings and edge statistics.
Does not prove
Regular robots.txt checks may just be permission checks. One site. GPTBot's fetch does not show the pages are in ChatGPT's search index. Blocking at our Cloudflare edge was ruled out by settings and edge data, though custom firewall rules could not be read.
Sources
What a site owner can do

Check your logs for OAI-SearchBot fetching pages, not just robots.txt. Make sure your firewall or CDN does not block it, and give it a page worth fetching linked from a page it already reads.

External sourceRecorded 2026-10-15, re-check by 2027-01-15

Google crawls the URLs in your sitemap exactly as written, and asks you to list full absolute URLs of the version you want shown in results (the canonical); a URL that redirects or returns an error is not that version.

Basis
Google's sitemap documentation (updated 2026-07-08), read 2026-10-15; plus our scan of 20 sitemap URLs each on three Japanese retail sites, 2026-10-14.
Does not prove
Google does not say redirecting URLs in a sitemap are penalised or how much they slow crawling. In our scan, 403 answers to our scanner's user agent do not prove Googlebot also gets 403.
Sources
What a site owner can do

List only final URLs that return 200: https, your canonical host and path, no redirects, no test or deleted pages. Fetch every sitemap URL once and fix any that redirect or fail.

External sourceRecorded 2026-10-16, re-check by 2026-11-16

In a pilot started in July 2026, signals from Cloudflare's network (new URLs, changed pages, broken or redirecting URLs) help OpenAI's search crawler find pages; on about 1% of Cloudflare's network it added hundreds of millions of new pages to OpenAI's index.

Basis
A Cloudflare blog post co-written with OpenAI (2026-10-15) and a Cloudflare press release (2026-07-08), read 2026-10-16 and 2026-10-21.
Does not prove
These are the two companies' own statements, with no data we can check. Not stated: how sites join, the bot's name, the delay, or gains per site or country. In our logs, 2026-10-13 to 2026-10-21, OpenAI's search bot fetched only robots.txt, never our pages.
Sources
What a site owner can do

There is no published way to join, so do not count on it. Return real 404s for missing pages, avoid redirecting unknown URLs to your homepage, and use Cloudflare AI Crawl Control to see how AI systems access your site.

External sourceRecorded 2026-10-16, re-check by 2026-11-15

Google accepts an RSS or Atom feed as a sitemap and says WebSub can announce changes; on our site, a WebSub ping brought Googlebot to the feed within 22 seconds, but to 0 of 2 new pages in 72 hours (2026-10-17 to 2026-10-20).

Basis
Google's sitemap documentation and the W3C WebSub specification, read 2026-10-16; our own server logs on pw.migurei.ai, 2026-10-17 to 2026-10-20.
Does not prove
One deploy, no comparison without a ping, on a site Googlebot crawled very rarely, so it does not show WebSub is slow or useless elsewhere. Google gives no speed for it, and no documentation says AI search bots read WebSub.
Sources
What a site owner can do

If you publish news or articles, add an Atom or RSS feed with a WebSub hub link and ping the hub (free, no account). For new pages, still use Request indexing and links from pages that are already crawled.

External sourceRecorded 2026-10-16, re-check by 2026-12-16

The official MCP Registry accepts remote MCP servers (a streamable-http URL plus a website URL) without an account: the domain owner proves control with a DNS TXT record or a /.well-known/mcp-registry-auth file, and aggregators are expected to copy its public API regularly, for example hourly.

Basis
The MCP Registry documentation and its GitHub reference, read 2026-10-16; our own listing via the HTTP file method on 2026-10-26.
Does not prove
No document says which aggregators read the registry or how they show the website URL; 'listed within a day' comes from one blog post. The registry is still a preview. Entries cannot be edited: a wrong one can only be marked deprecated or deleted.
What a site owner can do

If your site has a real MCP server, publish it in the official MCP Registry before other directories, using DNS or HTTP domain verification. Check the metadata carefully first, because it cannot be changed after publishing.

External sourceRecorded 2026-10-17, re-check by 2026-11-17

OpenAI states that if a site allows both OAI-SearchBot and GPTBot, it may use one crawl for both search and training, so seeing no OAI-SearchBot page fetches does not mean your pages are missing from OpenAI's search data.

Basis
OpenAI's crawler documentation, read 2026-10-16, plus our own logs on pw.migurei.ai showing GPTBot fetching pages right after OAI-SearchBot read robots.txt.
Does not prove
"May" says nothing about when, which bot, or how often. It does not say GPTBot's pages enter the search index, or how fast. Of 719 .jp sites we checked, 4.9% allow search but block training; that is what sites declare, not bot behaviour.
What a site owner can do

Allow OAI-SearchBot if you want to appear in ChatGPT search. When judging whether OpenAI has your pages, count GPTBot and OAI-SearchBot visits together, and allow about 24 hours after changing robots.txt.

Also underC3

Our lab observationRecorded 2026-10-17, re-check by 2026-10-23

GPTBot and ClaudeBot first fetched pw.migurei.ai on 2026-10-16, about an hour after we moved the site to Cloudflare Workers; GPTBot followed home-page links (15 of 17 sitemap URLs), while ClaudeBot took all 17 and kept rereading the sitemap.

Basis
Our own server logs on pw.migurei.ai, 2026-10-02 to 2026-10-16, with bot IPs verified against published ranges.
Does not prove
The cause: the same timing does not show the move drew the bots; Cloudflare network signals, an edge rule or chance are not ruled out. Fetching is not being in OpenAI's or Anthropic's search index; ClaudeBot is a training bot. One day, one host.
Sources
our own data, no outside link
What a site owner can do

Link every page from your home page with plain HTML links, because GPTBot reached only linked pages and missed the two unlinked ones. After a hosting change, check your logs for verified GPTBot and ClaudeBot visits.

Our lab observationRecorded 2026-10-21, re-check by 2026-10-28

On 2026-10-21, Bing Webmaster Tools showed the Japanese M-Meeting page on migurei.ai last fetched on 2026-10-03 with a size of 0 bytes, while the other three language versions were about 196 to 200 KB; Bing held no migurei.ai sitemap.

Basis
Bing Webmaster API read on 2026-10-21, plus our server logs for migurei.ai on the same day.
Does not prove
Size 0 may be how the API reports an unprocessed fetch, not an empty page in the index; the status field was 0 for all four URLs, so it says nothing. Bing knowing a URL does not mean it is indexed. Older Bingbot log lines were matched by user agent only.
Sources
What a site owner can do

Check each language version of key pages in Bing Webmaster Tools. If one shows size 0 or an old fetch date, submit that URL, and submit your sitemaps there directly: listing them in robots.txt was not enough here.

Our lab observationRecorded 2026-10-21, re-check by 2026-11-12

On 2026-10-21, M-Meeting's Japanese, Korean and Vietnamese pages were reachable only through the sitemap and llms.txt (no links, no hreflang), and in 14 days Googlebot never fetched the Vietnamese page; after Request indexing on 2026-10-26, Googlebot fetched all three within 24 hours.

Basis
Our fetch of the live pages and our server logs for migurei.ai, 2026-10-07 to 2026-10-29, plus Google Search Console data.
Does not prove
Request indexing does not promise a crawl; Google may have come anyway (the main page, not submitted, was also crawled). Early counts were by user agent only. A crawl is not ranking or AI citation. Google now reports links between versions; we have not rechecked the HTML.
Sources
What a site owner can do

Link each language version from the others with normal links and hreflang, add lastmod dates to sitemap entries, and use Request indexing in Google Search Console for new language pages.

Our lab observationRecorded 2026-10-14, re-check by 2026-10-28

Twice, after we submitted URLs in Brave's re-fetch form (2026-10-14 and 2026-10-21), a client calling itself Bravebot fetched exactly those URLs about 61 and 67 minutes later, including URLs it had never fetched; it never fetched our sitemap.

Basis
Our own server logs on pw.migurei.ai, 2026-10-14 to 2026-10-28.
Does not prove
The user agent cannot be verified: Brave publishes no user agent or IP ranges, and the requests came from Hetzner addresses. Only two occasions. A fetch is not a place in Brave's index. Claude did not name us more often afterwards.
Sources
What a site owner can do

After publishing a new page, submit it in Brave's re-fetch form, then check your logs about an hour later for Bravebot requests to that URL. Record them as unverified visits.

Our lab observationRecorded 2026-10-16, re-check by 2026-10-23

Our links from migurei.ai to pw.migurei.ai went live in stages: product-page sidebars by 2026-10-15, then the home page carousel and the scanner pages on 2026-10-16; in the following 72 hours no AI search bot followed the scanner-page links to pw.migurei.ai.

Basis
Plain fetches of migurei.ai pages on 2026-10-16 and 2026-10-22, plus our server logs for pw.migurei.ai.
Does not prove
We had no migurei.ai logs then, so we do not know which bots read the linking pages. 72 hours is short. On the home page the link sits in a carousel card that is not among the first four visible at 1280 px width.
Sources
What a site owner can do

After adding a link to another of your sites, fetch the live page to confirm the link is really in the HTML (a failed build can silently block a deploy), and read your logs before concluding the link does nothing.

Our lab observationRecorded 2026-10-16, re-check by 2026-10-30

In 1,571 test runs from 2026-09-30 to 2026-10-15, ChatGPT cited pages on migurei.ai (for example /en in 25 runs and the terms of service in 20), although OpenAI's search crawler had fetched only robots.txt there and on pw.migurei.ai.

Basis
Test questions we asked ChatGPT and other apps (2026-09-30 to 2026-10-15), checked against our server logs and Bing Webmaster Tools data up to 2026-11-03.
Does not prove
Almost every question pasted our URL or named Parallel World, so these were not discovery questions. ChatGPT may take results from Bing rather than from OpenAI's crawler, but Bing showed no impressions for the cited pages. To 2026-11-03, OAI-SearchBot fetched only robots.txt on migurei.ai.
Sources
our own data, no outside link
What a site owner can do

Link your newer pages from pages that AI apps already cite, and submit your sitemap in Bing Webmaster Tools. Do not assume OpenAI's crawler must fetch a page before ChatGPT can cite it.

Our lab observationRecorded 2026-10-16, re-check by 2026-10-24

On pw.migurei.ai in October 2026, new pages listed only in the sitemap were first fetched by Googlebot after 35 to over 72 hours, while a new page linked from the home page was fetched after about 6 hours. OpenAI's, Perplexity's and Anthropic's search bots fetched none of the sitemap-only pages.

Basis
Our server logs on pw.migurei.ai, 2026-10-14 to 2026-10-20, with IPs checked against vendors' lists, for two sitemap-only page pairs and one linked page.
Does not prove
Each case is one pair of pages. In the second window Googlebot rarely crawled the whole site, so the delay may come from the site, not the sitemap. Fetching is not indexing or citation. Bingbot fetched one pair within a minute and the other not at all.
Sources
our own data, no outside link
What a site owner can do

Link every new page from a page Googlebot already visits often, such as your home page, instead of relying on the sitemap alone. IndexNow (a protocol to tell Bing about new URLs) helps with Bing but is not guaranteed.

Our lab observationRecorded 2026-10-16, re-check by 2026-11-16

On a large Vietnamese site whose logs we read (2026-06-20 to 2026-09-30), verified PerplexityBot made 24,210 requests to 17,817 paths, none with a referrer, and read robots.txt only once. Volume fell from 15,851 requests in July to 76 in September.

Basis
Server logs of a large Vietnamese site, with PerplexityBot IPs checked against Perplexity's official list, read 2026-10-16.
Does not prove
One site only. Its sitemap was on another host not in the logs, so we cannot rule out sitemap use. No referrer may just mean the bot does not send one, not that it ignores links. The September drop may come from Perplexity, the site or missing logs.
Sources
our own data, no outside link
What a site owner can do

Verify PerplexityBot by IP in your own logs. Do not expect a referrer or a robots.txt read before its visits, and do not count on new links alone to bring PerplexityBot to a page.

Our lab observationRecorded 2026-10-18, re-check by 2026-11-01

On pw.migurei.ai, GPTBot read the sitemap every day from 2026-10-16 to 2026-10-19 but fetched none of 4 new URLs listed only there; ClaudeBot fetched all 4 and Googlebot 2.

Basis
Our own server logs on pw.migurei.ai, 2026-10-16 to 2026-10-21, with OpenAI IPs checked against OpenAI's published lists.
Does not prove
Four reads in four days on one site; OpenAI may fetch sitemap URLs on a longer schedule. Why GPTBot reads the sitemap is unknown. GPTBot is a training bot; how far OpenAI's search uses its crawl has not been measured.
Sources
our own data, no outside link
What a site owner can do

Do not rely on your sitemap to bring OpenAI's bots to new pages; link new pages from pages they already fetch, such as your home page. OpenAI publishes no way to submit URLs.

Our lab observationRecorded 2026-10-19, re-check by 2026-11-19

In October 2026 Amazonbot fetched a pw.migurei.ai URL it could only have seen in llms.txt or in JSON-LD on migurei.ai, and read llms.txt as raw text, so URLs stuck to punctuation became 404 errors.

Basis
Our own server logs on pw.migurei.ai: 13 Amazonbot requests, 2026-10-17 to 2026-10-19, and a 72-hour recheck ending 2026-11-02.
Does not prove
The early requests were not matched to Amazon's IP lists; 13 rows over 2 days. A later test found no fetch clearly due to llms.txt, since new URLs were also in sitemaps and links. A fetch says nothing about Alexa answers.
Sources
What a site owner can do

In llms.txt and other text files, separate every URL from punctuation with a space (watch for '):' and Japanese '）' or '。'), or redirect URLs with trailing punctuation to the real page.

Also underA8

Our lab observationRecorded 2026-10-28, re-check by 2026-11-28

From 2026-10-21 to 2026-10-27, OAI-SearchBot fetched migurei.ai's English home page 7 times and followed links to other migurei.ai pages, but never followed links on those pages to pw.migurei.ai, where it only read robots.txt (20 of 20 requests).

Basis
Our own server logs for migurei.ai and pw.migurei.ai, to 2026-10-28, with bot IPs checked against OpenAI's published list.
Does not prove
Small sample: it went deeper only once. It may choose links by position or priority, not by host. Link presence is from our source code, not live HTML. OpenAI may queue new hosts separately. A later same-host test was inconclusive because the bot stopped visiting.
Sources
What a site owner can do

Put content you want in ChatGPT search on a host OAI-SearchBot already crawls, linked from pages it already fetches. Check your logs before counting on a link to a new subdomain.

Our lab observationRecorded 2026-10-28, re-check by 2026-11-28

After our MCP server was listed in the official MCP Registry on 2026-10-26, its endpoint got 4,362 requests from 188 user agents in 2.3 days, versus 16 in the 25 days before, and GPTBot twice followed links from MCP directory pages (mcppedia.com, verifymcp.io) to pw.migurei.ai.

Basis
Our own server logs for pw.migurei.ai, 2026-10-01 to 2026-10-28, with GPTBot's IP checked against OpenAI's published list.
Does not prove
GPTBot is OpenAI's training crawler; we have not seen OAI-SearchBot read directory pages. Most new user agents are liveness or scoring probes, not real users. We have no logs of the directory sites. One listing, first week only.
Sources
What a site owner can do

If you run a real MCP server, list it in the official MCP Registry: copies on directory sites give you third-party links that OpenAI's crawler follows. Do not expect human visitors from it.

Also underA6

Our lab observationRecorded 2026-10-28, re-check by 2026-11-28

Our new blog on migurei.ai, with its sitemap declared only in robots.txt and no links from the main site, stayed unknown to Google and Bing for 13 days (28 of 28 URLs, 2026-10-28); after one submission through their tools, Google indexed 23 and Bing 20 within a week.

Basis
Google URL Inspection, Bing Webmaster Tools inspection and our server logs for migurei.ai, 2026-10-15 to 2026-11-04.
Does not prove
With no unsubmitted group to compare, waiting longer may also explain the change. A footer link added on 2026-10-29 was not followed by any search bot within 72 hours, but sitemap and link effects cannot be separated. Indexing is not AI citation.
Sources
our own data, no outside link
What a site owner can do

Do not rely on a sitemap listed only in robots.txt for a new section. Submit the sitemap in Google Search Console and the URLs in Bing Webmaster Tools, and link the section from your home page.

Our lab observationRecorded 2026-10-21, re-check by 2026-11-04

On pw.migurei.ai, from 2026-10-17 to at least 2026-11-02, Bing read our sitemap many times and accepted every submission (IndexNow, URL submission API, Request indexing, all HTTP 200), yet Bingbot did not fetch the submitted new pages.

Basis
Our server logs, the Bing Webmaster API and Bing's URL Inspection for pw.migurei.ai, 2026-10-20 to 2026-11-02.
Does not prove
One host over a few weeks. We do not know why. Bing's only recommendation was a lack of inbound links from high-quality domains, which fits but does not prove the cause. Links from our own blog on the same domain did not help. Bing's reports lag the logs by days.
Sources
What a site owner can do

Do not treat HTTP 200 from a submission as a fetch. Check your logs for Bingbot visits to submitted URLs and their URL Inspection status; if pages stay Discovered but not crawled, work on links from other domains.

Our lab observationRecorded 2026-10-14, re-check by 2026-10-28

When we pasted a pw.migurei.ai URL into Perplexity (October 2026), Perplexity-User never fetched it. PerplexityBot often arrived seconds to 34 minutes later, but it re-fetched pages it already knew and fetched the pasted page after only 2 of 14 questions.

Basis
Our server logs on pw.migurei.ai, 2026-10-12 to 2026-10-21, matched to test questions we asked Perplexity. PerplexityBot IPs were checked against Perplexity's published list.
Does not prove
That our questions caused the visits rather than coincidence (we asked Perplexity many questions), or that the pages entered Perplexity's index. Few questions and no pre-registered test. Two visits that looked like PerplexityBot by user agent came from our own office, so the user agent alone proves nothing.
Sources
our own data, no outside link
What a site owner can do

Check your logs for PerplexityBot and verify each visit's IP against Perplexity's published IP list before you count it. Do not expect that pasting your URL into Perplexity will get that page fetched.

Also underA4

Our lab observationRecorded 2026-11-01, re-check by 2026-11-12

The same submissions (Google sitemap submit, Bing URL submission) gave opposite results by host: our blog on migurei.ai, a host both engines already knew, had 17 of 28 URLs indexed by Google and 20 of 28 by Bing within 72 hours; a new GitHub Pages host had 0 of 11 known.

Basis
Google URL Inspection, Bing Webmaster Tools data and our server logs, 2026-10-28 to 2026-11-06.
Does not prove
The two hosts differ in age, Search Console setup, internal links and resubmissions; we cannot tell which mattered. 72 hours is not final. Indexing is not AI citation. A later post announced only via IndexNow was not indexed by Bing within 72 hours.
Sources
our own data, no outside link
What a site owner can do

Put text you need indexed soon on a host search engines already know. Submit its sitemap in Google Search Console and its URLs in Bing Webmaster Tools; do not rely on IndexNow (a protocol to tell Bing about new URLs) alone.

Our lab observationRecorded 2026-11-02, re-check by 2026-11-16

In November 2026, Bing Webmaster Tools reported that pw.migurei.ai lacks inbound links from high-quality domains, and its backlink report was empty. The links we placed on note and GitHub were nofollow or sat on GitHub Pages, which Bing had not indexed.

Basis
Bing Webmaster Tools recommendations and backlink reports for our sites, and a check of the rel attribute on each linking page, 2026-11-02.
Does not prove
Bing does not say this is why it has not crawled four of our URLs, and publishes no threshold. nofollow is only a hint, and Bing already knew those URLs, so the problem is choosing to crawl, not discovery. One read per page.
Sources
our own data, no outside link
What a site owner can do

Read the Backlinks report in Bing Webmaster Tools, check the rel attribute of links you placed elsewhere, and get at least one followed link from an outside domain Bing has indexed to the exact pages Bing has not crawled.

Our lab observationRecorded 2026-11-09, re-check by 2026-11-23

After we posted two dev.to articles (2026-11-03 and 2026-11-07), PerplexityBot fetched every linked page of ours within 3 to 10 minutes, and Bingbot fetched one article's canonical URL within 6 minutes, a page Bing had left uncrawled despite IndexNow submission.

Basis
Our own server logs on pw.migurei.ai, with bots verified against vendor IP lists, 2026-10-10 to 2026-11-09, and dev.to's public API for post times.
Does not prove
Only two articles, one with a canonical URL, so we cannot tell whether Bingbot came for the canonical URL or the first link. The mechanism is unknown. A bot fetch does not mean indexing. Google might pick the dev.to copy as canonical. An article on note did not pull PerplexityBot.
Sources
What a site owner can do

If Bing ignores a new page, try one dev.to cross-post whose canonical URL points to your page, then check your logs for Bingbot and PerplexityBot within an hour. Afterwards confirm Google still treats your page as canonical.

Our lab observationRecorded 2026-11-09, re-check by 2026-11-23

No page on pw.migurei.ai linked to our research articles, and in the 30 days to 2026-11-09 Googlebot had not fetched 13 of 49 sitemap URLs. Google's index check on 2026-11-03 listed 9 new articles as 'Discovered - currently not indexed'.

Basis
Our server logs (verified IPs) for 30 days to 2026-11-09, a link check of our pages, and Google's URL Inspection API on 2026-11-03.
Does not prove
That missing links are the cause: Googlebot crawls the site rarely, the articles were 6 days old, and one unlinked article was fetched anyway. Fetching is not indexing. It says nothing for Bing: same-site links did not bring Bingbot on migurei.ai.
What a site owner can do

Link each new article from pages Google already fetches, such as your home page or an article index page, and make sure the index page does not return 404. Do not rely on the sitemap alone.

Also relevant here

A4. An app fetching for a user is not an index bot

ChatGPT-User and Claude-User fetch a page while answering without putting it in an index; sometimes they bring the same company's index bot along, but not reliably.

Evidence for the principleHypothesis

HypothesisRecorded 2026-10-13, re-check by 2026-11-13

On one large site, when ChatGPT-User (the bot that loads a page for a user's chat) first loaded a URL that OpenAI's search crawler lacked, OAI-SearchBot fetched it within seconds in 13–40% of cases by month (28% overall). On pw.migurei.ai it fetched only robots.txt.

Basis
Server logs of a large Vietnamese site (14,969 first fetches, 2026-06-19 to 2026-10-02) and our own server logs on pw.migurei.ai.
Does not prove
OpenAI does not confirm this trigger. We only see two fetches close in time. We do not know why it stops at robots.txt on our site. Part of the effect may come from network signals that Cloudflare has sent to OpenAI's search bot since July 2026.
Sources
What a site owner can do

Nothing yet: this is a hypothesis.

External sourceRecorded 2026-10-15, re-check by 2027-01-15

ChatGPT and Claude can open your page at the moment a user asks, as ChatGPT-User and Claude-User; Anthropic says blocking Claude-User stops Claude from getting your content for that question and may reduce your site's visibility.

Basis
OpenAI, Anthropic and Perplexity bot documentation, read 2026-10-15 and again 2026-11-02; plus our own server logs during test questions, 2026-10-17 and 2026-10-27.
Does not prove
Our block test only changed the user agent from our own IP, not OpenAI's; a firewall that checks IPs may treat real fetchers differently. OpenAI does not say how much blocking ChatGPT-User reduces visibility. In our test, Claude-User's IPs did not match Anthropic's list for that day.
Sources
What a site owner can do

Check that robots.txt, your firewall and your CDN do not block ChatGPT-User, Claude-User or Perplexity-User by accident; if you filter by IP, allow OpenAI's published list. Link important pages from your home page, which these fetchers open first.

Also underD1

RefutedRecorded 2026-10-16, re-check by 2026-10-20

Refuted on our site: when ChatGPT or Gemini fetched a pw.migurei.ai page for a user, OAI-SearchBot and Googlebot did not fetch that page within 72 hours (0 of 10 pages for each, October 2026).

Basis
A test fixed in advance on 2026-10-13: 24 test questions with pasted URLs to ChatGPT, Gemini and DeepSeek, scored blind against our server logs.
Does not prove
0 of 10 only rules out a follow rate above about 26% per app; lower rates remain possible. One new subdomain, one account per app, one morning. Claude was not tested. Sites the bots already crawl may differ.
Sources
our own data, no outside link
What a site owner can do

Do not expect users pasting your URLs into ChatGPT or Gemini to bring those vendors' search bots to the pages. Give bots other ways to find URLs: links from crawled pages, sitemap.xml, and Search Console for Google.

Our lab observationRecorded 2026-10-17, re-check by 2026-10-24

In October 2026, Perplexity never fetched our pw.migurei.ai pages at answer time. Across 8 questions pasting our URLs on 2026-10-16 and 2026-10-17, no Perplexity-User request reached our server, and in 7 of them Perplexity said it could not retrieve the page.

Basis
Test questions we asked Perplexity with a free account, 2026-10-16 to 2026-10-17, our server logs, and Perplexity's bot documentation, read 2026-10-17.
Does not prove
Why it happens. Without Cloudflare edge logs we cannot tell whether a block at the edge or Perplexity itself stops the fetch (a pasted GitHub URL was refused too). One free account, one machine, two days. On another site, Perplexity-User traffic came from IPs not on Perplexity's list, so it may be fake.
Sources
What a site owner can do

Ask Perplexity about one of your URLs and check your logs for Perplexity-User from IPs on Perplexity's published list. If the answer says it could not open the page, check your CDN or firewall security events for blocks.

Our lab observationRecorded 2026-10-17, re-check by 2026-11-17

When we pasted our URLs into AI apps (2026-10-01 to 2026-10-17), each fetched the page with its own user agent: ChatGPT 'ChatGPT-User/1.0', Claude 'Claude-User/1.0', Gemini a bare 'Google', DeepSeek an ordinary Linux Firefox; Perplexity sent no request.

Basis
Our own server logs on pw.migurei.ai, 2026-10-01 to 2026-10-17, matched to the test questions we asked each app.
Does not prove
One site, one account per app, questions sent from our lab machine. User agents and IPs can change. We do not know whether Gemini uses 'Google' for all users; its addresses are in Google's general ranges but in no crawler or fetcher list, so they cannot be verified.
Sources
What a site owner can do

When reading your logs, remember that requests with the bare user agent 'Google' or a plain Linux Firefox may be Gemini or DeepSeek reading a page for a user, not human visitors.

Our lab observationRecorded 2026-10-21, re-check by 2026-11-04

On 2026-10-21, a Perplexity question that named pw.migurei.ai and the English title of one of our pages, with no URL, made PerplexityBot fetch that exact page 11 seconds later, and the answer cited it. Three similar questions that day did not.

Basis
Four test questions we asked Perplexity on 2026-10-21, matched to our server logs. PerplexityBot's IP was checked against Perplexity's published list.
Does not prove
One hit out of four similar questions, with no pre-registered test. We do not know how Perplexity found the URL or what it read: it said it could not get the article body, though our server sent the full page. A fetch is not indexing.
Sources
What a site owner can do

Ask Perplexity a question that names your site and the title of one page. Then check your logs for a PerplexityBot visit to that page from a verified IP within about a minute.

Our lab observationRecorded 2026-10-17, re-check by 2026-11-17

When our test questions named pw.migurei.ai without a link, Gemini (178 questions) and DeepSeek (165) never opened our pages, to mid-October 2026; ChatGPT and Claude opened our home page in 4 of 4 tries on 2026-10-17.

Basis
Test questions we asked four AI apps, matched against our own server logs on pw.migurei.ai, 2026-10-01 to 2026-10-22.
Does not prove
Gemini and DeepSeek modes and models were not recorded; free accounts, one machine. Only 4 ChatGPT and Claude tries, in one morning. Gemini may read Google's stored copy without visiting us; it even cited links to our pages without fetching them.
Sources
What a site owner can do

Link every important page from your home page, because ChatGPT and Claude start there. After you change a page, get Google to recrawl it (for example with Request indexing in Search Console), because Gemini answers from Google's copy.

Also underD1

External source(partly: our lab observation)Recorded 2026-10-22, re-check by 2027-04-22

Listing paths in robots.txt makes them public: RFC 9309 says the rules are not access control, and about 10 of 14 sites in our latest scan listed /admin or CMS paths under Disallow, mostly platform defaults.

Basis
RFC 9309 (2022-09), OWASP's web security testing guide, and robots.txt files from our latest scan of 14 sites, read 2026-10-22.
Does not prove
That the listed paths can be opened: we never request disallowed paths. Most listings were platform defaults, such as Shopify's, so they carry little weight.
What a site owner can do

Do not use robots.txt to hide admin or private paths. Protect them with authentication, and remove from Disallow any path whose name you do not want made public.

Also underC4

RefutedRecorded 2026-11-01, re-check by 2026-11-15

Refuted: PerplexityBot fetched our pages 12 times from 2026-10-29 to 2026-10-31 outside our tests, but these were not answer-time fetches for users. In 9 blind-graded questions on 2026-11-02, Perplexity cited us 0 times and fetched nothing while answering.

Basis
Our server logs with IPs checked against Perplexity's published PerplexityBot list, and 9 blind-graded test questions we asked Perplexity on 2026-11-02.
Does not prove
The refutation covers only the 3 discovery questions we asked, from one machine and one Japanese IP, probably logged in to a free account. The 12 visits are real and may be scheduled indexing. We cannot see what users asked.
Sources
What a site owner can do

Do not read PerplexityBot visits in your logs as proof that Perplexity users are being answered with your pages. Treat them as possible indexing, and test with your own questions.

Our lab observationRecorded 2026-11-01, re-check by 2026-11-15

Claude Code (Anthropic's coding agent, which fetches pages from the user's own computer) read one of our research pages on pw.migurei.ai in two sessions by people outside our team, on 2026-10-28 and 2026-10-29, not from any test we ran.

Basis
Our own server logs on pw.migurei.ai, read 2026-10-31: three fetches from two countries with a Claude-User (claude-code) user agent.
Does not prove
We do not know what the users asked; they may have pasted the link. Two people and three visits are not a trend. The user agent is self-declared and the IPs are users' own, so they cannot be verified.
Sources
our own data, no outside link
What a site owner can do

Check your logs for Claude-User requests whose user agent contains "claude-code": these are developers' coding agents reading your pages. Do not store or publish their IP addresses beyond need; they are personal data.

Also relevant here

A5. A consistent entity

Name, parent company, products and prices must match on your own pages and on other people's pages; a namesake or mismatched information elsewhere turns into a wrong answer.

Evidence for the principleExternal source

External sourceRecorded 2026-10-13, re-check by 2027-01-13

In AI agent tools, the name "Parallel" already belongs to at least two larger players, Parallel Web Systems (parallel.ai) and Parallel Labs' "Parallel AI" (parallellabs.app), while we found no GEO tool named "Parallel World" other than ours (2026-10-13).

Basis
The two companies' websites and web searches, 2026-10-13, plus test questions we asked AI apps, where 1 of 16 DeepSeek answers confused us with Parallel Labs.
Does not prove
Not finding something in a search does not mean it does not exist. Some AI descriptions of another "Parallel World" may be invented, not real namesakes. 1 of 16 is one round. It does not show the name is why AI apps do not mention us.
Sources
What a site owner can do

If your brand name is shared with others, pair it with your company name in page titles, descriptions and structured data, and check whether AI apps confuse you with namesakes.

Also underD2

Our lab measurementRecorded 2026-10-12, re-check by 2027-01-12

Free web Gemini sometimes does not open a URL you paste: in our October 2026 tests it skipped 9 of 48 pasted pages, answered from a web search on the brand name instead, and got 0 of those 9 right.

Basis
Two graded batches of 48 test questions to free Gemini, 2026-10-05 and 2026-10-10, matched against our server logs.
Does not prove
Two batches, one account, one machine, free version, Vietnamese IP; Gemini's model and mode were not recorded. Why Gemini skips the page is unknown, and blocking at the network edge is not ruled out. 2 of 24, then 7 of 24, is not yet a trend.
Sources
our own data, no outside link
What a site owner can do

Make sure pages about your brand on other sites, such as a parent company page or press releases, match your own site (for example the legal company name), because Gemini may answer from them instead of your page.

Also underB3

External sourceRecorded 2026-10-14, re-check by 2027-01-14

In an NTT Data survey (2026-08-28 to 2026-08-31, n=3,948), 37.9% of users had seen an AI answer that conflicted with a company's official information, and 66.6% of those dropped a purchase; 69.7% of people buying via AI followed its suggestion.

Basis
A consumer survey by NTT Data (n=3,948, August 2026), as reported by an LLMO agency blog; we have not read the original report.
Does not prove
Respondents are consumers in general, not B2B buyers. 'Dropped a purchase' is self-reported. The blog that reports the survey sells LLMO services.
Sources
What a site owner can do

Ask ChatGPT, Gemini and Google AI Mode about your company name, 'product + price' and 'company + reputation' on several days. Save answers and cited URLs, compare them with your official pages, and fix outdated facts on your site, PR TIMES and note.

Also underB5D6

External sourceRecorded 2026-10-17, re-check by 2027-01-17

Google says Organization structured data on your home page can help it understand your organization and tell it apart in search results; recommended properties include name, alternateName, description, legalName, sameAs, url and logo, but not disambiguatingDescription (read 2026-10-17).

Basis
Google Search Central page on Organization structured data, updated 2026-09-08, read 2026-10-17.
Does not prove
It is about the organization, not its products. It does not say the Gemini app uses this data, and no source says ChatGPT, Claude or Perplexity read JSON-LD to tell products apart. Can help is Google's wording, with no numbers.
Sources
What a site owner can do

Add Organization markup to your home page with name, alternateName, legalName, description, url, logo and sameAs links to your official profiles, so Google can tell your company apart from others with similar names.

Also underD2

RefutedRecorded 2026-10-14, re-check by 2026-10-21

Refuted: naming Parallel World next to competitors in a question did not make AI apps open pw.migurei.ai. In 24 tries each (2026-10-12 to 2026-10-14), ChatGPT fetched it 3 times, only when the question contained the host name, and Gemini 0 times.

Basis
Japanese test questions we asked ChatGPT and Gemini, with rules fixed in advance and blind grading, matched against our server logs, 2026-10-12 to 2026-10-14.
Does not prove
Two days, one free account per app, Japanese questions only. DeepSeek was inconclusive and Claude not measured. Blocks at the CDN edge were not checked. ChatGPT did search for the name (22 of 24); only the page fetch was refuted.
Sources
our own data, no outside link
What a site owner can do

Do not expect AI apps to read your own site because a question names you. Their answers come from search indexes and other pages about you, so keep your company pages and third-party mentions accurate.

Our lab observationRecorded 2026-11-05, re-check by 2026-11-19

After we published our weekly robots.txt census of the top 1,000 .jp sites on DEV Community, Perplexity used our numbers in 4 of 4 answers (0 of 2 before), but named migurei in only 1 and never linked pw.migurei.ai (2026-11-02 to 2026-11-04).

Basis
Six test questions we asked Perplexity (two statistics questions), 2026-11-02 to 2026-11-04, graded against criteria fixed beforehand.
Does not prove
That Perplexity picked up the numbers because we published the article (the two earlier answers were on other days), or that putting our name in the article title would make Perplexity name us. Only 6 answers from one account and one machine, and we do not know if the account was logged in.
Sources
our own data, no outside link
What a site owner can do

When you publish your numbers on another platform, put your brand name next to them in the title and first lines. Then ask Perplexity the question and check whether it names you, not just your numbers.

A6. Third-party confirmation

Apps name a brand when third-party sources (listicles, press releases, review pages, note articles, Wikipedia) mention it.

Evidence for the principleExternal source

External sourceRecorded 2026-10-10, re-check by 2027-01-10

A Peec study of about 200,000 AI answers (September 2025 to March 2026, US markets) found that brands present and ranked high in third-party list articles get named more; our own comparison page, live 2026-10-20, got us named in 0 of 125 test answers.

Basis
Peec AI's blog study 'The Listicle Rank Effect', read 2026-10-10; plus our test questions to ChatGPT, Gemini, DeepSeek and Claude, 2026-10-22 to 2026-11-06.
Does not prove
It is a vendor's study of US markets, not Japan, with a method we have not checked, and it covers only third-party lists, not your own. Our tests did not add us to any list, so the main point was not tested in Japan.
Sources
What a site owner can do

Find the third-party list articles that AI apps actually cite for your category and ask to be included. Do not expect a comparison page on your own site to get you named by itself.

Also underB4

External sourceRecorded 2026-10-14, re-check by 2026-12-14

Profound reports that from about 2026-08-08 ChatGPT cites companies' own pages more (51.8% of citations; product and service pages 22.6% to 30.4%) and list articles less (18.0% to 11.2%), but another study found third-party pages still take 82% of citations for commercial questions.

Basis
A Japanese LLMO agency blog (2026-10-09, read 2026-10-14) summarising studies by Profound, PromptWatch, Writesonic/NP Digital and Ahrefs. We did not read Profound's original.
Does not prove
Two vendor studies reach opposite conclusions. The data is mostly English and US, Profound's sample size is unknown, and OpenAI gives no explanation. Being cited once you are named says nothing about being named in the first place.
Sources
What a site owner can do

Keep your own product, pricing and documentation pages clear and open to OAI-SearchBot (OpenAI's search crawler), because ChatGPT may cite them directly. Keep your third-party coverage too, since the evidence conflicts.

Our lab observationRecorded 2026-10-16, re-check by 2026-10-30

Links on other sites brought almost no people to us: our GitHub repository linking to pw.migurei.ai sent 0 visits in its first 54 hours (October 2026), and a tools directory page with three links sent 0 in about two days (November 2026).

Basis
Our own server logs on pw.migurei.ai, 2026-10-14 to 2026-10-16 and up to 2026-11-02; and a Zenn article by a site owner whose 11 links from one company page brought 0 visitors in 6 months.
Does not prove
54 hours is short for a repository with no stars; the other site is in a different field, and nofollow or link placement may explain its result. Analytics only see visitors with a referrer. It says nothing about bots: links can still lead bots to your pages.
Sources
What a site owner can do

Add a unique utm_source to every link you place on another site, and count the visitors it brings after 14 days before putting more effort into that channel.

Also underD6

External sourceRecorded 2026-10-28, re-check by 2027-01-28

A study of 158,099 AI answers with 468,037 citations (arXiv preprint, 2026-10-10) found that whether the app's search step retrieves a page explains 32% of the variation in citations, page properties only 2.8%; apps also name brands from model memory.

Basis
An arXiv preprint submitted 2026-10-10; we read only its abstract, found through an LLMO agency blog.
Does not prove
Not peer-reviewed. The abstract does not say which markets or apps, or whether Japan is included. 32% is a share of variance in the authors' model, not a share of pages cited. We have not read the full text.
Sources
What a site owner can do

Before reworking page layout, make sure your pages are indexed by the search engines AI apps use, and that you appear on third-party pages those indexes already hold.

Our lab observationRecorded 2026-10-29, re-check by 2026-11-29

Asked on 2026-10-28 how many Japanese sites block AI bots, ChatGPT and Gemini quoted surveys giving 1.0% (of 363,583 sites), 20.4% and 32.3% (of 99 major sites); the figures differ about 30 times because each counts different sites, and no answer said so.

Basis
24 answers from ChatGPT, Gemini, Claude and DeepSeek to two test questions on 2026-10-28, later rounds to 2026-11-05; we read the cited sources on 2026-11-03.
Does not prove
Only correlation: a few answers per app per question, one day per round. An app quoting a source does not mean people read it. We have not proven that ChatGPT chooses sources by the words on the page.
Sources
our own data, no outside link
What a site owner can do

When you publish a statistic, put the denominator, the date and who measured in the title and first sentence, and show it next to other surveys' figures with their sample sizes.

Our lab observationRecorded 2026-10-31, re-check by 2026-11-28

Over three months to 2026-10-28, Search Console's Generative AI features report showed migurei.ai 155 times, mostly its home page, but pw.migurei.ai 0 times, though pw had 32 ordinary web impressions; by 2026-11-01 pw had only 2.

Basis
Search Console reports for our own domain, read 2026-10-30, 2026-11-01 and 2026-11-03.
Does not prove
The report has no queries, does not split AI Overviews from AI Mode, and has no clicks; home page appearances may come from brand-name searches. Some Vietnamese views may be our team. A link shown does not mean Parallel World was named. Not the Gemini app.
Sources
What a site owner can do

Open the Generative AI features report under Performance in Search Console and filter by page to see which of your pages Google already uses in AI answers; put your key text and links on those pages.

Also underB1

Our lab observationRecorded 2026-11-02, re-check by 2026-11-23

From 2026-10-25 to 2026-11-01, our MCP endpoint on pw.migurei.ai received 13,087 requests from 276 user agents naming 141 domains (MCP directories, agent indexes, trust checkers), yet not one browser visit to our sites came from those domains.

Basis
Our own server logs for pw.migurei.ai and migurei.ai, 2026-10-25 to 2026-11-04, and public directory pages read on 2026-11-01.
Does not prove
No referrer does not mean no people: many directories use noreferrer or redirects. The 141 domains include security scanners and uptime checkers. Of 16,050 successful calls in 30 days, only 70 actually used a tool.
Sources
our own data, no outside link
What a site owner can do

Expect bot traffic, not buyers, from MCP directory listings. Before claiming a listing, check whether that directory page appears in the search indexes AI apps use (such as Brave, Bing and Exa).

Also relevant here

A7. Freshness

Apps prefer pages that are new or really changed, the cited part of a page fades within weeks, and the sitemap's lastmod date is used only when it is accurate.

Evidence for the principleExternal source

External sourceRecorded 2026-10-14, re-check by 2027-01-14

Profound found a page cited by AI apps typically loses half its citations 11 days after its peak (median; 78% within two weeks); pages cited by several apps last longer, and Claude Sonnet 5.5 cites newer pages.

Basis
Two Profound blog posts (2026-09-30 and 2026-10-13) on about 883,000 pages across 7 AI apps, 2025-09-08 to 2026-09-08.
Does not prove
Profound sells a measurement tool, did not test whether updating a page extends citations, and has no Japan slice. In our own 12 test questions, ChatGPT kept citing an older survey over our newer page, so freshness did not decide the winner there.
Sources
What a site owner can do

For pages you want cited, update the data for real, show the date on the page and keep the sitemap lastmod accurate. One-off news and press releases fade fastest.

External sourceRecorded 2026-10-10, re-check by 2027-01-10

Google and Bing use the lastmod date in your sitemap to schedule crawling only if it is the real date the page changed; Google ignores priority and changefreq, and Bing may ignore lastmod dates that always equal today.

Basis
Google's sitemap documentation and a 2023 Bing Webmaster blog post, both read 2026-10-10; plus our own server logs, 2026-10-20 to 2026-10-26.
Does not prove
It does not say how much faster pages get crawled. The Bing post is from 2023. Our own test was not clean: our dates kept changing, and five pages fetched by one bot is too few. In 24 hours, URL submissions to Bing brought Bingbot to 0 of 9 pages.
Sources
What a site owner can do

Generate lastmod from the date each page's content really changed, for example from a hash of the main content, never from the sitemap build date or the commit date. Leave out priority and changefreq.

Our lab observationRecorded 2026-10-17, re-check by 2026-10-30

After we rewrote two pages on pw.migurei.ai (live 2026-10-16), Google and Bing kept the old versions: Googlebot refetched them only after we used Request indexing on 2026-10-18, and Bingbot had still not refetched them by 2026-10-27.

Basis
Our own server logs on pw.migurei.ai, plus Google Search Console and Bing URL Inspection, 2026-10-15 to 2026-10-27.
Does not prove
A refetch does not mean the index, or Gemini, now uses the new version. Two pages on one site, not a rate. The byte sizes differ, but we did not compare the text word by word. We do not know when the index updates after a refetch.
Sources
our own data, no outside link
What a site owner can do

After rewriting a page, set its sitemap lastmod to the date the new content went live, not the commit date. Then use Request indexing in Search Console and Bing Webmaster Tools, and check your logs that each bot refetched the page.

A8. Machine-readable

Content sits in HTML that needs no JavaScript, with no automatic redirect by browser settings, robots.txt allows it, and there is structured data. Bots run JavaScript more and more, so a JavaScript redirect affects bots too.

Evidence for the principleExternal source

Our lab measurementRecorded 2026-10-22, re-check by 2027-01-22

A JavaScript redirect to the English version made Google store our 5 Japanese pages with the English URL as their canonical. After we removed it on 2026-10-14, all 5 recrawled pages listed themselves as canonical, and this held through 2026-10-22.

Basis
Our Search Console URL Inspection readings on pw.migurei.ai (baseline 2026-10-15, then 2026-10-17, 2026-10-19 and 2026-10-22) and verified Googlebot visits in our logs.
Does not prove
No control group. The fix cannot be separated from sitemap dates, indexing requests and Google's own recrawl. One site, 5 pages, and criteria set after the change. Being on Google does not mean appearing for Japanese queries or in AI Overviews.
Sources
What a site owner can do

Remove automatic language redirects and keep a language menu and hreflang. In Search Console URL Inspection, check the user-declared canonical of localized pages. If it points to another language while your HTML points to itself, look for a redirect script.

RefutedRecorded 2026-10-13, re-check by 2027-01-13

Refuted: a JavaScript redirect based on browser language did not keep our Japanese pages out of Google. All 5 Japanese pages were already on Google before the change took effect (checked 2026-10-15), though Google had stored the English URL as their canonical.

Basis
Our own Search Console URL Inspection on pw.migurei.ai, 2026-10-13 to 2026-10-19, and Google's documentation on multi-regional sites.
Does not prove
No control group, and one lab site with 5 pages and one account. We cannot separate the effect of sitemap dates, manual indexing requests and Google's own recrawl. 'On Google' does not mean shown for Japanese queries or in AI Overviews.
Sources
What a site owner can do

Do not expect an automatic language redirect to keep a page out of Google's index. Still avoid such redirects, as Google advises, because they can make Google record the other language as the canonical page.

External sourceRecorded 2026-10-14, re-check by 2026-12-14

According to logs from a Japanese site (reported 2026-09-30), since 2026-09-25 OAI-SearchBot and GPTBot run JavaScript when fetching pages: Next.js prefetch requests rose from 2% to 84% of OAI-SearchBot's requests.

Basis
A site operator's log analysis on Zenn (2026-09-30), read 2026-10-14; OpenAI's bot documentation, which does not mention JavaScript.
Does not prove
One author, observation only, with no announcement from OpenAI. We have not seen OAI-SearchBot fetch pages on our own site, so we have not seen it render. We do not know what language its renderer reports.
Sources
What a site owner can do

Assume OpenAI's bots may run your JavaScript like a browser: make sure scripts do not redirect bots away (for example automatic language redirects) or hide content, and test pages with JavaScript on and off.

RefutedRecorded 2026-10-14, re-check by 2026-11-14

Refuted: llms.txt on pw.migurei.ai is not read only by search engines. From 2026-10-16, training bots (GPTBot, ClaudeBot, likely Amazonbot) fetched it like any other URL, while AI search bots and user-triggered fetchers of OpenAI, Anthropic and Perplexity fetched it 0 times.

Basis
Our own server logs on pw.migurei.ai, 2026-10-01 to 2026-10-21, with bot IPs verified where possible, and a test whose outcome we wrote down in advance.
Does not prove
Apps may use llms.txt through other indexes or caches our logs cannot see. AI search bots rarely visit our site at all, so 0 fetches says little. One site, three weeks, two .md URLs.
Sources
What a site owner can do

Do not rely on llms.txt or .md copies to reach ChatGPT, Claude or Perplexity answers. Put what you want AI to read in HTML pages listed in your sitemap. Keep llms.txt as a cheap extra, with each URL standing alone on its line.

External sourceRecorded 2026-10-16, re-check by 2026-11-16

Since 2026-07-01, Cloudflare (including the Free plan) groups AI bot settings into Search, Agent and Training. From 2026-09-15, Training and Agent bots are blocked by default on pages that show ads, while Search bots stay allowed.

Basis
Cloudflare's blog (2026-07-01), Help Net Security (2026-07-02), and a Zenn article on reading these settings through the API. We also read our own zone's settings in October 2026.
Does not prove
No source says which group OAI-SearchBot, Claude-SearchBot or PerplexityBot belongs to, or whether requests for robots.txt are exempt. Settings show only today's state. On our own zone, none of the three groups was blocked.
Sources
What a site owner can do

If your site uses Cloudflare, check the Search, Agent and Training settings in the dashboard or through the bot management API before you conclude that AI bots do not reach your pages.

External sourceRecorded 2026-10-16, re-check by 2027-01-16

Google finds new pages through HTML links (an a tag with an href); its documentation does not say it takes URLs from JSON-LD or plain text, and OpenAI does not say how OAI-SearchBot finds URLs.

Basis
Google's link best-practices page (read 2026-10-16), OpenAI's bot documentation (read 2026-10-13), and our own tests on migurei.ai, October 2026.
Does not prove
The documents cover Google, not OAI-SearchBot, Claude-SearchBot or PerplexityBot; not saying is not the same as not doing. In our 72-hour test no AI bot followed a URL placed only in JSON-LD, but AI bots rarely visited at all, so this does not refute it.
Sources
What a site owner can do

Make every link you want crawled a plain HTML link with an href, not a button that opens pages with JavaScript, and do not rely on URLs inside JSON-LD to get pages found.

External sourceRecorded 2026-10-17, re-check by 2026-12-17

Cloudflare continuously scores Verified bots on whether they respect robots.txt and HTTP 429 (slow down) responses, plans to publish trust levels on Radar and let site owners pick a threshold; bots apply through the BotBase form, with no minimum traffic.

Basis
Cloudflare's blog post on Verified bots (2026-10-16) and its Verified bots policy and Web Bot Auth docs, read 2026-10-17 to 2026-11-02.
Does not prove
No date for when site owners can pick the threshold. Which behaviour label a given bot gets (for example SEO or Data Collection), and so whether AI-blocking presets block it, is not stated. Nothing shows owners' dashboards link to a bot's info page.
What a site owner can do

If you run a bot: identify it by Web Bot Auth signature, a published IP list or reverse DNS, obey robots.txt, crawl-delay and 429, then apply via the BotBase form. If you own a site, decide which trust threshold you will use.

Also underC2D7

Our lab observationRecorded 2026-10-18, re-check by 2026-12-18

A number shown only through a JavaScript count-up animation reads as its start value to bots that do not run JavaScript: on 2026-10-18 the M-Meeting page served 'Languages supported 0+' in its HTML, and AI apps quoted '4 languages' from a PR TIMES press release instead.

Basis
Our plain fetch of the M-Meeting page on migurei.ai and test questions to four AI apps, 2026-10-18; Vercel's AI crawler study (2024) and Google's JavaScript SEO guide.
Does not prove
That the apps used the press release because the page showed 0+ is our inference, with no control. The Vercel study is from 2024; OpenAI bots may now run JavaScript. It does not say which language count is correct.
Sources
What a site owner can do

Put every number you want AI apps to quote in the HTML your server returns, and start any count-up animation from the real value, not 0. Check by fetching your page without JavaScript.

External sourceRecorded 2026-10-17, re-check by 2026-12-17

A study submitted in October 2026 found that AI models that search the web often answer from search-result snippets without opening the source page, and this goes with more outdated answers and more wrong citations.

Basis
The abstract of an academic preprint on arXiv (submitted 2026-10-15): one commercial model with live web search and six open models. Full text not yet read.
Does not prove
The abstract names no model, rate or language, and does not say whether snippets come from the meta description or are cut by the search engine. One paper, not peer reviewed. Our own tests neither proved nor refuted that editing a page's opening changes answers.
Sources
What a site owner can do

Put the facts you need answered correctly (what you are, your key numbers, your host name) in each page's title and meta description, so an answer built only from the snippet is still right.

Also relevant here