日本語EN
Parallel World
Scan your site free

D. The agent world

How the world of AI agents is changing: several clocks for one company, entities instead of pages, indexes with no door for site owners, identity by signature, and buyers who take AI answers into their own documents. Most of these are hypotheses.

Built from our knowledge base at every deploy. 31 principles, 200 claims. Newest note: 2026-11-12.

日本語Handbook index

Each principle is followed by the claims it rests on. Evidence levels are explained on the handbook index.

Show claims by evidence level

D1. Four clocks

A company exists at the same time in the model's memory (months), the search index (days), the fetch at answer time (seconds) and tools (live); a wrong statement must be fixed at the clock that produced it.

Evidence for the principleHypothesis

Our lab observationRecorded 2026-11-02, re-check by 2026-12-02

Common Crawl's CCBot has not fetched any page of pw.migurei.ai since our logs began on 2026-10-01, and Common Crawl's public index has no capture of it in its three latest releases (the newest crawled 2026-10-05 to 2026-10-18).

Basis
Our server logs since 2026-10-01 and Common Crawl's public index, read 2026-11-02.
Does not prove
Which AI models use which Common Crawl release. CCBot may have come before 2026-10-01. Most CCBot lines on migurei.ai were probes for secret files borrowing the name, not real visits. Our robots.txt allows CCBot, so that is not the cause.
Sources
our own data, no outside link
What a site owner can do

Look up your host in Common Crawl's public index. If you want to be included, confirm robots.txt allows CCBot. Treat CCBot requests for files like .env or key files as attack probes, not the real crawler.

Also relevant here

D2. Entities, not pages

Apps pick facts and entities assembled from many nodes; measuring presence means measuring the entity on every node, not only on your site.

Evidence for the principleHypothesis

Also relevant here

D3. Each app has its own door, some are closed

Indexes built for agents (Exa, Parallel) and the model's memory offer no tool for site owners; you can only change them by being present where they already read.

Evidence for the principleHypothesis

External sourceRecorded 2026-10-13, re-check by 2027-01-13

Parallel Web Systems sells web search and page-reading APIs to AI agent builders, on its own index built by its ShapBot crawler; as of 2026-10-28 it sells site owners no view of how their site appears to AI, though "Index by Parallel" promises to pay content owners.

Basis
Parallel's own website, pricing, crawler and Index pages, read 2026-10-13 and rechecked 2026-10-28, plus a SiliconANGLE report (2026-04-28).
Does not prove
These are Parallel's own pages. No data on how many agents in Japan use Parallel. Index by Parallel has no stated launch date or owner dashboard.
Sources
What a site owner can do

Treat Parallel as a channel: allow ShapBot in robots.txt if you want AI agents that use Parallel's API to find your pages, and check your logs for ShapBot visits.

Our lab observationRecorded 2026-10-24, re-check by 2026-11-07

From 2026-10-01 to 2026-10-24, without any submission from us, crawlers of search indexes sold to AI agents fetched most of pw.migurei.ai: ShapBot (Parallel Web Systems) and KeenableBot each took 19 of 25 sitemap URLs, while OAI-SearchBot only read robots.txt (42 times, 0 pages).

Basis
Our own server logs on pw.migurei.ai, 2026-10-01 to 2026-10-24, with IPs checked against the lists Parallel and Keenable publish.
Does not prove
Fetching a page is not being indexed, and not being returned to any agent. We do not know how these bots found the site. Visits are few (ShapBot came twice, ExaSearchBot once), so there is no pattern yet. Bravebot was identified by its user agent only.
Sources
What a site owner can do

Check your own logs for ShapBot, KeenableBot and ExaSearchBot, and verify their IPs against the lists Parallel and Keenable publish. Keep sitemap.xml complete: ShapBot read it before fetching pages.

Our lab observationRecorded 2026-10-24, re-check by 2026-11-05

Exa and Parallel both run free search MCP servers (tool interfaces for AI agents) that need no API key. On 2026-10-28 Exa ranked our English research page first for two queries, under 21 hours after its crawler fetched it; Parallel, whose crawler had not fetched it, did not.

Basis
Exa's and Parallel's documentation, read 2026-10-24, our own queries to both servers on 2026-10-28, and our server logs.
Does not prove
Free results may differ from paid ones. One query each, at one moment; real agents rarely search the exact question. We do not know which apps Japanese buyers use with these indexes, so ranking first in Exa does not mean ChatGPT, Gemini or Claude will name you.
Sources
What a site owner can do

Query Exa's and Parallel's free MCP search with your own page titles to see whether your pages are indexed. If not, check your logs for ExaSearchBot and ShapBot (Parallel's crawler) and make sure robots.txt and your sitemap let them in.

Our lab observationRecorded 2026-10-24, re-check by 2026-12-24

Felo, a Tokyo-made AI search app with Japanese users, left no crawler under its own name in logs we read: zero requests with "felo" or "sparticle" in the user agent over 3.5 months (10.66 million lines) and on our own sites up to 2026-10-24.

Basis
Server logs of a large Vietnamese site (2026-06-19 to 2026-10-02) and of our own sites (to 2026-10-24), plus a web search for a published Felo crawler.
Does not prove
Felo may fetch pages with a normal browser user agent or use another company's index. The Vietnamese site is not where Felo users ask questions. Finding no published crawler page does not mean none exists. A human check blocked our first anonymous question to Felo.
Sources
our own data, no outside link
What a site owner can do

Do not look for a Felo bot in robots.txt or your logs; none is published. Instead, ask Felo about your company and industry yourself and note which sources it cites.

Also relevant here

D4. Checking a site is a free commodity

Since 2026-04-17 Cloudflare grades "is the site ready for agents" for free, weekly, on 200,000 domains; what differs is the outcome and the nodes outside the site.

Evidence for the principleExternal source

D5. Fetched, used and sent back are three different numbers

Money is moving from the fetch to the use (Cloudflare Pay Per Use, beta 2026-09-30; blocked by default on pages with ads since 2026-09-15); a site owner needs all three numbers, per company.

Evidence for the principleHypothesis

External sourceRecorded 2026-10-28, re-check by 2026-12-24

Cloudflare Pay Per Use (beta since 2026-09-30) lets AI companies offer a price for a specific use of your content; you only accept or decline, the buyer self-reports each use (time, URL, event ID), and you see uses and earnings by buyer, domain and time.

Basis
Cloudflare blog posts (2026-07-01 and 2026-09-30), a Cloudflare press release, the pilot sign-up form and Cloudflare developer docs, read 2026-10-28.
Does not prove
Use counts are reported by the buyer; Cloudflare only checks that the URL belongs to an enrolled publisher. Unknown: whether Free plans qualify, size or traffic thresholds, when a payout account is required, who the beta buyers are, the program terms.
What a site owner can do

Do not expect to set your own price, or a price of zero: buyers set prices. If you want per-company usage data, apply through Cloudflare's Pay Per Use pilot form, and read the terms before accepting any offer.

Also relevant here

D6. Answers become sources

AI answers go back onto the web (articles written by AI) and into internal documents (ringi approval documents); a wrong sentence can become a wrong sentence with a source.

Evidence for the principleHypothesis

Also relevant here

D7. Identity by signature

Bots and agents will be recognised by signature (Web Bot Auth, an IETF draft of September 2026), then let in, blocked or charged by that identity.

Evidence for the principleExternal source

External sourceRecorded 2026-10-24, re-check by 2027-01-24

ExaSearchBot, the crawler of Exa's search index for AI agents, signs every request with Web Bot Auth (HTTP Message Signatures, RFC 9421, with an Ed25519 key published at crawler.exa.ai), so it can be verified whatever its IP; all its requests in our logs carried the signature headers.

Basis
Exa's crawler page (last updated 2026-07-30), read 2026-10-24 and 2026-10-30, and our own server logs on pw.migurei.ai and migurei.ai to 2026-10-30.
Does not prove
We only saw the headers; at the time we had not verified any signature ourselves. The statements on the page are Exa's own. We do not know whether Exa returns our pages to any agent.
Sources
What a site owner can do

Verify ExaSearchBot by its signature, not by IP or name. To control it, use the ExaSearchBot token in robots.txt (without it, it follows your rules for major search engines, then *), and use noindex to remove pages from its index.

Our lab observationRecorded 2026-10-24, re-check by 2026-11-24

Some bots already sign requests with Web Bot Auth (a signature in the request headers): from 2026-10-01 to 2026-10-24 we saw 64 signed requests from Ahrefs, Exa and Cloudflare; by 2026-11-06 more joined, but still none from OpenAI, Anthropic, Google, Microsoft or Perplexity.

Basis
Our own server logs on pw.migurei.ai (from 2026-10-01) and migurei.ai (from 2026-10-21), read 2026-10-24 and 2026-11-06.
Does not prove
Headers can be copied, and we have not checked any signature against the sender's public keys. We do not know why AhrefsBot signed only 35 of its 63 requests, or whether other Japanese sites receive signed requests.
What a site owner can do

Check your own logs for the signature-agent and signature-input headers to see which bots sign. Do not treat a signature as proof of identity until you check it against the sender's published key directory.

Our lab observationRecorded 2026-10-24, re-check by 2026-11-24

On migurei.ai from 2026-10-21 to 2026-10-24, many requests using AI bot user agents were secret-file probes (33 of 120 with OpenAI names, 20 of 107 with Anthropic names), and every probe came from IPs outside the vendors' published ranges.

Basis
Our own server logs on migurei.ai (from 2026-10-21) and pw.migurei.ai (from 2026-10-01) to 2026-10-24, filtered by user agent and path.
Does not prove
Path filters may miss probes or catch a real crawler. We do not know who sent the probes. Matching a published IP range is not a reverse DNS or signature check. The window is a few days.
Sources
our own data, no outside link
What a site owner can do

When counting AI bot visits, count only requests whose IP falls in the vendor's published range. Counting by user agent adds impostors: on migurei.ai, 66 of 120 OpenAI-named requests had no matching IP.

Also relevant here

D8. Walled gardens in Japan

LINE Yahoo's Agent i and the marketplaces answer from their own data; for retail and local businesses, being present there is different from being present on the web.

Evidence for the principleHypothesis

External sourceRecorded 2026-11-02, re-check by 2027-01-02

Amazon runs three agents: Amazonbot (may train AI models), Amzn-SearchBot (Alexa search, no training) and Amzn-User (fetches when a user asks Alexa, may not follow all robots.txt rules); Alexa+ launched in Japan on 2026-11-02.

Basis
Amazon's crawler documentation (updated 2026-10-26, read 2026-11-02) and a Web担当者Forum news item of 2026-11-02 (title and summary only).
Does not prove
Amazon does not say that Alexa+ in Japan uses Amzn-SearchBot or Amzn-User for Japanese sites. The number of Alexa+ users in Japan is unknown.
Sources
What a site owner can do

Check your robots.txt: blocking Amazonbot does not block Amzn-SearchBot or Amzn-User. To allow Alexa search but not training, block Amazonbot by name; Amzn-SearchBot follows your rules for other search bots unless named.

D9. Japanese B2B buyers use AI to build the table and people to decide

The goal is not "being named" but "being correct in the comparison table the buyer takes into the approval process".

Evidence for the principleHypothesis

Also relevant here