---
title: "D. The agent world | Parallel World - migurei"
description: "D. The agent world: each principle of the Parallel World lab and the 10 claims under it, with evidence level, sources, what it does not prove and what a site owner can do."
lang: "en"
url: "https://pw.migurei.ai/en/handbook/agent-world"
---

# D. The agent world

How the world of AI agents is changing: several clocks for one company, entities instead of pages, indexes with no door for site owners, identity by signature, and buyers who take AI answers into their own documents. Most of these are hypotheses.

Built from our knowledge base at every deploy. 31 principles, 200 claims. Newest note: 2026-11-12.

[日本語](https://pw.migurei.ai/ja/handbook/agent-world) · [Handbook index](https://pw.migurei.ai/en/handbook)

Each principle is followed by the claims it rests on. Evidence levels are explained on the handbook index.

## D1. Four clocks

A company exists at the same time in the model's memory (months), the search index (days), the fetch at answer time (seconds) and tools (live); a wrong statement must be fixed at the clock that produced it.

**Evidence for the principle:** hypothesis

### [our lab observation] Common Crawl's CCBot has not fetched any page of pw.migurei.ai since our logs began on 2026-10-01, and Common Crawl's public index has no capture of it in its three latest releases (the newest crawled 2026-10-05 to 2026-10-18).

- **Evidence:** our lab observation. Recorded 2026-11-02, re-check by 2026-12-02
- **Basis:** Our server logs since 2026-10-01 and Common Crawl's public index, read 2026-11-02.
- **Sources:** our own data, no outside link
- **Does not prove:** Which AI models use which Common Crawl release. CCBot may have come before 2026-10-01. Most CCBot lines on migurei.ai were probes for secret files borrowing the name, not real visits. Our robots.txt allows CCBot, so that is not the cause.
- **What a site owner can do:** Look up your host in Common Crawl's public index. If you want to be included, confirm robots.txt allows CCBot. Treat CCBot requests for files like .env or key files as attack probes, not the real crawler.

**Also relevant here:**

- [[our lab observation] When our test questions named pw.migurei.ai without a link, Gemini (178 questions) and DeepSeek (165) never opened our pages, to mid-October 2026; ChatGPT and Claude opened our home page in 4 of 4 tries on 2026-10-17.](https://pw.migurei.ai/en/handbook/bots-and-apps#c-d110f820)
- [[external source] ChatGPT and Claude can open your page at the moment a user asks, as ChatGPT-User and Claude-User; Anthropic says blocking Claude-User stops Claude from getting your content for that question and may reduce your site's visibility.](https://pw.migurei.ai/en/handbook/bots-and-apps#c-f283c43b)
- [[our lab observation] In the latest Common Crawl release (CC-MAIN-2026-43, crawled 2026-10-05 to 2026-10-18), note.com has only 36 captures, all of them robots.txt returning 403, and zero content pages, so text posted on note does not reach models through Common Crawl.](https://pw.migurei.ai/en/handbook/japanese-buyers#c-6c16830e)

## D2. Entities, not pages

Apps pick facts and entities assembled from many nodes; measuring presence means measuring the entity on every node, not only on your site.

**Evidence for the principle:** hypothesis

**Also relevant here:**

- [[external source] Google says Organization structured data on your home page can help it understand your organization and tell it apart in search results; recommended properties include name, alternateName, description, legalName, sameAs, url and logo, but not disambiguatingDescription (read 2026-10-17).](https://pw.migurei.ai/en/handbook/bots-and-apps#c-9d7a0e2f)
- [[external source] In AI agent tools, the name "Parallel" already belongs to at least two larger players, Parallel Web Systems (parallel.ai) and Parallel Labs' "Parallel AI" (parallellabs.app), while we found no GEO tool named "Parallel World" other than ours (2026-10-13).](https://pw.migurei.ai/en/handbook/bots-and-apps#c-4f5867a9)

## D3. Each app has its own door, some are closed

Indexes built for agents (Exa, Parallel) and the model's memory offer no tool for site owners; you can only change them by being present where they already read.

**Evidence for the principle:** hypothesis

### [external source] Parallel Web Systems sells web search and page-reading APIs to AI agent builders, on its own index built by its ShapBot crawler; as of 2026-10-28 it sells site owners no view of how their site appears to AI, though "Index by Parallel" promises to pay content owners.

- **Evidence:** external source. Recorded 2026-10-13, re-check by 2027-01-13
- **Basis:** Parallel's own website, pricing, crawler and Index pages, read 2026-10-13 and rechecked 2026-10-28, plus a SiliconANGLE report (2026-04-28).
- **Sources:** [parallel.ai](https://parallel.ai/) (2026-10-13), [parallel.ai](https://parallel.ai/pricing) (2026-10-13), [docs.parallel.ai](https://docs.parallel.ai/resources/crawler) (2026-10-13), [parallel.ai](https://parallel.ai/parallel-web-systems-bots) (2026-10-13), [docs.parallel.ai](https://docs.parallel.ai/resources/shapbot.json) (2026-10-13), [parallel.ai](https://parallel.ai/ai/blog/case-study-profound) (2026-03-04), [siliconangle.com](https://siliconangle.com/2026/04/28/parag-agrawals-startup-raises-100m-build-parallel-web-ai-agents) (2026-04-28), [parallel.ai](https://parallel.ai/index) (2026-10-13)
- **Does not prove:** These are Parallel's own pages. No data on how many agents in Japan use Parallel. Index by Parallel has no stated launch date or owner dashboard.
- **What a site owner can do:** Treat Parallel as a channel: allow ShapBot in robots.txt if you want AI agents that use Parallel's API to find your pages, and check your logs for ShapBot visits.

### [our lab observation] From 2026-10-01 to 2026-10-24, without any submission from us, crawlers of search indexes sold to AI agents fetched most of pw.migurei.ai: ShapBot (Parallel Web Systems) and KeenableBot each took 19 of 25 sitemap URLs, while OAI-SearchBot only read robots.txt (42 times, 0 pages).

- **Evidence:** our lab observation. Recorded 2026-10-24, re-check by 2026-11-07
- **Basis:** Our own server logs on pw.migurei.ai, 2026-10-01 to 2026-10-24, with IPs checked against the lists Parallel and Keenable publish.
- **Sources:** [docs.parallel.ai](https://docs.parallel.ai/resources/shapbot.json) (2026-10-24), [keenable.ai](https://keenable.ai/bot) (2026-10-24), [app.keenable.ai](https://app.keenable.ai/bot) (2026-10-24), [app.keenable.ai](https://app.keenable.ai/bot/keenablebot.json) (2026-10-09), [elevenlabs.io](https://elevenlabs.io/docs/bots) (2026-10-24)
- **Does not prove:** Fetching a page is not being indexed, and not being returned to any agent. We do not know how these bots found the site. Visits are few (ShapBot came twice, ExaSearchBot once), so there is no pattern yet. Bravebot was identified by its user agent only.
- **What a site owner can do:** Check your own logs for ShapBot, KeenableBot and ExaSearchBot, and verify their IPs against the lists Parallel and Keenable publish. Keep sitemap.xml complete: ShapBot read it before fetching pages.

### [our lab observation] Exa and Parallel both run free search MCP servers (tool interfaces for AI agents) that need no API key. On 2026-10-28 Exa ranked our English research page first for two queries, under 21 hours after its crawler fetched it; Parallel, whose crawler had not fetched it, did not.

- **Evidence:** our lab observation. Recorded 2026-10-24, re-check by 2026-11-05
- **Basis:** Exa's and Parallel's documentation, read 2026-10-24, our own queries to both servers on 2026-10-28, and our server logs.
- **Sources:** [exa.ai](https://exa.ai/docs/reference/exa-mcp) (2026-10-24), [parallel.ai](https://parallel.ai/blog/free-web-search-mcp) (2026-10-24), [pw.migurei.ai](https://pw.migurei.ai/en/research/japanese-sites-and-ai-bots) (2026-10-24)
- **Does not prove:** Free results may differ from paid ones. One query each, at one moment; real agents rarely search the exact question. We do not know which apps Japanese buyers use with these indexes, so ranking first in Exa does not mean ChatGPT, Gemini or Claude will name you.
- **What a site owner can do:** Query Exa's and Parallel's free MCP search with your own page titles to see whether your pages are indexed. If not, check your logs for ExaSearchBot and ShapBot (Parallel's crawler) and make sure robots.txt and your sitemap let them in.

### [our lab observation] Felo, a Tokyo-made AI search app with Japanese users, left no crawler under its own name in logs we read: zero requests with "felo" or "sparticle" in the user agent over 3.5 months (10.66 million lines) and on our own sites up to 2026-10-24.

- **Evidence:** our lab observation. Recorded 2026-10-24, re-check by 2026-12-24
- **Basis:** Server logs of a large Vietnamese site (2026-06-19 to 2026-10-02) and of our own sites (to 2026-10-24), plus a web search for a published Felo crawler.
- **Sources:** our own data, no outside link
- **Does not prove:** Felo may fetch pages with a normal browser user agent or use another company's index. The Vietnamese site is not where Felo users ask questions. Finding no published crawler page does not mean none exists. A human check blocked our first anonymous question to Felo.
- **What a site owner can do:** Do not look for a Felo bot in robots.txt or your logs; none is published. Instead, ask Felo about your company and industry yourself and note which sources it cites.

**Also relevant here:**

- [[external source] Anthropic lists Brave Search, along with TurboPuffer, as a subprocessor for Web Search in all its products, but does not say how much of Claude's answers comes from Brave's index.](https://pw.migurei.ai/en/handbook/bots-and-apps#c-e7096d69)

## D4. Checking a site is a free commodity

Since 2026-04-17 Cloudflare grades "is the site ready for agents" for free, weekly, on 200,000 domains; what differs is the outcome and the nodes outside the site.

**Evidence for the principle:** external source

## D5. Fetched, used and sent back are three different numbers

Money is moving from the fetch to the use (Cloudflare Pay Per Use, beta 2026-09-30; blocked by default on pages with ads since 2026-09-15); a site owner needs all three numbers, per company.

**Evidence for the principle:** hypothesis

### [external source] Cloudflare Pay Per Use (beta since 2026-09-30) lets AI companies offer a price for a specific use of your content; you only accept or decline, the buyer self-reports each use (time, URL, event ID), and you see uses and earnings by buyer, domain and time.

- **Evidence:** external source. Recorded 2026-10-28, re-check by 2026-12-24
- **Basis:** Cloudflare blog posts (2026-07-01 and 2026-09-30), a Cloudflare press release, the pilot sign-up form and Cloudflare developer docs, read 2026-10-28.
- **Sources:** [blog.cloudflare.com](https://blog.cloudflare.com/pay-per-use/) (2026-09-30), [blog.cloudflare.com](https://blog.cloudflare.com/making-ai-search-smarter/) (2026-07-01), [www.cloudflare.com](https://www.cloudflare.com/press/press-releases/2026/cloudflare-allows-the-agentic-internet-to-flourish-with-a-simple-philosophy-your-content-your-rules/) (2026-10-28), [developers.cloudflare.com](https://developers.cloudflare.com/ai-crawl-control/llms.txt) (2026-06-16), [www.cloudflare.com](https://www.cloudflare.com/pay-per-use/) (2026-10-28), [developers.cloudflare.com](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) (2026-10-28), [www.cloudflare.com](https://www.cloudflare.com/paypercrawl-signup/) (2026-10-28), [developers.cloudflare.com](https://developers.cloudflare.com/ai-crawl-control/get-started/) (2026-10-28), [developers.cloudflare.com](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price/) (2026-10-28), [developers.cloudflare.com](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/) (2026-10-16)
- **Does not prove:** Use counts are reported by the buyer; Cloudflare only checks that the URL belongs to an enrolled publisher. Unknown: whether Free plans qualify, size or traffic thresholds, when a payout account is required, who the beta buyers are, the program terms.
- **What a site owner can do:** Do not expect to set your own price, or a price of zero: buyers set prices. If you want per-company usage data, apply through Cloudflare's Pay Per Use pilot form, and read the terms before accepting any offer.

**Also relevant here:**

- [[external source] AI bots fetch hundreds to thousands of pages per visitor their app sends back: Cloudflare Radar for 2026-10-16 to 2026-10-23 shows Perplexity 3.3K:1, OpenAI 590:1, Anthropic 495:1, against Google 4.5:1.](https://pw.migurei.ai/en/handbook/open-and-closed#c-6211536e)

## D6. Answers become sources

AI answers go back onto the web (articles written by AI) and into internal documents (ringi approval documents); a wrong sentence can become a wrong sentence with a source.

**Evidence for the principle:** hypothesis

**Also relevant here:**

- [[our lab observation] Links on other sites brought almost no people to us: our GitHub repository linking to pw.migurei.ai sent 0 visits in its first 54 hours (October 2026), and a tools directory page with three links sent 0 in about two days (November 2026).](https://pw.migurei.ai/en/handbook/bots-and-apps#c-bdbb156d)
- [[external source] In an NTT Data survey (2026-08-28 to 2026-08-31, n=3,948), 37.9% of users had seen an AI answer that conflicted with a company's official information, and 66.6% of those dropped a purchase; 69.7% of people buying via AI followed its suggestion.](https://pw.migurei.ai/en/handbook/bots-and-apps#c-b95846b0)

## D7. Identity by signature

Bots and agents will be recognised by signature (Web Bot Auth, an IETF draft of September 2026), then let in, blocked or charged by that identity.

**Evidence for the principle:** external source

### [external source] ExaSearchBot, the crawler of Exa's search index for AI agents, signs every request with Web Bot Auth (HTTP Message Signatures, RFC 9421, with an Ed25519 key published at crawler.exa.ai), so it can be verified whatever its IP; all its requests in our logs carried the signature headers.

- **Evidence:** external source. Recorded 2026-10-24, re-check by 2027-01-24
- **Basis:** Exa's crawler page (last updated 2026-07-30), read 2026-10-24 and 2026-10-30, and our own server logs on pw.migurei.ai and migurei.ai to 2026-10-30.
- **Sources:** [crawler.exa.ai](https://crawler.exa.ai/) (2026-10-24), [crawler.exa.ai](https://crawler.exa.ai/.well-known/http-message-signatures-directory) (2026-10-24), [crawler.exa.ai](https://crawler.exa.ai/) (2026-10-24)
- **Does not prove:** We only saw the headers; at the time we had not verified any signature ourselves. The statements on the page are Exa's own. We do not know whether Exa returns our pages to any agent.
- **What a site owner can do:** Verify ExaSearchBot by its signature, not by IP or name. To control it, use the ExaSearchBot token in robots.txt (without it, it follows your rules for major search engines, then *), and use noindex to remove pages from its index.

### [our lab observation] Some bots already sign requests with Web Bot Auth (a signature in the request headers): from 2026-10-01 to 2026-10-24 we saw 64 signed requests from Ahrefs, Exa and Cloudflare; by 2026-11-06 more joined, but still none from OpenAI, Anthropic, Google, Microsoft or Perplexity.

- **Evidence:** our lab observation. Recorded 2026-10-24, re-check by 2026-11-24
- **Basis:** Our own server logs on pw.migurei.ai (from 2026-10-01) and migurei.ai (from 2026-10-21), read 2026-10-24 and 2026-11-06.
- **Sources:** [ahrefs.com](https://ahrefs.com/) (2026-10-01), [crawler.exa.ai](https://crawler.exa.ai/) (2026-10-16), [web-bot-auth-directory.radar.cloudflare.com](https://web-bot-auth-directory.radar.cloudflare.com/) (2026-10-01), [app.keenable.ai](https://app.keenable.ai/bot) (2026-10-24)
- **Does not prove:** Headers can be copied, and we have not checked any signature against the sender's public keys. We do not know why AhrefsBot signed only 35 of its 63 requests, or whether other Japanese sites receive signed requests.
- **What a site owner can do:** Check your own logs for the signature-agent and signature-input headers to see which bots sign. Do not treat a signature as proof of identity until you check it against the sender's published key directory.

### [our lab observation] On migurei.ai from 2026-10-21 to 2026-10-24, many requests using AI bot user agents were secret-file probes (33 of 120 with OpenAI names, 20 of 107 with Anthropic names), and every probe came from IPs outside the vendors' published ranges.

- **Evidence:** our lab observation. Recorded 2026-10-24, re-check by 2026-11-24
- **Basis:** Our own server logs on migurei.ai (from 2026-10-21) and pw.migurei.ai (from 2026-10-01) to 2026-10-24, filtered by user agent and path.
- **Sources:** our own data, no outside link
- **Does not prove:** Path filters may miss probes or catch a real crawler. We do not know who sent the probes. Matching a published IP range is not a reverse DNS or signature check. The window is a few days.
- **What a site owner can do:** When counting AI bot visits, count only requests whose IP falls in the vendor's published range. Counting by user agent adds impostors: on migurei.ai, 66 of 120 OpenAI-named requests had no matching IP.

**Also relevant here:**

- [[external source] Cloudflare continuously scores Verified bots on whether they respect robots.txt and HTTP 429 (slow down) responses, plans to publish trust levels on Radar and let site owners pick a threshold; bots apply through the BotBase form, with no minimum traffic.](https://pw.migurei.ai/en/handbook/bots-and-apps#c-c8773272)

## D8. Walled gardens in Japan

LINE Yahoo's Agent i and the marketplaces answer from their own data; for retail and local businesses, being present there is different from being present on the web.

**Evidence for the principle:** hypothesis

### [external source] Amazon runs three agents: Amazonbot (may train AI models), Amzn-SearchBot (Alexa search, no training) and Amzn-User (fetches when a user asks Alexa, may not follow all robots.txt rules); Alexa+ launched in Japan on 2026-11-02.

- **Evidence:** external source. Recorded 2026-11-02, re-check by 2027-01-02
- **Basis:** Amazon's crawler documentation (updated 2026-10-26, read 2026-11-02) and a Web担当者Forum news item of 2026-11-02 (title and summary only).
- **Sources:** [developer.amazon.com](https://developer.amazon.com/amazonbot) (2026-10-26), [webtan.impress.co.jp](https://webtan.impress.co.jp/n/2026/11/02/53547) (2026-11-02)
- **Does not prove:** Amazon does not say that Alexa+ in Japan uses Amzn-SearchBot or Amzn-User for Japanese sites. The number of Alexa+ users in Japan is unknown.
- **What a site owner can do:** Check your robots.txt: blocking Amazonbot does not block Amzn-SearchBot or Amzn-User. To allow Alexa search but not training, block Amazonbot by name; Amzn-SearchBot follows your rules for other search bots unless named.

## D9. Japanese B2B buyers use AI to build the table and people to decide

The goal is not "being named" but "being correct in the comparison table the buyer takes into the approval process".

**Evidence for the principle:** hypothesis

**Also relevant here:**

- [[external source] Japanese surveys from 2026 report that 85 to 91% of B2B buyers use generative AI when choosing a service, 53.8% ask AI for comparison tables, and 56.8% want prices before meeting sales; but only 1.6% find AI alone enough, and they verify on official sites.](https://pw.migurei.ai/en/handbook/japanese-buyers#c-62730b38)

```json
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "D. The agent world",
  "inLanguage": "en",
  "dateModified": "2026-11-12",
  "author": {
    "@type": "Organization",
    "name": "migurei"
  },
  "mainEntityOfPage": "https://pw.migurei.ai/en/handbook/agent-world",
  "isPartOf": {
    "@type": "CreativeWork",
    "name": "Handbook for AI agents: how a website exists for AI apps",
    "url": "https://pw.migurei.ai/en/handbook"
  }
}
```