When do AI search bots visit a new subdomain? A 14-day log of pw.migurei.ai (October 2026)

pw.migurei.ai is the own site of Parallel World, run by migurei. For 14 days, from 1 to 14 October 2026, we logged on the server when search-engine and AI-search bots came to this site and how many pages they fetched. This page is that record.

Data taken: 15 October 2026 (the last request in the log is 14 October, 16:54 UTC)

日本語

• Googlebot: did not come once in about 11 days after the log began. It first came on 12 October; on the same day we submitted the sitemap in Search Console and used "Request indexing" (for the order of the times, see the table and its notes below). It fetched all 13 pages.
• Bingbot: fetched all 13 pages about 38 hours after we sent the URLs through IndexNow.
• A visitor calling itself Bravebot: fetched all 13 pages 61 minutes after we entered the 13 URLs in the Brave Search URL submission form. This rests on the User-Agent alone; its IP cannot be verified.
• OAI-SearchBot: came 19 times in 14 days, every time for /robots.txt; it fetched 0 pages.
• Claude-SearchBot and PerplexityBot: 0 pages fetched on their own initiative.

Bot	First visit (UTC / Japan time)	Pages fetched	How the URLs were notified
Googlebot	2026-10-12 01:20 UTC (10:20 JST): robots.txt and sitemap.xml First page: 01:58 UTC (10:58 JST)	13 / 13 (83 requests in all)	Search Console: the sitemap was submitted and "Request indexing" used the same day. The first visit (01:20 UTC) came before the first visit of Google-InspectionTool (01:56 UTC)
Bingbot	2026-10-10 13:53 UTC (22:53 JST): the IndexNow key file First page: 2026-10-12 04:22 UTC (13:22 JST)	13 / 13 (31 requests in all)	IndexNow (sent 10 October, 13:53 UTC). About 38 h 29 min until the pages
Bravebot (self-declared User-Agent, IP not verified)	2026-10-14 05:43 UTC (14:43 JST): robots.txt Pages: 05:44 to 05:46 UTC	13 / 13 (14 requests in all)	The Brave Search URL submission form (Brave describes it as a way to re-fetch updated or removed pages; a person entered the 13 URLs; sent at 04:42:54 UTC). 61 minutes after sending
OAI-SearchBot	2026-10-01 07:46 UTC (16:46 JST): /robots.txt	0 (all 19 visits were /robots.txt)	None
Claude-SearchBot	No visit	0	None
PerplexityBot	No visit on its own initiative (see the note below for the 2 requests we excluded)	0	None
Google-InspectionTool	2026-10-12 01:56 UTC (10:56 JST)	12 / 13 (146 requests in all)	The same time window as our URL Inspection in Search Console (Google describes it as the fetcher behind Search testing tools such as URL Inspection)
GoogleOther	2026-10-12 07:03 UTC (16:03 JST)	9 / 13 (22 requests in all)	Not tied to any action of ours

• "Pages fetched" is the number of URLs, out of the 13 in the sitemap, that the bot fetched with a successful (2xx or 3xx) response. robots.txt, sitemap.xml, the IndexNow key file, images and scripts are not counted. "Requests in all" includes them.
• Googlebot's first visit (01:20 UTC) was only robots.txt and sitemap.xml. It fetched its first page at 01:58 UTC; Google-InspectionTool had come at 01:56 UTC, 1 minute 47 seconds earlier. We did not record the exact time we pressed "Request indexing".
• PerplexityBot: at 10:06 UTC on 12 October, PerplexityBot from an official IP (not the 2 requests from outside the official ranges) fetched /robots.txt and the home page. This coincided with a question we were asking Perplexity, so under our counting rules it is excluded (see the next section).
• User-triggered fetches, where an AI app fetches a page because a user asked (ChatGPT-User, Claude-User), are not in this table: most of them came from our own tests.

1. The log: 4,297 GET and HEAD requests that reached pw.migurei.ai, from the first request at 2026-10-01 07:13 UTC to the last at 16:54 UTC on 14 October. It is the server (origin) log.
2. Exclusions: we count 1,507 requests, after removing 1,624 requests tied to our own AI-app tests and 1,166 requests from our own scanner (User-Agent contains migurei-ParallelWeb).
3. Judging the bot: a request counts as "verified" only if its User-Agent names the bot and its source IP is in the operator's official IP feed (the IP ranges published by Google, Microsoft, OpenAI, Anthropic and Perplexity). If the IP is not in the feed, the request is not counted even when the User-Agent names a bot (2 requests naming Googlebot and 2 naming PerplexityBot fall here).
4. The IP feeds are daily snapshots (2026-09-30, 10-02, 10-11, 10-13); each request is judged against the newest snapshot on or before its day. The list of bots and how to verify them lives in a versioned registry (bots-2026-10-13), not in code.
5. "Fetched" means the server answered the request with 2xx or 3xx (3xx includes redirects).
6. Bravebot: Brave does not publish IP ranges, so it cannot be counted as verified. The table lists requests whose User-Agent contains Bravebot separately, and says so.
7. All times are UTC; the main ones carry Japan time (JST, UTC+9) beside them. The judging used the code at commit 6a75347 of the Parallel World repository.

• One site, 13 URLs, 14 days. The numbers are counts, not stable rates.
• How we notified a URL and the visit cannot be separated as cause and effect. Googlebot came on the same day we submitted the sitemap, and we did not record the sitemap submission time. Its first visit was before the first visit of Google-InspectionTool (01:56 UTC), so it cannot be called a result of the request. For Bingbot we have no record of whether another route than IndexNow, such as registering in Bing Webmaster Tools, existed before 12 October. Bravebot is a single observation on a self-declared User-Agent. None of this shows that the way we notified the URL caused the visit.
• A bot fetching (crawling) a page does not mean it was indexed. A page being fetched and a page being used in search results or AI answers are different things. Google itself says that, even after a crawl request, crawling can take from a few days to a few weeks.
• We only see the origin log. A request blocked in front of Cloudflare would not appear here.
• OAI-SearchBot's 0 pages does not prove it will not come. It only says 0 in 14 days. It fetches robots.txt every 16 to 17 hours (the 7 visits we could confirm in the raw log from 10 October on have a median gap of about 16.0 hours and a mean of about 16.4 hours; across all 19, the mean from first to last is about 17.2 hours), but we do not know whether that means more than a routine robots.txt check. The same goes for Claude-SearchBot's 0.
• Excluding the 2 PerplexityBot requests changes the result. Counted, PerplexityBot would stand at 1 / 13 pages (the home page). They came at the same time as our question, so we cannot tell whether the bot came on its own initiative or was prompted by the question.
• What we could not count: the publication time of each page, the time we submitted the sitemap in Search Console, and the exact time we pressed "Request indexing" were not recorded, so they are left out.
• AI answers are not covered. A bot fetching a page and an AI app using that page in an answer are different things.

An outside article (Zenn, 11 October 2026: https://zenn.dev/hirajo/articles/crawler-google-vs-chatgpt) records a new site whose domain was acquired on 29 August 2026, where OAI-SearchBot fetched only robots.txt, 1 to 6 times a day, for the first 6 days, and first fetched an article on 4 September, two days after the article was published on 2 September. The author says they checked the IPs against OpenAI's official list. This is one author's self-report about one site, and we have not checked it. If there is a difference, it may be that this site had articles while pw.migurei.ai, at the time of this record, had only pages introducing the service; that is a hypothesis.