Why does OAI-SearchBot read only robots.txt and leave?

• We / our: migurei (a subsidiary of HBLAB), which runs Parallel World and the test site pw.migurei.ai ("pw" below).
• IP check: anyone can claim a bot's name (UA, user agent), so we count a request only when its IP address is in the list the bot's operator publishes.
• Page fetched: a request for the content of a URL listed in the sitemap. /robots.txt and /sitemap.xml are not counted.

Same period, same site, same sitemap (29 URLs, read 26 October 2026 15:58).

• Split by how it named itself, OAI-SearchBot's 49 requests were: OAI-SearchBot/1.4; robots.txt; (permission check only) 11, OAI-SearchBot/1.0 38, and the page-fetching OAI-SearchBot/1.4 0. OpenAI's official page (https://developers.openai.com/api/docs/bots, read 18 October 2026) describes the first two as robots.txt checks.
• OAI-SearchBot has never fetched pw's /sitemap.xml. pw's robots.txt has a Sitemap: line.
• GPTBot is OpenAI's other bot; the pages it fetched are bunched on 16 October 2026 and 26 October 2026. OpenAI says that when a site allows both bots it "may use the results from just one crawl for both use cases" (more in Is ChatGPT visiting your site?).

† KeenableBot is outside our automatic IP check (Keenable's list is not in our bot registry yet); it is counted by the name it gave only. We matched it by hand against Keenable's published list (https://app.keenable.ai/bot/keenablebot.json); on 24 October 2026 all 24 requests were from listed addresses.

On 21 October 2026 at 18:45 Japan time we deployed two changes together.

1. A robots.txt group just for OAI-SearchBot. Before, one User-agent: * group said Allow: / to every bot. The permission is the same, but we added an explicit User-agent: OAI-SearchBot group, thinking a robots.txt change might prompt OpenAI to look at the site again. The basis was thin: once, after a robots.txt change on our company site migurei.ai on 12 October 2026, GPTBot crawled the whole site.
2. Sitemap `<lastmod>` set to the day a page's content really changed. The date moves only when a hash (a value computed from the content) of the page body changes. Google and Bing say they use lastmod only when it is accurate.

These are hypotheses. We do not know which one holds, whether several combine, or whether the reason is something else.

1. It finds pages by following links, and pw gives it nowhere to start. On another site we monitor (logs from June to October 2026), 784 of the 853 times (92%) OAI-SearchBot fetched a new page for the first time, the page was linked from a page OAI-SearchBot itself had just fetched. On pw it has fetched no page yet, so that path cannot start. This is an observation on one other site, not something we tested on pw.
2. Checking robots.txt and fetching pages are separate jobs. OpenAI documents separate names for the check. Regular robots.txt reads may be permission checks only, not a sign that page fetches will follow.
3. GPTBot's crawl is enough. OpenAI says one crawl may serve both search and training. GPTBot has fetched pages, so OAI-SearchBot may have no need to come for them. Whether they were in fact used for search, our log cannot tell.
4. New, barely linked sites wait. pw is a new site, public since early October 2026, with almost no links from other sites. An article on Zenn (by hirajo, 11 October 2026, https://zenn.dev/hirajo/articles/crawler-google-vs-chatgpt) reports that on a new site OAI-SearchBot read only robots.txt for the first days and began fetching pages two days after an article went up (one report, not checked by us). On pw the robots-only period has lasted longer than that.

• Log: pw runs on Cloudflare Workers and records each request that reaches it (time, path, UA, source IP, status code). It does not record page bodies or form contents.
• IP check: matched against the lists each operator publishes (OpenAI, Parallel Web Systems); only requests from listed addresses are counted. We use dated, saved copies of the lists. KeenableBot alone is counted by the name it gave, because Keenable's list is not in our bot registry yet.
• Excluded: requests during our own trials that put questions to AI apps (Runs), and our scanner's requests.
• Sitemap: pw's /sitemap.xml as read on 26 October 2026 15:58 (29 URLs).
• Counting: our tool log-import bot-crawl (code 788c947, bot list bots-2026-10-24). Dates are in Japan time.

After publishing, we will record whether OAI-SearchBot comes to fetch this page and add the result here, including if it does not. Our expectation before publishing is "it will not": on pw, OAI-SearchBot has read nothing but robots.txt.