When the question named the shop, the price source the AI apps showed was the shop's own page: five Japanese Shopify shops, 600 answers
The answer first: we tested the idea that, in Japan, AI apps take a price from marketplaces (Rakuten, Qoo10) or price feeds (Google Shopping, Kakaku) rather than from the shop's own page, on 600 answers from 31 October and 2 November 2026. When the question carried the shop's address or pasted its page, the share of answers showing a price source that showed the shop's own page was 40 of 42 and 40 of 42 for ChatGPT (80 of 84 over the two days). Marketplaces appeared in at most 2 answers per app per day, and price feeds in at most 2. The idea was refuted for all four apps.
Measured on 31 October and 2 November 2026; the rule was fixed on 29 October 2026, before the first question. Recorded 3 November 2026 (graded by the Analyst). We name neither the shops nor their products. The result holds only for questions that name the shop; questions without the shop name were only recorded.
The rule fixed before the first question
Idea tested: in Japan, AI apps take a product's price from marketplaces and price feeds more often than from the shop's own page. Rule: for an app on a given day, if the shop's own page was the source in at least half of the answers that showed a source, that day refutes the idea; both days for the same app refute it for that app.
How we measured
We asked ChatGPT, Gemini, Claude and DeepSeek for the price of items from 5 Japanese fashion shops that run on Shopify, from a Japanese IP, with free accounts. Of the 600 answers, 480 were the checked questions, each carrying the shop's address or pasting the page, and were graded against the true price. The other 120 were exploratory questions that did not name the shop, outside the rule.
The numbers: how often the source was the shop's page
Of the answers that showed a source, the number that showed the shop's own page (day 1 and day 2):
| App | Day 1 (31 Oct) | Day 2 (2 Nov) |
|---|---|---|
| ChatGPT | 40 / 42 | 40 / 42 |
| Gemini | 22 / 25 | 22 / 25 |
| DeepSeek | 15 / 15 | 16 / 16 |
| Claude | 25 / 26 | 22 / 23 |
- A marketplace appeared in at most 2 answers per app per day, Google Shopping in at most 2, and Kakaku in 0. For the items one shop does not sell on any marketplace, 0 of 96 answers showed a marketplace as the source.
- Only 15 to 42 of the 60 checked answers per app and day showed a source we could read; Claude often shows none.
The source and the right price are different things
Of 120 checked answers per app, the right price came in: ChatGPT 103, Gemini 116, DeepSeek 77, Claude 75. A wrong price or wrong item: ChatGPT 9, Gemini 4, DeepSeek 26, Claude 3. No price at all: Claude 42, DeepSeek 17, ChatGPT 8, Gemini 0. On day 1, 3 ChatGPT answers gave an old (struck-through) price. Whether product markup (JSON-LD) helps the price come out right is undecided: it was supported in one of four apps (ChatGPT).
Questions that did not name the shop (outside the rule, recorded only)
The exploratory questions leaned the other way. On day 2, 4 of 15 Gemini answers showed a marketplace source (7 source readings, as one answer can show several) and none showed the shop. This was not tested against a rule. A buyer who does not yet know the shop may be sent through a marketplace; we have not measured that.
Basis and limits
- The questions pointed at the shop. Naming the shop or pasting its page leads an app there, so "the price came from the shop" says nothing about a buyer who does not name it. We did not test that.
- Five Japanese fashion shops on Shopify are not Japanese e-commerce. Apps and accounts were free versions, from a Japanese IP, over two days.
- Few answers showed a source: 15 to 42 of 60 per app and day, so the share of shop sources is on those only.
- Our source classes come from a seed list of hosts that nobody has reviewed yet; it lacks some resellers and does not read prices written in some ways. Three ways of reading the sources (the registry, with an added host, and marketplace first when links are mixed) gave the same result.
- The graded answers were checked by one model; a person has not yet re-read 10% of them. We could not match answers to the shops' own server logs. The
code_shaof the runs isunknown(only theharness_shaof the runner). - Sources:
lab/grades/2026-11-03-ec-source-step1-shopify5.json(the Analyst's grades, rule fixed 29 October 2026) andknowledge/claims/ja-ai-apps-take-price-from-marketplace-or-feed-over-shop-page.md(level: refuted). - What would show more: the same questions without the shop's name, with a rule fixed first, and a shop whose price differs between its page and a marketplace.