Asked the same question four times, ChatGPT gave 83 tool names across nine questions and 37 of them appeared in only one of the four answers
The answer first: on 12 October 2026 we put 9 Japanese questions about finding an AI-visibility tool (LLMO or GEO: getting your site into AI apps' answers) to ChatGPT, 4 times each. Over the nine questions there were 83 pairs of a question and a name. 37 names (45%) appeared in only one of the four answers, 23 in two, 12 in three and 11 in all four.
Measured on 12 October 2026 (36 answers, collected in about 50 minutes) and counted again on 2 November 2026 from the experiment record. One app, one day, small numbers.
What we asked
9 questions of the kind a Japanese buyer asks when choosing an AI-visibility tool, for example 「LLMO 診断ツール おすすめ」 ("which LLMO diagnostic tools do you recommend"). None names a company. Each question was asked 4 times in a row, and for each answer we noted the tool names (service names) it gave. The names were picked out of the answer text by a program (an extractor).
The numbers
The 83 pairs of a question and a name, by how many of the four answers carried the name:
| In how many of the 4 answers | Names |
|---|---|
| 1 | 37 |
| 2 | 23 |
| 3 | 12 |
| all 4 | 11 |
- Some questions were steadier than others. For one question only 5 names appeared at all, 1 of them once. For another, 14 names appeared, 10 of them in a single answer.
- A check on "are three asks enough": take the names in at least half of the answers as a "core". The core made from all four answers and the core made from any three of the four (a name in at least 2 of the 3) were the same in 10 of 36 cases (28%). By question: 0% for 3 questions, 25% for 2 and 50% for 4. Because the core comes from the same four answers, this is a ceiling for "three asks are enough", not an estimate.
Basis and limits
- One app, one day, small numbers: ChatGPT only, four answers per question, 9 questions, about 50 minutes on 12 October 2026. Nothing here says how Gemini, Claude or DeepSeek vary.
- The names were picked out by a program, which flags some for review; it may have merged or split a name wrongly.
- We do not know how many asks are enough. The "10 of 36" figure is a ceiling, not an estimate.
- The questions were about AI-visibility tools, a crowded field where many names compete for a few slots. A question with one obvious answer may be steadier.
- We did not check whether a given answer was right. This is about how much the list moves, not about which list is correct.
- The count was made by a one-off script over the record of 99 runs in that file (using the 4 ChatGPT runs of each of the 9 questions); no new run was made.
- Sources:
docs/evidence/2026-10-13-n3-cited-domains-v2.json(theservicesfield of each run) andknowledge/claims/named-services-vary-across-repeat-runs.md(level: observed in our lab). - What would show more: the same nine questions, more asks per question, on other days and in other apps.