Why ChatGPT gives a different answer every time, and what that means for your business

If you’ve ever asked ChatGPT “who is the best solar installer near me?”, taken a screenshot, then asked again and got a different list, you’ve seen what makes AI search so hard to measure: the answer keeps changing.

That’s normal. It’s how these systems work, and it changes how you should check where you stand.

What the research shows

In late 2025, SparkToro and Gumshoe collected 2,961 AI responses to identical prompts. They found less than a 1 in 100 chance that two runs of the same question return the same list of brands, and less than 1 in 1,000 that they return the same list in the same order.

Separately, Mike Sonders ran 1,200 ChatGPT sessions across 12 software-buying questions. On average, around 44 different brands appeared across 100 runs of the same question, and only about five appeared in 80% or more of them.

Those studies looked at brands and software rather than local service businesses. But the same mechanisms sit behind every answer, and there is no reason to expect “best roofer in Parramatta” to be more stable than “best software for X”.

Why AI answers change

There is no single cause. Several things combine:

• Language models choose words with some randomness. The same prompt can lead the model down a slightly different path, and a different path can name different businesses.
• Search-connected answers depend on what was retrieved. When ChatGPT, Perplexity or Google’s AI features look things up before answering, the pages they pull in can differ between runs, and the answer follows the pages.
• Location and context matter. Platforms such as ChatGPT use approximate location, and the wording of the question changes the answer. “Best dentist near me” and “best cosmetic dentist in Bondi” are different questions, and the same question asked from different places can be answered differently.
• The systems themselves keep changing. Models, retrieval and ranking are updated constantly, often without notice.

None of these is something a business can switch off. What a business can do is stop treating one answer as the truth.

What this means for your business

A screenshot is not evidence. If you, a staff member or an agency checked ChatGPT once and saw a competitor, that tells you what happened once. It does not tell you how often it happens, and it does not tell you whether anything you change later has made a difference.

A single visibility score can mislead you. If a tool runs each question once and reports a percentage, the number may move from week to week for no reason other than chance. That is especially awkward for agencies, who have to explain the movement to a client.

Being mentioned is not the same as being recommended. An answer that lists three installers and then says “other options include…” has treated those businesses differently. If you count both as “visible”, you hide the gap that matters.

How to measure it properly

Good measurement in AI search looks a lot like good measurement anywhere else: repeat the observation, keep the raw data, and do not read meaning into small differences.

1. Pick the questions your customers actually ask. Write them the way a real person would, including the location and the decision they are making.
2. Run each one many times, on each platform that matters. One run is a sample. A pattern across many runs is a measurement.
3. Keep every answer. Store the prompt, the platform, the date and the full response, so any number can be checked later.
4. Record three outcomes, not one. For each answer, note which businesses were recommended, which were only mentioned, and whether you were absent.
5. Compare like with like over time. When you change something, keep asking the same questions the same way, and compare the before and after periods rather than two single answers.

You can do a small version of this by hand. Our guide on how to check if ChatGPT recommends your business walks through it step by step.

How Ansoar handles it

Ansoar was built around this problem, for businesses where one sale is worth thousands and the agencies that market them. It runs every tracked search once a day on the three AI platforms you choose, from ChatGPT, Perplexity, Gemini, Google AI Mode and Google AI Overviews, keeps every answer, and reports how often you were recommended, mentioned or absent for each customer decision. It doesn’t show trends until there are enough answers to back them up, and when you make a change, it compares the weeks before and after on the same searches.

It’s messy, so check it more than once.

Ben Rosenbaum is the founder of Ansoar. More about Ben.