Why do AI answers change every time? Because ChatGPT, Gemini, Perplexity and Google's AI features write each answer from scratch. They pick words by probability, often run a live web search, and may read different sources on each try. So the same question can name different businesses on two runs. One screenshot tells you little; you need to ask the same question several times and read the result with a margin of error.
Here is a situation many owners know. On Monday you ask ChatGPT for "a good family dentist in South Austin" and your practice shows up second. On Tuesday you ask again and you are gone. On Wednesday your office manager asks on her phone and sees you fourth. Which answer is real? This guide explains why AI answers change every time, in plain language, and then shows how to measure your AI search visibility reliably despite the noise. That measurement is the backbone of AEO, answer engine optimization, which you may also see called GEO (generative engine optimization). If you want to see first how AI reads your website today, run the free AI visibility analysis. It lists what is missing and how to fix each item.

Why AI answers change every time: five causes
In classic Google search, two people searching the same phrase from the same place usually see a similar list of links. AI answers work differently. There is no stored list waiting to be shown; the answer is composed at the moment the question is asked. Five separate forces drive the variation.
1. Answers are sampled, not looked up. A language model chooses each next word from a range of likely options instead of a single fixed one. Ask the same question twice and the first sentence may match, but the answers tend to drift apart as they go on. This is how the models are built, not a glitch. Microsoft's Azure OpenAI documentation states plainly that asking the same question several times will likely produce different responses, and that even with its reproducibility setting, determinism is not guaranteed. In the consumer chat apps your customers use, those settings are not exposed at all.
2. Answers lean on a live web search. When someone asks for a business recommendation, AI assistants often search the web in that moment. Google's own documentation says AI Overviews and AI Mode may use a "query fan-out" technique, issuing several related searches across subtopics, and that because the two features may use different models and techniques, the responses and links they show will vary. Which sub-searches run and which pages get read can differ from one run to the next.
3. Engines and models get updated. ChatGPT, Gemini, Perplexity and Google's AI features change behind the scenes. The model changes, the search layer changes, the way sources are picked changes. Last month's answer can shift this month even if you touched nothing.
4. The person asking is different. Wording ("best" versus "affordable" versus "near me"), location, language, whether the user is signed in, and, where enabled, past chats or saved preferences can all shape the answer. What you see on your phone is not necessarily what a patient sees on theirs.
5. The sources themselves change. A competitor adds photos to its Google Business Profile, a new review lands on Yelp, a Healthgrades or Angi listing gets updated, a local news site publishes a roundup. The next time the AI searches, it sees the new information. This is not noise. It is real change, and it is worth tracking.
| Cause | What shifts | Your influence |
|---|---|---|
| Sampling | Who is named and in what order | None; handled by measuring with repeats |
| Live web search | Pages read and sources cited | Indirect; grows with strong, consistent sources |
| Engine updates | Overall behavior and source choice | None; monitor only |
| User and context | Answer by person and place | None; hold constant when measuring |
| Source updates | What is known about you and rivals | Direct; through your site, profiles and reviews |
The last row matters most. The first four causes are about measurement: you cannot control them, but you can account for them. The fifth is where your work happens. As the information on your website, your profiles and your reviews gets stronger and more consistent, the odds of being named go up. The variation never disappears, but the center of it moves your way.
Why this matters for a local business
Not knowing about the variation pushes owners toward bad calls in both directions.
- False alarm. One run leaves you out, so you rewrite the website, fire the agency or pull content. In reality you might be named in two runs out of three.
- False comfort. One run shows you first, so you assume the job is done. In reality only a few of the customers asking that question may ever see you.
- Effort in the wrong place. A single screenshot suggests a competitor has pulled ahead, so you start copying them. On the next run they may not appear at all.
- Reports you cannot check. A report that shows one screenshot labeled "you rank in ChatGPT" without saying when, how often and under what conditions the question was asked cannot be verified.
From the customer's side it is simpler. They ask once and see one answer. What matters to you is not what one answer said, but how many of the people asking that question end up seeing your name. You can only estimate that with repeated measurement. In AI search, visibility is not a position number. It is a probability.
Why a one-off test misleads: an example
Take an example family dental practice in a mid-size Texas city. The dentist asks ChatGPT "family dentist that takes new patients near me" on Monday morning and sees the practice in the top three. She sends the screenshot to her team. On Wednesday the front desk asks Gemini the same question and the practice is not mentioned. On Friday a new patient says, "I couldn't find you when I asked AI."
None of these contradict each other. Monday's result was one draw from a range of possible answers. Wednesday's was a different engine leaning on different sources. Friday's patient probably phrased the question differently and asked from another part of town. Three observations like these cannot measure visibility. The useful question is: if we ask the same question, under the same conditions, several times on each of four engines, how often is the practice named, and in what position?

Do not confuse this with wrong information. If the answer names you but with an old phone number or a closed address, that is not variation, it is a data problem. We cover the fix in wrong address or phone number in AI answers.
Three runs and a margin of error: how to measure answers that change
You cannot remove the variation, but you can measure it. The approach is simple: ask each question on each engine more than once and read the results together.
Why repeat? A single run is a coin toss. Three runs are enough to see how much the result moves. Named in all three, you are in a strong spot. Named in one, you are on the edge. Named in none, something is missing for that question.
Why the median? If three runs return positions 2, 4 and "not named," an average is awkward. The median, the middle value, gives a sturdier summary: here, 4th. One outlier does not drag it around much.
Why a margin of error? How far the three runs spread tells you how much to trust the number. Close runs mean a small margin. Runs far apart mean a large margin, and a one-week change deserves little weight. A "± X" next to a score gives the reader exactly that context.
Why four engines separately? ChatGPT, Gemini, Perplexity and Google's AI features draw on different sources. A business can be strong on one and absent on another. Rolling them into one number hides that; seeing them side by side shows where the gap is. For one engine in detail, see Perplexity rank tracking.
Which questions swing the most?
Not every question varies equally. The broader a question is, and the more plausible answers it has, the more the runs tend to differ. The narrower it is, and the more it depends on one clear fact, the steadier the answer. That tells you where to read results more cautiously.
| Question type | Example | Expected swing | How to read it |
|---|---|---|---|
| Broad recommendation | "Best restaurant in Chicago" | High; many candidates | Never trust a single run |
| Neighborhood plus service | "Pediatric dentist in Logan Square" | Medium | Look at the median of three runs |
| Specific need | "Vet open Sunday with parking" | Low to medium | If never named, a fact may be missing |
| Brand name question | "What time does X close" | Low | Errors here are data problems, not noise |
This table describes a general tendency, not a measured rate. The practical point: on broad questions, "sometimes named" is normal and the goal is to be named more often. On narrow brand questions, the answer should be right every time; if it is not, fix the information first.
How to run your own repeated test, step by step
You can measure this by hand. It takes time, but it is the best way to understand the method.

- Write your question list. Pick five to ten questions customers really ask: "service in neighborhood," "best service in city," "affordable service," "who handles a specific need." To build the list, use the approach in find the questions customers ask AI.
- Lock the wording. Keep the questions in a document and paste them every time. Even "good" versus "best" can change the answer.
- Use a clean session. Open a private window or turn off chat history and memory. Keep location and language fixed. Start a new chat for every run; asking again in the same thread lets the earlier answer act as context.
- Ask each question three times on each engine. With four engines and ten questions, that is 120 runs. To start small, use five questions and two engines.
- Log every run. Were you named, in what position, who came first, which sources were cited.
- Summarize. For each question, record the median and the gap between best and worst run. Across all questions, compute the share of questions that name you.
- Repeat weekly. Same weekday, similar time, same list. Judge the three- to four-week trend, not one week.
A simple log template:
| Question | Engine | Run 1 | Run 2 | Run 3 | Median | Sources cited |
|---|---|---|---|---|---|---|
| Family dentist South Austin | ChatGPT | Not named | 4 | 2 | 4 | Healthgrades, Yelp |
| Family dentist South Austin | Gemini | 3 | 3 | 2 | 3 | Google Business Profile |
| Emergency dentist open Saturday | Perplexity | Not named | Not named | Not named | None | Competitor site, directory |
The values are an example. The first two rows show "visible but moving." The third shows a real gap: missing in all three runs cannot be explained by variation. Something in the sources is missing for that question, perhaps Saturday hours that appear nowhere online.
Manual testing has limits. It is slow, error-prone and hard to keep up week after week. If your team is used to rank tracking, we compare the two approaches in rank tracker vs. AI visibility tracker.
How Welda handles it: three runs, a median and a margin
Welda runs the method above inside software, on a schedule. This section covers the three parts that bear directly on variation: the three runs on the questions screen, the open score formula, and the margin of error next to the score. First, the facts about availability. The Welda dashboard is in early access and is opening first to businesses in Turkey. Regular four-engine measurement is being added during early access, and the dashboard screens shown in demos use representative data. What anyone can use today is the free analysis, which works in English and German.
Questions screen: every question, every engine, three runs. In the dashboard, the questions your customers would ask are sent to ChatGPT, Gemini, Perplexity and Google AI three times each. Each row is a question, with a column per engine. Open a question and the answer detail shows the three runs side by side (for example "not named, 4, 2") with the median below. The decision this screen supports: is this question solid for me, on the edge, or missing? Edge questions are often the quickest wins, because the AI already knows you but does not pick you every time.
The answer detail also lists the sources each answer cited. Comparing a run that named you with one that skipped you is the most practical way to find where the variation comes from. If the run that named you leaned on one directory and the run that skipped you leaned on another, you have found the directory where you are missing.
An open score formula. The visibility score is not a black box. It is 100 times the sum of three parts: the share of questions that name you (weight 0.5), a position score for where you are named (0.3), and your share against competitors (0.2). When the score moves, you can see which part moved. If the number of questions naming you stays flat while your position drops, the problem is not invisibility; a competitor is being named ahead of you.

The margin next to the score. The further the three runs spread, the larger the ± value beside the score. The score itself is the average of the three runs, and the margin sits right next to it. If a weekly change stays inside the margin, it is too early to call it real. A change that exceeds the margin and holds for a few weeks is a trend. A "what changed this week" view shows which questions named you for the first time and which piece of work came before it, so you can tie a change to an action.
The rest of the dashboard builds on this measurement. The task list shows missing information and ready-to-use copy for edge questions; Welda drafts content and profile text, and publishing approval stays with you. The competitor and source views show which sources feed the variation.
The test inside the free analysis. The free analysis also asks five likely customer questions to an AI assistant that can search the web, and checks whether your business name appears. That is a one-time measurement. For all the reasons in this article, read it as a starting snapshot: if you are never named, fix the gaps on your site first; if you are named, it does not mean you will be named on every run.
How to read a score that moves
Once measurement is regular, a new question appears: should I react to this week's change? The chart below is for that.

- Note small changes inside the margin, but do not act. Look again next week.
- Trace a large change on one engine to that engine's sources. Its source choice may have changed; if nothing moved elsewhere, the issue is not your website.
- Trace a change across all four engines to yourself. Was a page removed? Did your profile change? Did your robots.txt start blocking AI crawlers?
New content usually takes weeks to show up in AI answers, so a flat score in the first week after a fix is normal. For realistic timelines, see how long AEO takes.
Limits, and what Welda does not do
Repeated measurement manages variation; it does not remove it. So it is worth being clear about what cannot be done.
- No one can promise a fixed position. Welda does not guarantee rankings or that you will "show up in ChatGPT." With answers generated fresh every time, such a promise would not even make sense.
- Measurement is not what one customer sees. Repeated runs estimate what happens under clean, fixed conditions. A customer with personal settings may see something different.
- The free analysis is a one-off. Its five-question test is a snapshot, not a trend, and it does not track rankings.
- Welda does not invent facts. If you are on the edge for a question because a fact is missing, such as hours or service area, Welda asks you instead of guessing.
- No fake reviews. There is no shortcut through the variation. Replying to negative reviews, directory accounts and press work stay with the business.
- Nothing sensitive is published without approval. Content always lives on your own website, and you approve what goes live.
Checklist: before you trust a result
When a screenshot lands on your desk, whether you took it or a vendor sent it, run it through these questions.

Use the same lens when you evaluate any AEO tool or service. Ask three things: How many times is each question asked? Are results shown per engine? Is there a margin of error or confidence range next to the score? If those answers are missing, you have no way to know how much the numbers can swing.
Frequently asked questions
I asked the same question twice and got different answers. Is something wrong?
No. AI answers are generated fresh each time and often rely on a live web search. Different answers are expected. What matters is how often you are named across several runs.
How many runs are enough?
Three runs are enough to see how much a result moves and to take a median. More runs shrink the margin of error but cost more time. In the Welda dashboard, each question is asked three times on each engine.
What does the margin of error mean?
It is a range based on how far the three runs spread. If the score is, say, 30 with a small margin, your real visibility is close to 30. With a large margin, do not read much into small weekly changes.
Is a private browser window enough?
It reduces personalization, but the answer is still generated fresh each time. Use a clean session and repeated runs together.
Why is the five-question test in the free analysis one-time only?
The free analysis exists to show the gaps on your website and where to start. The five-question test adds a starting snapshot. Regular, repeated, four-engine measurement happens in the Welda dashboard, which is in early access.
I show up on one engine but not another. Which one is right?
Both. Each engine reads different sources. Showing up on one and not the other points to a gap in the sources the second engine relies on. The cited sources list is the fastest way to find it.
Bottom line: why AI answers change every time, and what to do about it
The short reason why AI answers change every time is that every answer is written anew. That variation is permanent, and fighting it is pointless. The job is to build measurement around it: ask the same question under the same conditions several times, read engines separately, judge results against a margin of error, and base decisions on weekly trends instead of a single screenshot. The real work is strengthening your website, profiles and reviews so the center of the spread moves your way. To see your starting point, run the free AI visibility analysis. You get one report showing how AI reads your site, which fix would raise your score the most, and a ready-made fix package to apply.