Why AI answers differ every time — and how to measure your AI visibility honestly

Two people can ask an assistant the same question on the same day and be shown different brands. Neither answer is wrong, and neither is the real one — which changes what it means to ask whether you are "in" the AI answer.

The instinct, when AI answers start to matter commercially, is to check once. Someone types the obvious question into an assistant, sees the result, takes a screenshot, and the screenshot becomes the company's belief: we are mentioned, or we are not. It feels like checking a search ranking, and it is not. The screenshot is genuine, the belief built on it is not justified, and the reason is benign: the system is working exactly as designed.

Why do AI answers change every time you ask?

Because each answer is generated at the moment of the request rather than retrieved from a fixed list. The result depends on how the question is phrased, who is asking, where they are, which session they are in and what day it is. This is normal behaviour, not a fault: two people asking the same question on the same day can be shown different brands, and neither answer is the real one.

A classic search results page has some of this variation too — location and personalisation move rankings around — but the page is still a ranked list assembled from an index, and position three today is usually near position three tomorrow. A generated answer is a different kind of object. The words are composed fresh for each request, the sources consulted can differ from one request to the next, and small changes in phrasing can steer the answer towards a different set of pages entirely. Ask for "accounting software for small businesses" and then for "bookkeeping tools for a small company" and you have asked, as far as the answer is concerned, two different questions.

So a screenshot proves one instance, not a state. It is evidence that one request, from one session, phrased one way, on one day, produced that answer. That is worth knowing. It is not the same as knowing what your customers see, because each of them is running their own instance of the same experiment.

How do I check whether my business appears in AI answers?

Not with a single search, because one answer only proves one instance. The honest method is to fix a set of questions your customers would plausibly ask, ask them on a schedule, and count how often you appear. Presence in AI answers is a rate over repeated questions, not a yes/no state, so the useful number is "mentioned in six of the last ten runs", not a screenshot.

The discipline that makes this work is keeping the questions fixed. If the questions drift — reworded a little each time, or swapped for whatever seems topical — the numbers stop being comparable and the trend line means nothing. The questions should be written the way a customer would actually type them, in plain words, and then left alone. What varies is the answers; what stays constant is the instrument.

Once the measurement runs on a schedule, the question "are we in the AI answer?" gets replaced by better ones. Which of our questions do we appear in, and which never mention us? Is the rate moving? Did the change we made to the site show up in the numbers a month later? These are questions a business can act on, and none of them can be answered by checking once.

What does it mean when an AI answer recommends a competitor instead of you?

It is the most useful finding a measurement can produce, because it names which business is actually winning the question in the AI channel. That is often a different company from the ones above you in the classic search results, because answers draw on different signals than the ranked list. Knowing who is recommended in your place turns "we are absent" into something you can study and respond to.

This is worth sitting with, because it is the opposite of how most people read an absence. "We are not mentioned" feels like a dead end — a zero with nothing attached. But an answer to a commercial question is rarely empty. When it does not recommend you, it usually recommends someone, and that someone is a fact about the channel: a named site that the answer treated as the source or the suggestion for exactly the question your customer asked.

The competitor list this produces frequently surprises. The businesses contesting your keywords in classic search are visible to you; you have watched them for years. The sites being cited in generated answers can be a different set — a specialist blog, a comparison page, a competitor you considered too small to track — because being citable is a different property from ranking. A measurement that only reports your own presence throws this half of the finding away. The absence, with a name attached, is the finding.

Can a tool tell me for certain that I am absent from AI answers?

No, and a tool that claims certainty is overstating what it measured. Sometimes an AI feature does not render for a given request, and sometimes the answer arrives in a form that cannot be read reliably. The honest label for those runs is unknown. A measurement that quietly counts unknown as "not mentioned" will report you as absent more often than you are, so ask how a tool handles the runs it could not read.

Both failure modes are ordinary. AI features on search results pages do not appear for every query on every request — the same search can produce an AI overview one time and none the next, which is itself a form of the variability this article is about. And when an answer does appear, it is a block of generated content whose format shifts; a check that parses it will sometimes fail to parse it. Neither situation says anything about your visibility. It says the instrument could not take a reading this time.

The reason to insist on the three-way distinction — present, absent, unknown — is what each one licenses you to do. "Absent" justifies action: studying the competitor who was named, improving the page that should have answered. "Unknown" justifies nothing except waiting for more runs. A tool that folds unknown into no will send you off fixing a problem that may not exist, and it will also make your trend line lie, because a week of unreadable answers looks identical to a week of lost visibility.

Is appearing in Google's AI features the same as appearing in ChatGPT?

No. Google's AI features and the chat assistants are separate channels with separate retrieval and separate results. Being cited in one says nothing about the other, in either direction. If both channels matter to your business, both have to be measured — as a presence rate per question per surface rather than one combined score.

The distinction matters because the channels fail independently. A site can be well represented in Google's AI features — which draw on Google's index and crawling — while being invisible to a chat assistant whose retrieval crawler is blocked by a robots.txt rule or a firewall that Googlebot sails past. The reverse happens too. A single "AI visibility score" averages these into a number that hides exactly the difference you need to see, so the honest report is a grid: this question, this surface, this rate.

What actually moves the needle

Measurement tells you where you stand; it does not by itself change anything. What changes the rate is the same short list that governs whether generated answers can use you as a source at all.

• Be reachable by the retrieval crawlers. An assistant can only cite pages it can fetch. A robots.txt rule, a security plugin or a CDN setting can silently exclude you from the AI channel while Google keeps working — checking what your site actually tells AI crawlers is the first move, because everything else is wasted while the crawlers are turned away at the door.
• Answer the question that is actually asked, in plain words. Generated answers are assembled from pages that address the question directly. A page whose heading is the customer's question and whose first paragraph answers it is easy to draw from; a page that circles the topic in industry vocabulary is not. The fixed questions in your measurement double as a content brief: each question with a low rate is a page that does not yet exist or does not yet answer plainly.
• Be the kind of source answers cite. Specific, verifiable, well-structured pages get cited; vague ones get paraphrased without attribution or skipped. The longer treatment of this is in how to get cited by AI search.

What to do this week

Write down five to ten questions your customers would plausibly ask an assistant, in their words rather than yours. Ask them once across the surfaces you care about and record what came back — including the answers you could not read, recorded as unknown, and the names of whoever was recommended when you were not. Then put the same questions on a schedule, because the single run you just did is one instance, and everything useful lives in the rate.

TrustCtrl runs this measurement as part of its site checks: the same fixed questions asked on a schedule across Google's AI features and the chat assistants, each run recorded as present, absent or unknown, with the sites that were cited or recommended named alongside. It sits next to the plumbing that decides whether you can appear at all — whether AI crawlers can reach you, whether your content survives without JavaScript — so the report says not only where you stand but what is standing in the way.