Open any AI SEO visibility dashboard and you will see tidy numbers: a share of voice, an average position, a trend line. They look as authoritative as a rank tracker. They are not. The same question asked twice often produces different answers, and a dashboard built on single runs can turn randomness into a story. Sarkar SEO reads AI visibility data the way a careful analyst reads a small survey: useful in aggregate, dangerous in the details.
What the consistency research found
The most cited evidence comes from SparkToro and Gumshoe.ai. SparkToro had 600 volunteers run 12 prompts a total of 2,961 times across tools including ChatGPT, Claude and Google’s AI. The headline result: the odds of getting the same brand list twice from ChatGPT were less than 1 in 100, and the same list in the same order about 1 in 1,000. geotoolboxgetpassionfruit
Two other findings matter more than the headline. First, despite wildly different phrasing, AI tools still returned similar brand sets for the same underlying intent. In the headphones test, Bose, Sony, Sennheiser and Apple appeared in 55 to 77% of 994 answers. Second, the authors concluded that measuring brand visibility, the percentage of responses in which a brand appears, may be more reliable than ranking position. Position tracking, in one summary’s words, is mostly noise.
Fishkin himself softened his starting suspicion. He wrote that his hypothesis that AI answer lists are so random that tracking is entirely useless was likely wrong. So the lesson is not “tracking is fake.” It is “single readings are unreliable, patterns across many runs are not.”
Variation exists between features, too
Randomness is not the only source of disagreement. Ahrefs reported that Google’s AI Mode and AI Overviews cite different sources 87% of the time for the same query. That measures a different thing from run-to-run variation: same platform and prompt versus different features. For reporting, it means AI Overviews and AI Mode should be tracked separately, not blended into one “Google AI” score.
Why two tools give two scores
Run the same brand through two trackers and you may see 40% in one and 12% in the other. A tracker guide lists the usual causes: different prompt sets, different sampling depths, and different engines and model versions. This comes from a vendor blog, but the reasoning is sound. Different inputs produce different outputs, and neither number is necessarily wrong.
The practical rule that follows is to pick one method and keep it stable, so changes over time reflect your brand and not your measuring stick.
A fairness note: Gumshoe.ai is itself a tracking startup, and most of my coverage of the study is secondary. The study’s design, with a large volunteer pool and many runs, is a strength. Its scope, 12 prompts mostly about product recommendations, is a limit.
Define what you are counting
Dashboards often blur three different things. One analysis argues you should separate citations (a URL linked), mentions (a brand named) and recommendations (a brand actively suggested). They have different business meaning. A link to your guide is not the same as being named the best option, and a mention in a warning is not a win.
Track a fourth item as well: accuracy. If an assistant describes your pricing wrongly or places you in the wrong category, visibility is working against you.
A measurement method you can defend
- Group prompts by intent. Choose 30 to 50 real customer questions, sorted into categories such as comparison, problem-solving and “best for” recommendations.
- Run each prompt many times. SparkToro ran prompts 60 to 100 times before patterns became clear, which is impractical for most teams. The same guide suggests that thirty runs a month per prompt per platform gives you a trend line. ecommercefastlaneecommercefastlane
- Report visibility rate. For each intent group, record the share of runs in which your brand appears, separated into citation, mention and recommendation.
- Report ranges, not points. Show last month’s and this month’s rate with a note on sample size.
- Keep conditions constant. Same engine, same prompts, same location, and web search switched on or off consistently.
How much change is real?
Here is simple binomial arithmetic that most dashboards skip. If your brand truly appears in about 30% of answers and you test 20 runs, the measured rate will often land anywhere from roughly 15% to 45% by chance alone. Comparing two months of 20 runs each, a swing smaller than about 25 to 30 percentage points is within noise. With 100 runs each, the threshold drops to about 13 points.
That is my own calculation, not a figure from the studies, and it assumes independent runs. Use it as a rule of thumb: do not celebrate or panic over small moves on small samples.
Questions to ask any tracking vendor
From the tracker-selection guidance and the study’s open questions:
- How many times is each prompt run, and how often?
- Does it use the real consumer app or the API? The guide notes that SparkToro lists API-versus-app fidelity as an open research question.
- Is web search enabled on each platform?
- Which engines and which Google features are covered separately?
- Can it connect visibility to traffic and leads?
Sarkar SEO would add one more: can it export raw responses, so you can read what the assistants actually said?
Anchor reports to business outcomes
Visibility is an input, not a result. Pair it with signals that are harder to randomize: branded search demand, referral sessions from AI tools, leads that mention an assistant, and sales conversations. When tracker data and business data move together, you have evidence. When they diverge, trust the business data and investigate.
Common reporting mistakes
Reporting a rank. “We are number two in ChatGPT” describes one run on one day.
Switching tools mid-year. The score will change even if nothing else did.
Crediting one update. A content change and a visibility jump in the same month may be coincidence.
Ignoring text. Numbers cannot tell you that an assistant is describing you incorrectly.
