A new category of software promises to show where your brand ranks in ChatGPT, Gemini, Perplexity, and Google’s AI features. The dashboards look convincing, with positions, scores, and weekly charts. The research behind AI SEO answers raises a harder question: are those numbers measuring anything stable?
This guide is for anyone about to pay for AI visibility tracking, or already paying and unsure what to trust.
The Core Problem: Answers Change Every Time
Anyone who has asked an AI tool the same question twice has seen different wording. The brand-level effect is larger than most people expect. SparkToro, working with the AI tracking startup Gumshoe.ai, ran each prompt 60 to 100 times per platform and reported less than a 1-in-100 chance that ChatGPT or Google’s AI in Search would return the same list of brands across repeated runs of the same prompt. A summary of the same work says there was less than a 1 in 1,000 chance of getting the same list in the same order. searchenginejournalpropellernet
Two cautions belong beside those figures. The research was conducted in partnership with a company that sells AI tracking tools, and another summary notes it is not peer reviewed, with the authors disclosing a commercial interest in AI tracking. The authors also left open how many runs are needed for reliable data. searchenginejournalthrivestack
What Still Holds Up
Randomness does not mean chaos. The same report observed that, despite varied prompts, the AI tools returned brands from a relatively consistent consideration set. One summary adds that how often a brand appears across 60 to 100 runs showed statistical validity despite the randomization.
That distinction is the key to buying well. A single answer is a poor measurement. The share of answers in which you appear, across many runs, is a much better one. Position claims are the weakest of all. The data led Rand Fishkin, who led the study, to say, as quoted by one vendor blog, that any tool giving a “ranking position in AI” is not credible. That statement is a strong opinion, but it follows from the numbers.
How Many Runs Is Enough?
A follow-up analysis, reported by Search Engine Journal, proposed a stopping rule. In its 30 platform and topic tests, the number of answers needed to reach a stable result ranged from 33 to 94, counting only answers with citations. The same report cautions that the same number of answers on two engines does not buy the same confidence, and that the work is not a finished, peer-reviewed study.
Take the practical lesson rather than the exact figures: dozens of runs per prompt, per engine, before you read anything into a percentage. A tool that checks each prompt once a week and draws a line chart is mostly plotting noise.
Engines Disagree With Each Other
Even within one company, features differ. The SEJ report mentions Ahrefs data showing that Google’s AI Mode and AI Overviews cite different sources 87% of the time for the same query. That means a “Google AI” score that blends the two hides real differences. Ask any vendor whether they report each surface separately.
Seven Questions to Ask Before You Buy
- How many times is each prompt run per period? Single runs are not enough. Ask for the number and the reasoning.
- Are results reported as ranges or percentages, rather than rank positions? Prefer share of answers over claimed “positions.”
- Which surfaces are covered, and are they separated? AI Overviews, AI Mode, ChatGPT, Gemini, Perplexity, and Claude behave differently.
- Is data collected through consumer interfaces or APIs? The SparkToro authors left open whether API calls yield the same variation as manual prompts, so the answer affects how far results can be trusted.
- Who chooses the prompts? Prompts should reflect how your customers phrase questions, not generic category terms.
- Can you export raw answers? If you cannot read the underlying responses, you cannot audit the score.
- What is the vendor’s own interest? Several firms publishing visibility research also sell tools. That does not invalidate the work, but it should shape how you read it.
A Low-Cost Alternative You Can Run Yourself
Many businesses do not need a platform to start. Build a small, disciplined sampling routine instead:
- Choose 15 to 25 prompts that mirror real customer questions, including a few that do not mention your brand.
- Run each prompt at least 10 times per engine at first, in fresh sessions, and log the answers in a spreadsheet.
- Record three things per run: whether you were mentioned, whether you were linked, and which competitors appeared.
- Report the percentage of runs, not the result of any single run.
- Repeat every month and watch trend direction, not decimal points.
Ten runs will not reach the stability suggested by the larger tests, so treat this as directional. It is still far more honest than a screenshot.
Pair Sampled Data With Harder Evidence
Because sampled answers are noisy, anchor your reporting to numbers that do not wobble. Search Console impressions and clicks, branded search trends, direct enquiries, and “how did you hear about us” responses are all steadier. One agency writing about this problem describes AI response tracking as directional samples rather than absolutes, and that is a sensible stance for any team.
At Sarkar SEO, AI visibility numbers are always shown beside enquiry and revenue data, so a client never has to make a decision from one noisy chart. Sarkar SEO also suggests setting a minimum sample size before anyone is allowed to call a change “real.”
How to Use What You Learn
Tracking is only useful if it changes what you do. When a prompt shows you absent, check three things: does a page exist that answers it directly, can the relevant crawlers reach that page, and is your information consistent across the web? When you appear sometimes but not reliably, look at whether competitors are cited more often for the same prompt and study what their pages offer that yours do not.
If you want an outside view on whether a vendor’s dashboard stands up to scrutiny, Sarkar SEO can review the methodology alongside your own analytics before you commit budget.
