AI Visibility Rankings Are Mostly Statistical Noise
A new preprint from IQRush researcher Ron Sielinski, covered in depth by Search Engine Journal, makes a case that should worry anyone paying for an AI visibility tracker: a single reading from ChatGPT, Gemini, or Perplexity is basically a coin flip dressed up as data.
Here’s the core problem. Generative engines don’t return the same answer twice. Ask the same question repeatedly and you’ll get a different mix of cited sources almost every time, because these models are built with intentional randomness baked into how they generate a response. So when your dashboard shows “Brand A: 12% citation share, Brand B: 9% citation share,” that’s one snapshot — not a fact. The actual gap between the two brands might be real, or it might completely disappear on the next run.
The research lays out two conditions that must both be true before a ranking can be trusted:
- The order has to stop changing as you collect more answers.
- The gap between the top competitors has to be bigger than the margin of error.
Across 30 platform-and-topic combinations tested, it took anywhere from 33 to 94 citation-bearing answers before both conditions were reliably met — and in three cases, even 125 questions weren’t enough to separate the top sites on SearchGPT. In other words, there’s no universal “sample size” that works for every platform or topic, and the clean, single number your dashboard hands you might be a warning sign rather than reassurance.
There’s a second wrinkle: not all platforms need the same amount of data. Gemini tends to stack multiple citations onto the same few sites within one answer, so those citations carry less independent information. SearchGPT spreads citations more thinly across more sources, so each answer tells you more — but a chunk of its queries return no citations at all, which means your effective sample is smaller than your question count suggests.
The practical takeaway: if your AI visibility tool pulls data once and shows you a tidy percentage, treat that number with real skepticism. A three-point jump in citation share after a content update might just be normal noise between runs, not proof your optimization worked. You need repeated measurements — before and after, more than once each — before you can honestly claim a win.
Your Rank Number and Your AI Citation Number Aren’t the Same Kind of Number
The second piece, Duane Forrester’s column for Search Engine Journal, tackles a different but related mistake: teams that put their Google rank and their AI citation rate side by side in a report and treat them as comparable metrics.
They’re not. A traditional search index matches the literal string you typed against a pile of documents. A language model does something entirely different — it interprets your prompt, infers what you actually meant, and then writes its own shorter retrieval queries to go fetch supporting sources. Research cited in the piece found the average typed prompt runs around 23 words, but the actual query the model sends off to be matched is closer to four or five words, and often the model fires off more than one of these mini-queries per prompt.
That means three transformations happen between what a user types and what your dashboard reports: the model paraphrases the prompt, generates its own search queries, and then applies its own judgment about which sources deserve a citation. None of that is visible in the number you’re staring at.
This has a strange side effect: query phrasing style alone can flip your whole visibility profile. A team that writes tracked queries as tight, keyword-style noun phrases (the old SEO habit) will often look weak on both surfaces — those phrases are too competitive for search rank and too thin for a model to confidently cite. A team that writes tracked queries as full, conversational questions will often look strong on both — long, specific phrasing is easier to rank for and gives the model enough context to cite confidently. Two companies with genuinely identical real-world visibility can produce opposite-looking reports purely because of how their queries were phrased.
The article’s guardrail: never read a search rank number without search volume next to it, because a top ranking on a phrase nobody searches is a hollow win. But that same trick doesn’t transfer to AI citations — there’s no reliable “prompt volume” data anywhere. The best substitute is tracking citation frequency across a repeated prompt set over time, read as a directional signal, not a precise demand figure.
What This Means for Your AI Visibility Strategy
Both studies point in the same direction: AI visibility data is directional, not exact, and it takes discipline to read it correctly. Here’s how we’d translate that into an actual plan.
1. Stop trusting single-snapshot reports. If your tool (or your agency) reports a citation share from one pull, ask for the sampling methodology. A trustworthy tracker should tell you when it doesn’t have enough data yet, not just print a confident-looking number every time.
2. Measure before-and-after changes more than once. A single before/after comparison after a content update can’t separate a genuine gain from ordinary noise. Run the check repeatedly, on both sides of the change, before you report a lift to a client or a boss.
3. Build your tracked prompt set the way real users talk, not the way you’d type into Google. Since long, specific, conversational phrasing tends to perform better on both rank and citation, your keyword research for AI visibility should lean heavily into full questions and intent-rich phrases — not three-word head terms.
4. Treat rank and citation as two different KPIs, not one blended score. Report them separately, with their own context (search volume for rank, repeated-sample frequency for citation), instead of stacking them into a single “visibility score” that hides which system is actually doing what.
5. Focus on structure and clarity your content can be cited from. Because the model is authoring its own retrieval queries and judging what’s citation-worthy, content that answers a specific question clearly, in a self-contained passage, is easier for a model to lift and cite than content that requires piecing together context from multiple paragraphs. This overlaps heavily with the fundamentals we cover in how RAG helps prevent LLM hallucinations — grounded, structured, source-clear content wins on both the RAG side and the citation side.
6. Don’t abandon rank volume as a guardrail — just don’t force it onto AI metrics. Keep using search volume to sanity-check your organic rankings. For AI citations, use repeated-run frequency instead, and be upfront that it’s a trend line, not a demand count.
Why This Matters More as AI Search Grows
None of this is a reason to ignore AI visibility — quite the opposite. It’s a reason to measure it properly. The businesses that win in generative search over the next few years won’t be the ones chasing a single flattering percentage on a dashboard; they’ll be the ones who understand the mechanics well enough to tell a real signal from statistical noise. That’s the same discipline that separates good SEO from guesswork, just applied to a newer, messier surface.
If you want a deeper look at how we approach this for clients, our piece on AI content optimization for ranking higher walks through the content-structure side of the equation, and our full blog has more on how AI agents, RAG, and generative search are reshaping how businesses get found online.
Want an AI visibility audit that actually accounts for statistical noise instead of hiding it?
and we’ll show you where your brand really stands across ChatGPT, Gemini, and Perplexity — sampled properly, not glanced at once.
