Before You Buy an AI Visibility Tool, Audit Your Own Prompts

An AI visibility score can look convincing until you ask who picked the prompts. Give a vendor twelve friendly questions and one run per question, and the resulting number may tell you more about the test than about your brand.

The AEO Glossary for People Who Learned Search on Google Ads helped with the vocabulary. This is the second look: with more tools selling AI visibility scores, I want you to check whether those numbers hold up before you pay for one. The way to do it is familiar if you have ever audited a Google Ads search-terms report. Start with buyer intent, control the conditions, and inspect the answers before you trust the score.

The question for this test: Which buyer prompts surface your brand, in which engine, as a mention, a linked citation or a recommendation? And where does a competitor appear instead? I used to answer the paid-search version by pulling what people actually typed and sorting by spend and conversions. The waste showed itself. AI answers deserve the same treatment: cause first, then the score.

Setup: build prompts from buyer intent, not vendor suggestions

Export the last 90 days of your Google Ads search-terms report and your Search Console queries. Sort the paid terms by spend and conversions rather than clicks, then pull 25 to 50 buyer questions from the language you find. Sales calls and support tickets can fill in the questions your query data misses; that is also the basis of this prompt-inventory method.

Translate the intent into a question someone could put to an AI engine. For example, a search term such as “emergency plumber sunday surcharge” becomes “Who charges a Sunday surcharge for emergency plumbing in Austin?” Do not invent clever prompts to make the test interesting. Test the questions that already cost you money.

Split the list into four buckets, and record how many prompts land in each:

• Category: “best CRM for a 20-person agency”
• Comparison: “HubSpot vs Pipedrive for follow-up”
• Problem-first: “how do I stop no-shows for quotes”
• Brand: “does [your brand] do weekend installs”

Aim for at least 30 prompts so a handful of answers cannot dominate the result. For a larger manual benchmark, one method starts with 100 high-intent prompts and scores mentions against total answers. Write two phrasings for your ten highest-spend intents. Near-identical wording can produce different brand lists; that difference is data, not duplication. Keep the buckets visible, or a score driven by brand-name prompts may masquerade as strength on category questions.

Controls: lock what you can before the first run

Run the prompts logged out, with location and language set to your market and no previous chat history in the session. Start a new chat for each ChatGPT and Gemini run; use a clean browser profile for AI Overviews and AI Mode. Record the date, engine, exact prompt and location setting with every answer. A weekly tracking method for 20 to 30 queries uses the same discipline, including fields for Overview presence, citations and competitors. If you do not record the conditions, you cannot compare the answers later.

Run each prompt at least three times in each engine. The same question can return different brand lists or orders, a problem illustrated by reports of unstable results for identical prompts. One platform comparison describes 12 sessions per prompt across four engines. You do not need to replicate that workload by hand. Three runs across ChatGPT, Gemini, AI Overviews and AI Mode let you see whether a placement holds or moves. Add Perplexity if your buyers use it.

My expectation is that some answers will change between runs because the output is not a fixed search-results page. That is why one screenshot cannot carry the finding. One run is an anecdote; repeated runs expose the variance.

Measurement: give every answer its own row

Use one spreadsheet row per run, not per prompt. I would group the fields so a reader can audit both the answer and the conditions:

• Conditions: date, engine, exact prompt text and location setting.
• Brand outcome: mentioned yes or no; cited with a link yes or no; recommended yes or no; position in the answer.
• Other evidence: competitors named and the source URLs the answer used.
• Paid placement: ad present, advertiser name and whether the ad points to you or a competitor.

The distinctions matter. The manual measurement method separates mention rate from citation rate, share of voice and sentiment; another gap-diagnosis method distinguishes a brand appearance from a cited page. A mention with no link is not a citation. A citation halfway down an answer that recommends someone else is not a recommendation.

Keep paid placement separate from the organic answer. If an AI Overview has a Sponsored label above, beside or inside the answer, record the ad in its own fields. Do not count an appearance only in the ad as an organic mention. If the same run also mentions your brand in the organic answer, keep that organic mention; the ad does not erase it.

Say you run 40 prompts three times in AI Overviews and your brand appears organically in 36 of the 120 answers. Your organic mention rate for that set is 30%, whether or not an ad appeared on some of those runs. Count ad appearances separately. Mixing the two is how a media buy starts looking like content authority.

Read the result by engine, intent and repeatability

Calculate the mention rate for the prompt set, then break it down by engine and prompt bucket. Put the denominator beside every percentage. Thirty-six organic mentions in 120 AI Overview runs means 30% for those prompts, in that engine, on those dates. It does not mean your brand owns 30% of AI search. Then count linked citations and recommendations separately, and inspect which prompts change across the three runs.

The split is the diagnosis. If you are mentioned but never cited, you may have an answerability or third-party source gap rather than a total coverage gap, as this gap framework explains. If you are neither mentioned nor cited and no page on your site answers the prompt, put it on the coverage list. Report the number, denominator and engine breakdown together. Broad category questions and narrow buyer questions can tell different stories, so do not roll them into one reassuring average and stop reading.

The first pattern I look for is competitor cited, me absent. Suppose eight comparison prompts cite two rivals and a review site, while your site never appears in the source lists. The engine found an answer without you. Rank those gaps by frequency and evaluation-stage intent before you write anything. Brand-containing prompts deserve their own attention too: in one set of client interviews, 44% of real user prompts already contained brand names. A missing citation on a buyer’s comparison question is a work order. A missing citation on trivia is not.

The second pattern is page exists, engine skips it. Check whether the page is indexed and retrievable before you rewrite it. Then look at whether the answer relies on a third-party source instead. I treat this like tracking in paid search: eligibility first, creative second. Weekly logging with freshness checks helps distinguish a persistent omission from an answer that changes as pages and citations change. If the page cannot be retrieved, another paragraph will not fix the mechanical problem.

The third pattern is plain noise. A result that flips within one engine across three runs is not a mandate to change strategy. A competitor appearing across all three runs in two engines, with the same source domains recurring, deserves investigation. I would give a change at least eight weeks before a strategy decision and keep checking the same 20 to 30 queries weekly. I learned the paid-search version of this lesson moving bids after one bad afternoon. Only repeatable patterns earn a work order.

Turn a recurring gap into a page, source or access task

For prompts where a competitor keeps appearing, work backward from the source list. Record three things for each gap:

1. The page on your site that should answer the prompt, if one exists.
2. The passage that could answer it directly in two or three sentences.
3. The third-party page the engine cited instead.

I used to tell clients to rewrite the whole page. I was wrong. If the answer is buried, a tight answer block near the top is a better first edit than a 2,000-word rewrite. If no page exists, add one to the build list and rank it by spend and evaluation-stage intent. If a page exists but the engine relies on a review site or publisher, investigate retrieval and corroboration before adding more copy. Missing From AI Answers? Diagnose the Four Gaps Before You Write More Content is the follow-up for separating those jobs.

Rerun the set after you ship fixes: same prompts, engines and location setting. Put the new answers beside the baseline and compare the same questions in the same tool against competitors and against your earlier result. I would check every two weeks and give a new passage at least two cycles before judging it. One good answer proves no more than one bad one. A gap without an owner and a retest date is a note, not a plan.

Vendor check: make the tool reproduce your sheet

You now have a baseline a vendor should be able to explain. A tool worth paying for should show what your spreadsheet shows, only faster and across more prompts. My pass-or-fail list is short:

• Full prompt-level answers, with the exact prompt text visible, not just a score.
• Repeat runs per prompt, so variance is measured rather than hidden.
• Mentions, linked citations and recommendations counted separately.
• Paid AI Overview placements flagged apart from organic appearances.
• Source URLs exposed, so you can see which pages occupy the answer.

If its dashboard disagrees with your hand test on the same prompts, ask why before you pay. Different conditions may explain the difference. An unexplained score does not.

Then ask the vendor three questions:

1. “Show me the prompt list. What changed in it last month?” Two tools can disagree because their prompt sets, engines and time windows differ.
2. “How do you record an answer that mentions me but recommends a competitor, and where is the paid flag?” Those distinctions should survive the move from sheet to dashboard.
3. “What do you measure alongside visibility?” AI answers move, and visibility alone does not establish demand. I would check branded search and direct traffic beside the visibility trend.

A vendor who cannot answer those questions has not beaten your spreadsheet.

Stop doing it by hand when the audit becomes the job

Forty prompts, three runs each, four engines: 480 answers to inspect for mentions, links, recommendations, competitors, source URLs and ads. Repeat that every two weeks and you have stapled a part-time job to your marketing role. I know the 1am negative-keyword-mining version of that job. The manual test earns its keep as a baseline and a check on vendor claims. It stops scaling when you need weekly per-engine trends, a list growing past 50 prompts, or source tracking that catches when a review site replaces you.

That is where continuous measurement can earn its cost: repeat sampling of the same questions across the same engines, with paid placements apart from organic answers. groas tracks prompts, AI answers, citations and share of voice. I would still check any tool’s output against the sheet before treating its trend as a decision.

If your brand holds comparison prompts across runs and earns citations to pages you own, do not buy a dashboard just to admire the result. Keep the sheet as a fortnightly check and put the money into the pages and corroboration behind it. If the test comes back thin, with competitors cited and your sources missing, you have a ranked work list instead of a score to worry about. Either result changes the purchase: you know which prompts matter and what a tool must reproduce. For the longer version of the buying questions, read An Open Letter to the Owner About to Buy an AI Visibility Dashboard. Run the test first. Make any tool beat it.