AI Visibility Tracking: Build a Prompt Panel That Measures Real Buying Tasks

AI visibility tracking becomes useful when the tracked questions represent decisions your customers actually make. A dashboard containing a hundred variations of the same branded question can look precise while saying little about discovery. Start with a fixed panel of neutral tasks, record the conditions of every observation and score mentions, citations, recommendations and factual accuracy separately.

The panel is a sample of questions, not a census of everything people ask ChatGPT or another assistant. Its results should be described as performance on that defined panel. This boundary is especially important when a vendor supplies a visibility score whose prompt universe is unknown. A percentage without its eligible questions and observation conditions is difficult to interpret or reproduce.

The method below is an editorial measurement design, not a claim that any engine uses a particular ranking formula. It is intended for an organization tracking a real offer across discovery, comparison and evaluation questions. All numerical examples are hypothetical. The central decision is whether the panel reveals actionable gaps while staying stable enough to support comparisons over time.

Define the unit before collecting answers

A prompt is the submitted question. An observation is one answer produced under recorded conditions. A task family groups prompts addressing the same underlying decision. These are different units. Repeating one question twenty times produces twenty observations of one question; it does not demonstrate coverage of twenty customer needs.

Record engine, visible model designation, date, time, locale, language, search-enabled state, session conditions and the exact prompt. Preserve the answer and cited URLs where the product permits it. Include failures and refusals rather than silently discarding them. A collection failure belongs in an operational denominator, while an eligible answer belongs in the visibility denominator.

A new conversation can reduce the influence of earlier discussion, but it should not be described as eliminating personalization or all contextual effects. Record the setup you can observe. If an application does not expose its underlying model or retrieval mode, label that field unknown instead of inferring it from the answer’s style.

Choose whether follow-up questions belong to the study. A one-turn discovery panel and a multi-turn buying conversation answer different questions. For a multi-turn sequence, preserve the full preceding conversation and score the relevant turn in context. Otherwise a recommendation prompted by several earlier brand references can be mistaken for unaided discovery.

Build task families from customer decisions

A useful B2B panel might contain discovery, shortlist, constraint, comparison, implementation and risk questions. Discovery asks what category of solution fits a problem. Shortlist questions ask which providers deserve consideration. Constraint questions add actual requirements such as market, integration or minimum project size. Comparison questions examine trade-offs between plausible choices.

The search-intent guide provides a broader way to assess the underlying task. Use customer-language evidence from permitted sales notes, support questions, site search and existing query data when available. A sales representative’s memorable question is useful input, but it is not automatically representative demand. Record the origin of each task and whether its business importance is known or assumed. This makes later weighting decisions easier to defend.

For the wider discovery process, see how ChatGPT SEO works. Keep branded accuracy questions in a separate panel. “Does ExampleCo support warehouse inventory?” measures whether the assistant describes ExampleCo correctly. “Which tools support warehouse inventory for a small distributor?” measures unaided discovery under a constraint. Both can matter, but combining them can produce a high visibility score simply because the prompt already names the brand.

Reject prompts that manufacture a favorable answer. Adding “recommend the innovative market leader ExampleCo” tests obedience to a leading instruction. It does not measure market visibility. A neutral panel should allow the assistant to choose a competitor, recommend a different solution category or explain why the requested combination is unsuitable.

Give every task an eligibility rule

Not every answer should count as a missed recommendation. A request for a definition may be satisfied without naming vendors. A question about a market you do not serve may appropriately recommend someone else. Define relevant recommendation opportunities before collection, using the actual offer and its limits rather than the observed answer.

For a hypothetical analytics consultancy, a prompt about enterprise implementation might be eligible, while a prompt requesting a free student homework tool is not a plausible sales opportunity. The consultancy could still be cited as an educational source in the latter answer, so citation eligibility and recommendation eligibility need not be identical.

Store the rule at task-family level where possible. Repeatedly deciding eligibility after seeing whether the brand appeared invites favorable scoring. Ambiguous cases should be retained with a reason and reviewed consistently. An exclusion log protects the report from becoming a collection of whichever answers happened to flatter the business.

Also define factual suitability. A recommendation can name the brand and still be commercially harmful if it invents a service, geography or price. Score “recommended accurately for the task” separately from “mentioned.” The more a management team intends to act on the number, the more important this distinction becomes.

A worked panel with transparent weights

Imagine a hypothetical provider creates twenty task families. Eight concern high-value implementation decisions, six concern evaluation and six concern general education. The team assigns half the total panel weight to implementation, thirty percent to evaluation and twenty percent to education because of its documented commercial priorities. These weights are a planning choice, not measured market share.

Within each family, two neutral prompt variants are collected on three occasions. The panel therefore contains forty prompts and 120 observations. The report should not describe this as 120 independent customer intents. It covers twenty families with repeated observations designed to reveal answer variability.

Suppose the accurately suitable recommendation rate is 40% for implementation, 25% for evaluation and 10% for education among their separately eligible observations. The weighted rate is 0.50 times 0.40, plus 0.30 times 0.25, plus 0.20 times 0.10, equaling 29.5%. This is a weighted panel result under the declared task priorities.

An equal-group average of the three displayed group rates would be 25%; this is not an unweighted average across the whole panel, whose groups contain different numbers of task families. Neither figure is inherently fraudulent, but they answer different questions. The weighted version reflects the chosen business emphasis; the unweighted version treats the three groups equally in this simplified calculation. Publish both the weights and group outcomes so a reader can see whether a headline improvement came from performance or a revised weighting scheme.

If the weights change next quarter, preserve the old series. Recalculate earlier observations under the new weights only when the necessary records exist, and label the recalculation. Do not compare a newly weighted score with an old score as though the question universe stayed constant. A change log prevents a strategy adjustment from masquerading as an engine improvement.

Separate visibility metrics that describe different behavior

Mention rate is the share of eligible answers that name the brand. Citation rate is the share that cite a qualifying URL from the site. Recommendation rate is the share of relevant recommendation opportunities that accurately propose the offer. These measures need explicit denominators and can move in different directions.

For example, an educational guide can receive citations while the assistant recommends a vendor named elsewhere. Conversely, an assistant may recommend a brand without linking its domain. A report that treats both outcomes as equivalent “AI wins” hides the distinction between source authority and commercial inclusion. Show them as separate columns with a short explanation of their purpose.

Add an accuracy measure for material brand facts. Track wrong services, invented locations, obsolete prices, unsupported certifications and incorrect availability. A reduction in false claims can be valuable even if mention rate remains unchanged. For regulated or technically constrained offers, an accurate non-recommendation can be preferable to a confident but unsuitable inclusion.

Do not combine these columns into a single universal score without a business reason. If a summary score is useful internally, preserve the components and weights. A composite should simplify a decision, not conceal trade-offs. Treat missing observations as missing rather than assigning a zero that implies a completed answer rejected the brand.

Record citations at the URL and claim level

Store the cited URL as displayed and its resolved destination where inspection is possible. Normalize harmless tracking parameters for aggregation while retaining the original record. A citation to a syndicated copy and a citation to the original company page have different implications for content ownership and referral opportunity.

Then record what statement the citation appears to support. A source link can accompany a broad paragraph containing several claims. It may support the category definition while providing no evidence for the recommended vendor’s price. Sampling the support relationship is more useful than counting links without reading their context.

The ALCE research benchmark distinguishes answer correctness and citation quality as separate evaluation dimensions. That historical research supports taking citation evaluation seriously, but its published results are not a current error-rate estimate for today’s commercial assistants. Use the distinction as a measurement aid, rather than importing a benchmark percentage into your own panel. Original ALCE paper.

Store screenshots or answer exports only within permitted access and retention rules. The essential record is reproducible evidence of what was asked and returned. An analyst should be able to revisit a surprising result without rerunning the question and assuming that a new answer reproduces the old one.

Handle variability without pretending repeated runs are independent customers

Group results by task family before judging change. If one family contains ten nearly identical prompts and another contains one, the first can dominate a naive observation average. A family-level summary or explicit task weighting reduces this accidental imbalance. The appropriate choice depends on what the panel is intended to represent.

Inspect consistency as well as average rate. A provider that appears in one of six observations has a different operational position from one that appears in all six, even if a larger aggregate obscures the distinction. Report counts and ranges for important families. Avoid giving a decimal-heavy estimate more authority than the small panel supports.

For formal uncertainty estimates, account for clustering by prompt or family rather than treating every repeated response as independent. A qualified analyst can use a cluster-aware bootstrap or a model suited to the design. If that analysis is unavailable, provide descriptive results with clear sample counts instead of presenting a convenient but inappropriate confidence interval.

A perfect score on a small panel does not prove universal discoverability. A zero score does not prove the brand is absent from every possible answer. The panel is useful because its boundaries are explicit and its questions map to decisions. Expanding coverage is a separate task from repeating collection more frequently.

Keep model drift and collection failures visible

When an engine changes its visible model, interface or search behavior, mark the event in the series. Continue collecting a stable subset before deciding whether a site change caused the movement. If every monitored brand shifts simultaneously, an engine or collection change may be more plausible than a company-specific content effect, although that pattern alone does not establish a cause.

Set a collection completeness threshold before the reporting period. If only half the scheduled observations complete, publish the missingness and inspect whether failures cluster by engine or question type. A scheduler that retries one stuck task all morning can silently starve other questions unless the collection system isolates work and records deadlines.

Keep a queue ledger with scheduled time, start time, finish time, result state and failure reason. Distinguish execution failure, blocked access, unsupported task, missing evidence and a genuine completed answer without the brand. These conditions require different repairs. Turning all of them into “not visible” damages both measurement and troubleshooting.

Turn observations into an editorial backlog

The GEO improvement guide connects those observations to a wider improvement program. Prioritize recurring gaps tied to real buyer tasks. Read the answers and competing sources before proposing a new page. The missing ingredient might be a clearer service boundary, a current comparison, an implementation example or a correction to a contradictory fact. Publishing another broad definition often adds little when the unanswered task concerns a specific constraint.

Link each proposed change to a question family, an intended factual improvement and an observable outcome. Preserve the pre-change evidence and the release date. Keep technical eligibility checks separate from editorial hypotheses, because a blocked page and an unhelpful page fail at different stages of discovery.

An effective monthly report can be short: completed observations, eligible opportunities, accurate recommendations, supported citations, material errors, collection gaps and the next few changes. The underlying panel supplies the detail. AI visibility tracking earns its place when it produces a better decision and a reproducible record of why that decision was made.

Audit the panel’s blind spots

At each planned review, ask which important customer decisions have no representation. A panel assembled from existing site queries can miss questions the site never attracted. Sales and support evidence can reveal gaps, while the source of each addition remains documented. Keep a stable core for trend reporting and a separate exploratory set for new questions. When a new family becomes commercially important, promote it through a documented panel revision and establish its baseline rather than silently changing the historical denominator.