AI visibility tracking needs a fixed prompt panel

AI visibility tracking becomes useful only when you treat it as a repeated measurement, not a collection of flattering screenshots. Ask a chatbot one question, see your company named, and you have learned almost nothing. The same system may omit you on the next run, recommend you for the wrong buyer, or describe a limitation that your product no longer has.

A workable program needs a fixed panel of prompts, a written collection protocol, separate scores for mentions and sentiment, and a monthly comparison against its own baseline. You can run that program with a spreadsheet, a few model subscriptions, and disciplined review. Expensive monitoring software may save labor later, but it cannot repair a weak measurement design.

The aim is not to manufacture a single visibility number for a board slide. It is to find where an answer engine includes your brand, what claim it attaches to you, which source seems to support that claim, and whether the pattern changes after you improve public evidence. That is close enough to an evaluation program to demand the same care you would give a product test.

AI visibility is a sample, not a ranking

An LLM answer is a sampled response under particular conditions, so one answer is not your position in a stable index. Search rank and answer presence are different measurements. A search engine can report impressions for a query because it owns the result page and its logs. A brand usually has no equivalent first party log for every answer generated across outside assistants.

Google Search Console documentation defines its Performance report around links shown in Google Search, News, and Discover, with query, click, impression, and position data. That scope is useful, but it does not make Search Console a proxy for inclusion in an LLM answer. Treating organic rank as AI visibility hides the exact gap you are trying to measure.

The unit of observation should be one response to one prompt on one named model, collected at one time under recorded settings. Your monthly metric then summarizes a set of observations. This distinction sounds fussy until a founder sends a screenshot of a positive mention and asks why the dashboard says visibility fell. The screenshot and dashboard can both be accurate because they describe different samples.

Four kinds of variance matter. The model provider may replace or tune a model. The same model may sample different wording. Search or browsing features may retrieve different pages. Account context, location, language, and prior conversation may change the answer. You cannot remove all of that variance, but you can record the conditions and repeat enough observations to see whether a change persists.

This is why I argue against the popular advice to track a short list of dream prompts manually whenever someone remembers. It feels authentic because a person is asking natural questions. It produces a biased diary, not a trend. Keep room for exploratory prompts, but never mix their results into the fixed panel score.

The prompt panel must mirror actual buying questions

A useful prompt panel covers the decisions a buyer makes, including questions that do not contain your brand name. Branded prompts test what systems say when users already know you. Category and comparison prompts test whether you enter the conversation before the buyer has chosen a shortlist. Both matter, and combining them without labels ruins the diagnosis.

Start with customer language from sales calls, support tickets, win and loss notes, community discussions, and search queries. Remove prompts that merely repeat your preferred positioning. A founder may call a product an orchestration layer while buyers ask for a way to reduce manual handoffs. The panel should use the buyer's words.

Give each prompt one intent label and one audience label. A compact panel usually needs these intent families:

• Discovery prompts asking for ways to solve a problem
• Comparison prompts asking which options fit stated constraints
• Validation prompts asking about strengths, limits, safety, or suitability
• Branded prompts asking what the company does or who it serves
• Replacement prompts asking for alternatives to an incumbent or current process

Do not add twenty phrasings of the same question to make the sample look large. Coverage beats volume. Include materially different constraints such as company size, regulated data, implementation capacity, geography, or budget only when they change the buying decision. If a constraint would not change a good recommendation, it does not deserve another row.

Add a comparison set, but do not turn the panel into a league table. Record which other brands appear, their position, and the recommendation language under the same prompts. This shows whether your mention rate fell because you disappeared or because the answer began naming more options. It also reveals category confusion, such as a model comparing you with an adjacent product that solves a different job.

Freeze that comparison set for reporting, even when a new competitor attracts attention. Add newcomers in a current cohort and retain the original cohort for trend. Acquisitions, renames, and discontinued products need dated alias records rather than deleted columns. Report your share of substantive mentions only when every eligible answer uses the same competitor definition. A ratio with a changing denominator is an editorial choice disguised as performance.

Do not score a competitor more harshly than your own brand. The same alias rules, prominence rubric, and conditional recommendation labels must apply to every company in the comparison set. If reviewers know which content campaign your team just launched, blind the monthly quality check to that context. The goal is consistent coding, not a chart that rewards internal enthusiasm.

For each core intent, write a natural prompt, a controlled variant, and an adversarial variant. The natural prompt sounds like a buyer. The controlled variant changes one factor, such as team size. The adversarial variant asks about drawbacks, complaints, or reasons not to choose the category. That third version catches a common blind spot: teams record praise and never measure whether models repeat damaging or outdated claims.

Keep the panel stable once the baseline begins. You can add a new cohort of prompts, but label its first month and exclude it from like for like trend lines until it has a prior observation. Editing old prompts in place rewrites the ruler while you measure.

Freeze the collection protocol before the first run

The collection protocol must make two months comparable. Write it before anyone starts copying answers, then place it beside the data rather than in one analyst's memory. The protocol should name the model, model version when exposed, interface, date, language, country or region, browsing state, account state, conversation state, and repetitions per prompt.

Run every panel prompt in a fresh conversation. Prior turns can prime a model with your category or brand and create visibility that a new user would not see. Use a clean account or the same defined account state each month. If personalization cannot be disabled, record that limitation and keep the account constant.

Decide whether you are measuring answers with browsing, answers without browsing, or both. These are separate cohorts because they use different evidence paths. Browsing may favor recent indexed pages, while a closed model may rely on material represented in its training process. Never average those cohorts into one unexplained number.

Repeat each observation. Three runs per prompt and model is a defensible minimum for a small manual panel, although it does not make the result statistically certain. The point is to stop one lucky completion from becoming a strategy. If cost or time forces one run, report the result as a spot check and do not draw a smooth trend line through it.

Record the full answer immediately. Do not save only the sentence containing your brand. The surrounding recommendation may say that your product suits large enterprises when you serve small teams, or may rank you below options that meet a condition you fail. Context changes the commercial meaning of a mention.

OpenAI's Evals documentation describes an evaluation as testing criteria plus a data source configuration, with runs that can compare models and settings. You do not need its API to borrow that discipline. Your prompt panel is the data set, your rubric is the grader, and each monthly collection is an evaluation run. The useful lesson is repeatability, not a particular vendor feature.

Store raw answers before scoring them

The raw response is evidence, and the score is an interpretation that you may revise. Store both. If you keep only scores, a rubric change forces you to discard history or pretend old and new labels mean the same thing.

A plain CSV can carry the first year of a modest program. Use one row per response, not one row per prompt with twelve answer columns. The long format makes filtering, pivoting, and quality checks much easier. This header is enough to start:

run_month,prompt_id,prompt_version,intent,audience,model,model_version,browsing,region,repetition,response_text,brand_mentioned,mention_position,recommendation_status,sentiment,citation_status,source_domains,reviewer,review_note

Keep response_text unchanged apart from safe CSV escaping. Store source domains only when the interface displays or names a source. Never infer a citation from a phrase that sounds familiar. citation_status can distinguish a linked source, a named source without a link, and no visible source.

Assign durable prompt IDs such as DISC-001 and keep the prompt text in a separate prompt table. That table should include prompt_version, active_from, active_to, the exact text, intent, audience, and the reason for any revision. When wording changes, create a version rather than overwriting the cell.

Access control matters because answers may include prompt text derived from private sales notes or reviewer comments about competitors. Keep the measurement data in the same governed workspace as other marketing research. Do not paste confidential customer details into public assistants to make prompts realistic. Replace them with meaningful but invented constraints.

Save a small audit sample each month. A second reviewer should rescore a selection without seeing the first score, then the two reviewers should discuss disagreements and clarify the rubric. You are checking whether humans apply the labels consistently, not trying to declare one reviewer correct by seniority.

Mentions, position, and recommendation are separate scores

A brand mention answers only whether the name appeared. It does not tell you whether the model recommended the brand, buried it in a long list, used it as a warning, or named it only because the prompt did. Score these dimensions separately and retain their counts before creating any index.

Use a binary mention field first: 1 if the answer refers unambiguously to the brand or product, otherwise 0. Maintain an alias table for spelling variants, former names, and product names. Do not use loose substring matching when a short brand name is also an ordinary word. Automated detection should flag candidates, while a person resolves ambiguous cases.

Position captures prominence. A practical ordinal rubric is 3 for the primary recommendation or first substantive option, 2 for another substantive option with meaningful explanation, 1 for a passing mention, and 0 for no mention. If the prompt includes your brand by name, report position but exclude that row from unprompted visibility. A system repeating a name from the question did not discover you.

Recommendation status captures direction: recommended, conditionally recommended, neutral description, discouraged, or not applicable. Conditional recommendations deserve their own label because they often contain the best product intelligence. An answer such as suitable for technical teams but difficult for a nontechnical operator tells you more than a generic positive sentence.

Calculate transparent rates rather than hiding everything behind a weighted score:

mention_rate = unbranded_answers_with_brand / all_unbranded_answers
recommendation_rate = recommended_or_conditional / all_answers_with_brand
average_position = sum(position_score) / all_answers_with_brand
citation_rate = answers_with_visible_support / all_answers_with_brand

You may add a composite for executive reporting, but publish its formula and show the component metrics beside it. Weights turn business opinion into arithmetic. A team that values shortlist entry may weight mentions, while a risk sensitive buyer may care more about discouraging claims. There is no universal set of weights, and a vendor's proprietary visibility score should not be treated as one.

Sentiment scoring needs a claim and a target

Sentiment should describe what the answer says about your brand for the stated use case, not whether the prose sounds pleasant. LLM answers often use polite language around a commercially negative judgment. “A capable option, although it lacks the controls required here” is negative for that buyer even though the adjective is positive.

Use a five label scale: positive, mixed positive, neutral, mixed negative, and negative. Add not enough evidence when the answer only lists a name. Forcing a passing mention into neutral creates false precision. Require reviewers to copy the shortest passage that supports the label into the note field.

Score the brand target separately from category sentiment. A model may criticize an entire product category and mention your brand as the best exception, or praise the category while warning against your implementation. The label must answer a written question: “For the audience and task in this prompt, does the response increase or decrease the chance that the reader would consider this brand?”

Sarcasm is uncommon in recommendation answers, but qualification is everywhere. Train reviewers on qualifiers such as only suitable when, lacks, requires, compared with, and may be excessive for. Do not convert every caveat into negative sentiment. A correct limitation attached to a positive fit is usually mixed positive. A false limitation needs a factual accuracy flag in addition to sentiment because the remedy differs.

Keep factual accuracy outside the sentiment score. Reviewers can verify claims against current public product material and label them accurate, outdated, unsupported, or disputed. An accurate negative claim may require product work. An outdated claim may require clearer current documentation and stronger distribution. An unsupported claim may come from low quality third party pages or model inference. One sentiment number cannot tell those cases apart.

Automatic sentiment classification can reduce reviewer labor after you have a labeled sample and a tight rubric. It should return the label, the supporting passage, and a confidence or review flag. Test it against human decisions whenever the grader model or rubric changes. A classifier that agrees with itself is not proof that it understands your commercial context.

Monthly trends need fixed cohorts and visible denominators

A monthly trend is credible only when each point uses the same eligible prompts, models, settings, and repetition policy. Report counts beside percentages. A 50 percent mention rate means something different when it comes from four answers than when it comes from four hundred.

Create a frozen baseline cohort and a current coverage cohort. The baseline cohort contains prompts and model configurations that remain comparable across the full period. The current cohort includes new models and prompts that matter now. Show both. The baseline gives continuity, while the current cohort stops the program from becoming a museum of old interfaces.

When a provider retires a model, close that cohort and start a successor cohort with an overlap month if possible. Do not splice the successor onto the old line as though nothing changed. Mark model releases, browsing changes, prompt revisions, major site releases, and public communication campaigns as annotations. An annotation supplies context; it does not prove causation.

Use a trailing average only as a secondary view. Smoothing can make a chart readable, but it can also hide a sharp failure after a model update. Keep the raw monthly value visible. Compare both absolute counts and rate changes, and inspect which prompt families moved.

A simple monthly operating rhythm works:

1. Freeze the active panel and configuration for the run.
2. Collect all repetitions within a short window and save raw answers.
3. Run automated candidate detection, then complete human review.
4. Recheck a sample, resolve rubric disagreements, and lock the month.
5. Compare cohorts, inspect changed claims, and assign evidence work.

Do not rerun only the answers you dislike. If a collection failure affects a prompt, rerun every repetition for that prompt and mark the replacement. Selective reruns quietly turn measurement into reputation management.

A change in the chart is a lead, not a cause

Visibility movement tells you where to investigate, not why it happened. Model changes, source retrieval, competitor activity, news, your own pages, and ordinary sampling can move the result together. The honest output of the monthly review is a short list of supported hypotheses and checks.

Start at the row level. Find prompt families with changed mention or recommendation status, then compare the actual answer passages. Look for a repeated new claim, a newly cited domain, the disappearance of a source, or a change limited to one model. Aggregate charts often hide that one obsolete comparison article has started appearing across several answers.

Separate durable movement from noise with replication. If an important prompt flips, run the prespecified additional sample for the full affected prompt cohort, not just the one row. Check whether the change appears across models and whether the supporting sources changed. Record this as an investigation run so it does not contaminate the scheduled monthly series.

NIST's work on AI test, evaluation, validation, and verification stresses reliable measurement and evaluation methods. The useful application here is restraint: define the measurement target, document the test conditions, and avoid claims that the sample cannot support. A brand dashboard does not become scientific because it has decimals.

Avoid declaring that a content update caused a visibility gain merely because it came first. You can strengthen the case if the changed answer cites that page, repeats its corrected claim, moves only in browsing enabled cohorts, and persists across later runs. Even then, call it evidence consistent with an effect unless you ran a design that isolates the intervention.

The awkward result may be no meaningful movement for months. That does not make the program worthless. It may show that your public evidence is not reaching the sources models use, that the category rarely appears in these answers, or that your chosen prompts do not represent enough demand. Each explanation leads to a different decision, which is why raw evidence and cohort labels matter.

Improve the evidence that answers can use

The proper response to weak visibility is better public evidence and a clearer product, not pages stuffed with prompt variants. Answer systems need material that states who the product fits, what it does, where it falls short, how it compares, and what proves those claims. Buyers need the same material, so useful work does not depend on guessing a model's preferences.

Map each repeated answer problem to an owner. Missing brand mentions for a category may expose weak category association. Wrong audience descriptions may point to vague positioning. Outdated limitations may require refreshed documentation and third party corrections. Accurate criticism may belong in the product backlog rather than the content calendar.

Write pages around complete decisions, not isolated keywords. A comparison should define the buyer, constraints, tradeoffs, and evidence. A case study should explain the starting condition, intervention, result, and limits without invented precision. Product documentation should use stable names and explicit statements that another writer can verify.

Do not publish dozens of nearly identical question pages. That tactic is popular because the prompt panel provides a ready list of headings and content production is cheap. It is wrong because it fragments evidence, creates contradictions, and gives readers thin pages. Group related questions into the strongest page that can answer them fully.

The monthly review should end with a small evidence queue: claim to correct, source or page involved, owner, planned change, and the cohort expected to respond. Keep product changes separate from communications changes. If a false answer says a feature is absent, first confirm whether public documentation is unclear before asking engineering to build something that already exists.

For founders who want this tied to a wider engineering and AI operating plan, oleg.is offers a fixed five day Team & AI Audit that identifies at least $50,000 per year in savings or costs nothing. The audit is broader than brand monitoring, so it fits when the prompt panel exposes unclear ownership, manual research, and scattered AI work across the team.

Cheap tools work until review labor becomes the bottleneck

A spreadsheet and manual collection are enough when the panel is small, the monthly cadence is manageable, and a decision maker still reads the answers. Early automation should handle copying, normalization, alias checks, pivots, and change flags. It should not conceal raw evidence or replace judgment before the rubric settles.

Estimate cost in reviewer hours, subscription access, and maintenance rather than API tokens alone. Manual interfaces may include product features that differ from API access. Automated collection can violate interface terms or trigger controls if done through brittle browser scripts. Use supported APIs when you automate, and record that the API experience may not match the consumer product you originally measured.

A paid platform starts to earn its place when you need many regions, languages, models, daily collection, role based review, source extraction, or alerts tied to a governed workflow. Evaluate it with the same discipline as the metric. Ask whether you can export raw responses, preserve prompt versions, see model and run metadata, change scoring rules, and recompute history. If the vendor exposes only a proprietary score, you are renting a conclusion.

Keep a manual benchmark even after automation. Each month, compare a sample of stored responses and scores with what a reviewer sees in the intended interface. This catches parsing errors, silent model substitutions, missing citations, and sentiment labels that drift after a grader change.

Your first deliverable should be a baseline that another person can reproduce, not a polished dashboard. Freeze the prompt panel, collect repeated answers, retain every raw response, and publish component rates with denominators. After three scheduled runs, you will know which collection steps waste time and which labels drive decisions. Automate those steps. A tool should reduce the cost of a sound method; it cannot supply the method after the fact.