Build the query library before you build the dashboard

Someone on your team asked ChatGPT about your category last week, screenshotted the answer, and now the screenshot is in a slide deck driving decisions. Next month someone else will ask a slightly different question, get a different answer, and the deck will update. Nothing in that loop is a measurement. It is two anecdotes with a slide template.

The instinct behind it is right: the answers matter. What is missing is the instrument. In software, nobody trusts a benchmark whose test cases change every run. AI answer measurement deserves the same respect, and the part of the instrument everyone skips is the most boring one: the list of questions.

What is a query library?

A query library is a versioned set of locked questions that represent how real buyers actually question your category, run repeatedly across AI platforms so results are comparable over time. AI visibility, in one sentence, is how those platforms find, describe, cite, compare, and recommend your company when a buyer asks. It borrows a principle evaluation engineering settled long ago: define the test set first. OpenAI's evals guide defines evals as testing model outputs against criteria you specify, against a fixed dataset with ground-truth expectations. Anthropic's evaluation guidance says to design tests that “mirror your real-world task distribution” and to include the edge cases. Your buyers' questions are that task distribution. The library is your eval set.

The five query families

One family of questions cannot see all five layers of visibility. This is the canvas I use: five families, each mapped to a buyer situation and to the layer it exercises. The five layers themselves are covered in the companion piece on measuring AI visibility.

Write the families from real buyer situations: sales-call questions, lost-deal notes, search queries, analyst questions. Keep the questions you expect to lose. A library built only from questions you win is a brochure with extra steps.

Lock the wording, version the changes

Comparability is the entire point, and wording is where it dies quietly. “Best AI visibility consultant” and “top AI visibility firms” are different questions to a model. So the rule is mechanical: every query has an ID, exact locked wording, and a version. Changing a word is a version change, recorded with a reason and a date, and runs across a version break are never compared silently.

Paraphrases still matter, because buyers do not use your locked wording. Handle them the way a lab handles conditions: as a separate cohort, a group that runs and reports on its own. Three or four paraphrases of a critical query, run as their own group, tell you whether an answer is stable or fragile across phrasings. Mixing them into the main set as interchangeable would blur both signals.

The variance is not hypothetical. In one vendor panel of 100,000+ prompt responses across 100+ brands from spring 2026, a preprint by Pratyush Kumar of Ranqo, first-run appearance rates ran 73% for household names, 44% for mid-market, and 11% for niche brands, and sentiment framing flipped about 6.7 times more often than mention presence. Treat the exact numbers as illustrative, one vendor's panel in a preprint. The stable lesson is the shape: answers vary a lot run to run, which is exactly why unstructured testing produces confident nonsense.

Record the full run

Every run of every query gets the same metadata, and the metadata is what turns answers into a dataset:

• Query ID, version, and family
• Date and time
• Platform, model, and version
• Whether the model was allowed to search the web (grounding mode)
• Locale, if the platform localizes
• The raw answer, preserved before any scoring
• Retrieved sources and asserted citations, kept separate
• Non-success states: refusal, error, timeout, empty retrieval

Here is what one locked entry and one run of it look like, with made-up values, so the shape is copyable:

That record is what makes retesting meaningful. NIST's AI Risk Management Framework treats Measure as one of its four named functions for a reason: measurement is a discipline with its own design decisions, not a byproduct of whatever a dashboard happened to log. Decide the record first. Then any dashboard, including a spreadsheet, can display it.

Cadence and retirement

Two more rules complete the instrument. First, cadence: pick a rerun rhythm you can keep, monthly for most companies, and hold it. A trend needs at least three comparable points before it deserves a decision, because single runs are samples.

Second, retirement: a query leaves the library the same way it entered, deliberately and on the record. When the category vocabulary moves, add the new phrasing as a new versioned query, mark the old one retired with a date, and keep its history. An immutable library slowly measures a market that no longer exists. An undocumented one never measured anything.

Own the library itself, in your own files, wherever you can export and inspect it. Platforms churn: OpenAI is retiring its hosted Evals platform on November 30, 2026, per its own documentation, and Google's AI search guidance tells site owners to be wary of third-party tools claiming internal metrics. The questions that define your measurement should not live or die with someone else's product decisions.

How I run this

My assessments run on locked prompt and query libraries in a production research engine I helped architect and build at Trinzik, the Austin AI company I co-founded; Trinzik operates it for clients. Queries are locked and versioned, paraphrase cohorts run separately, every run records model, date, grounding state, raw output, retrieved sources, and failures, and results normalize into records built for retesting. The findings ride on the citation gate covered in the companion piece on grounding. None of that requires my tooling to start. The canvas above, twenty real buyer questions, and a spreadsheet with the full run record will teach you more in an afternoon than a quarter of screenshots.

Questions worth asking next

Sources

1. Anthropic, "Create strong empirical evaluations," test and evaluation documentation. https://platform.claude.com/docs/en/docs/test-and-evaluate/develop-tests
2. OpenAI, Evals guide (fixed datasets, graders; hosted Evals platform retirement dated November 30, 2026). https://developers.openai.com/api/docs/guides/evals
3. NIST, AI Risk Management Framework (Govern, Map, Measure, Manage). https://www.nist.gov/itl/ai-risk-management-framework
4. Google Search Central, "AI features and your website" guidance. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
5. Pratyush Kumar (Ranqo), "Generative Engine Optimization at Scale," arXiv preprint 2606.20065, June 2026. https://arxiv.org/abs/2606.20065