Choosing an AI-search agency requires measuring the provider’s method before buying its result. The platform surface is variable, observations are sample-dependent, and a citation or answer appearance is not the same as demand or revenue. Build a ledger that distinguishes evidence, intervention, external conditions, ownership, and decision value.
Define the appointment hypothesis
Write what the provider is expected to change: page answerability, ordinary Search foundations, query coverage, content evidence, sales enablement, or observation quality. Record audience, market, language, query panel, platform, period, budget, owner, and stop rule.
Build the AI Search Provider Measurement Ledger
| Field | Measurement | Limitation | | — | — | — | | Query panel | exact questions, audience, market, language | sample is not total demand | | Baseline | page, index, ordinary result, answer observation | platform state changes | | Exposure | answer, citation, link, ordinary result | observation is not reach | | Intervention | page, source, structure, process, date | contribution may be shared | | Evidence | capture, export, page version, source | quality depends on repeatability | | Outcome | qualified action, opportunity, revenue | downstream definitions are local | | Ownership | account, content, data, report, decision | portability protects continuity | | Limit | unknowns, alternatives, non-guarantees | prevents overclaiming |
Google’s AI features guidance says eligibility does not guarantee serving or citation. Use that as a measurement boundary. A provider should report what it observed, not transform eligibility into a guaranteed outcome.
Measure the baseline and panel quality
Require the exact query, date, location, language, device, logged-in state, answer surface, cited URLs, capture method, and reason the query belongs in the panel. Check whether the panel reflects buyers or only easy questions selected after a result appears.
Repeat a small sample with the same protocol. Record cited, linked, ordinary result, not observed, not eligible, and unknown. Do not report a single citation percentage without the denominator and observation window.
Score the panel for relevance, repeatability, coverage of the intended buyer, source quality, and decision usefulness. Keep the score descriptive, not a universal benchmark. If the provider changes the panel after an observation, record the change and start a new comparison.
Add a confidence note for each observation: repeatable, directional, disputed, or unknown. The note should state what would raise or lower confidence. Do not turn a small panel into a precise market share.
Add a confidence note for each observation: repeatable, directional, disputed, or unknown. The note should state what would raise or lower confidence. Do not turn a small panel into a precise market share.
Measure the intervention and alternatives
Record which pages, sources, authors, links, structured data, or technical changes the provider controls. Add client changes, platform changes, seasonality, demand, brand activity, and competitor movement as alternative explanations. A provider can improve answerability without causing every later click or deal.
Google’s people-first content guidance is a useful baseline for useful, original content. Measure whether the work improves the reader’s task, evidence, and scope, not just the number of generated pages.
Record the intervention as a versioned change: page, source, author, structure, internal link, technical fix, or observation method. Assign an owner and review date. Keep a sample page before and after so a reviewer can understand what was actually changed.
Measure whether the page is useful to the intended reader and whether the organisation can maintain it. Include subject-matter review, content freshness, technical dependency, legal review, and update owner. A change that cannot be maintained is not a durable improvement.
Measure whether the page is useful to the intended reader and whether the organisation can maintain it. Include subject-matter review, content freshness, technical dependency, legal review, and update owner. A change that cannot be maintained is not a durable improvement.
Measure claims and ownership
The FTC’s advertising FAQ explains that advertising claims need evidence and that agencies can bear responsibility for misleading claims. Register every material promise, source, date, denominator, owner, and expiry. Mark visibility, traffic, pipeline, and revenue as different claims.
Test account and data portability before approval. Confirm client access to domain, analytics, content, query panel, captures, reports, prompts, source files, and history. A proprietary score without raw evidence cannot support a renewal or dispute.
Measure provider capacity as well: owner hours, subject-matter input, technical dependency, review queue, and time to correct an error. A plan that produces recommendations faster than the client can validate them is not ready to scale.
Record the cost of measurement and correction. Include research, captures, content, engineering, review, and reporting time. Compare the proposed phase with the decision it is meant to improve, not with a visibility number alone.
Record the cost of measurement and correction. Include research, captures, content, engineering, review, and reporting time. Compare the proposed phase with the decision it is meant to improve, not with a visibility number alone.
Use a decision threshold
Approve a bounded phase when the panel is relevant, baseline repeatable, intervention explicit, limitations honest, owner named, and decision value clear. Buy evidence repair when the sample is weak. Reject or rewrite an outcome promise when it depends on platform behaviour the provider cannot control. Hold when the data is not portable.
Review the ledger at the first checkpoint and before renewal. Keep the baseline, captures, page versions, changes, exceptions, and decision together. Measurement is successful when the owner can explain what changed, what was observed, what was inferred, and what remains unknown.
Set explicit statuses: continue, change, repair evidence, hold, or stop. Use the same statuses for the provider’s report and the owner’s decision so activity cannot silently become approval.
Keep the status history and reasons; do not overwrite a prior decision when the panel or platform changes.
Keep the status history and reasons; do not overwrite a prior decision when the panel or platform changes.
How did this article land?
Choose one reaction. You can change it anytime.
