There’s a moment in most AI feature projects where someone asks whether the latest change made things better, and the room goes quiet. Not because nobody cares — because nobody can answer. The demo looks good. It looked good last week too, with different code.
This is the position a lot of teams are in. The feature is live, people use it, and the team is making changes based on vibes and complaints. It’s not negligence. It’s that the obvious answer — build a test set with correct answers and measure against it — doesn’t survive contact with the problem.
Why ground truth usually doesn’t exist#
For a classifier, ground truth is tractable. Someone labels a few thousand examples, you hold some out, and you have a number.
For most AI features shipped into products, it isn’t:
There’s no single correct output. If the feature summarizes a document or answers a question from a corpus, there are many acceptable answers and many unacceptable ones, and the boundary is not crisp. You can’t diff against an expected string.
Labelling requires the expertise you’re trying to scale. If the domain expert who’d have to write the correct answers is the bottleneck the feature exists to relieve, then building a large labelled set costs roughly what the feature was supposed to save.
The distribution moves. Your corpus changes, user behavior changes, and a test set built in March quietly stops representing what the system sees in September. A stale evaluation set is worse than none, because it produces confident numbers about a world that no longer exists.
Correctness depends on context you don’t have at label time. The same answer can be right for one user and wrong for another, depending on what they already knew and what they were trying to do.
So the honest starting position is: you will not get a clean accuracy number, and pretending otherwise is how teams end up optimizing a metric that doesn’t correspond to user experience.
What to build instead#
The goal shifts from measuring quality absolutely to detecting changes in quality reliably. That’s a weaker claim, and it’s achievable, and it’s most of what you actually need — because nearly every real question is comparative. Did this change help? Is it getting worse? Should we roll back?
Three components, in increasing order of cost and decreasing order of how often you can run them.
1. A small curated regression set#
Build 50–200 examples by hand. Not a representative sample — a deliberately adversarial one. Include:
- The queries that broke it before
- Edge cases you know are hard
- A few easy ones, as a canary for catastrophic regressions
- Cases where the correct behavior is to decline
For each, record what a good response must contain and what it must not. Not a gold string — a set of assertions. “Must cite a document from the correct section.” “Must not state a figure that doesn’t appear in the source.” “Must decline.”
This is cheap to build, cheap to run, and catches the class of regression that matters most: the change that broke something that used to work. It will not tell you how good the system is. It will tell you, reliably, when you’ve made it worse.
The discipline that makes this work is adding to it every time something goes wrong in production. A regression set that grows from real failures becomes sharply targeted at your actual weaknesses within a few months.
2. Behavioral signals from real usage#
Users tell you a great deal without being asked, provided you instrumented for it before you needed it.
- Citation click-through. Did they open the source? For a retrieval-backed feature, this is a reasonable proxy for whether the answer seemed worth verifying.
- Reformulation rate. Did they immediately rephrase and ask again? Usually a signal the first answer missed.
- Abandonment. Did they leave without engaging with the result?
- Downstream action. Did they do the thing the feature was supposed to help them do?
Each of these is weak individually, and each has a failure mode you have to say out loud:
Reformulation might mean the answer was bad — or that it was good and prompted a sharper follow-up. Citation clicks might mean the answer was credible enough to verify, or so implausible the user went to check. Abandonment might mean the answer was so complete that nothing further was needed.
This is why you track them as a set and watch for correlated movement. Reformulation up, citation clicks down, and abandonment up together is a real signal. Any one of them alone is noise with a story attached.
3. Structured human review, sampled#
You cannot review everything. You can review a sample, consistently, with a rubric.
Pull a stratified sample weekly — some high-confidence responses, some low-confidence, some from query types you care about. Have a reviewer score them on a small number of specific dimensions rather than a single quality score. “Was the retrieved context relevant?” and “was the answer supported by the context?” are separately answerable and separately actionable. A single 1–5 quality rating collapses them and tells you nothing about where to look.
Keep the rubric stable. Changing it resets your time series, and the trend is the point.
Separate retrieval failure from generation failure#
This is the single most useful structural decision, and it’s easy to miss because both failures look identical to the user: a wrong answer.
They have completely different remedies. If the right document was never retrieved, no amount of prompt engineering will help. If the right document was retrieved and the answer still misrepresented it, retrieval work is wasted effort.
Log the full retrieval context with every response. Then evaluate the two stages separately:
- Retrieval: was the necessary information in the candidate set? This is much closer to a classical information retrieval problem, and it’s substantially easier to evaluate — a human can answer “is the answer present in these passages?” quickly and consistently, without composing a correct response.
- Generation: given this context, is the response supported by it?
Splitting the problem this way converts one intractable question into two tractable ones. It’s also where most of the diagnostic value lives: in my experience the majority of bad answers are retrieval failures wearing a generation costume.
Signals that need no labels at all#
Some of the most useful metrics require no ground truth whatsoever, and they’re the ones I’d instrument first because they’re nearly free:
Abstention rate. How often does the system decline? A rising rate means retrieval is degrading or your corpus has gaps. A falling rate might mean your threshold has drifted permissive. Either direction is informative, and it’s available immediately.
Retrieval score distribution. The distribution of relevance scores across queries shifts before answer quality visibly degrades. It’s an early warning for corpus drift, embedding changes, and indexing problems.
Query pattern changes. If users start asking things they never asked before, your evaluation set is aging out from under you.
Coverage gaps. Queries that consistently trigger abstention identify documentation that doesn’t exist. That’s arguably more valuable than any retrieval improvement, and the system produces it as a byproduct.
None of these tell you whether an answer was correct. All of them tell you when something changed, which is when you should look.
On using a model to evaluate a model#
It works better than it has any right to, and the caveat is real: a model evaluating model output shares the evaluating model’s blind spots. If both systems misunderstand a domain in the same way, the evaluation will confidently confirm the error.
The defensible use is as an amplifier, not a replacement. Calibrate the automated judge against your human-reviewed sample. Measure agreement. If the judge agrees with human reviewers most of the time on cases where humans agree with each other, you can use it to scale — while continuing to sample for human review to detect drift in the judge itself.
Treating it as ground truth because it’s cheap is how you get a system that optimizes for what one model thinks another model should say.
The part nobody wants to hear#
Reviewers disagree with each other. Before concluding your system has an error rate, measure inter-reviewer agreement on the same examples. It’s routinely lower than people expect.
This matters for two reasons. It bounds how good your evaluation can be — you cannot measure quality more precisely than your reviewers agree on it. And it reframes some apparent model errors as genuine ambiguity in the task, which is a different problem with a different fix: usually clarifying the specification rather than changing the system.
Start smaller than feels responsible#
The failure mode I’ve seen most often isn’t building a bad evaluation. It’s planning a rigorous one, discovering it’s a quarter of work, deprioritizing it, and shipping changes uncontrolled for a year.
Fifty hand-written examples with assertions, run on every change, is worth more than a comprehensive framework that doesn’t exist yet. Add the behavioral instrumentation now, because retrofitting it means a long blind period. Everything else can come later.
The bar isn’t a number you can put in a slide. It’s being able to answer, honestly, whether the thing you changed yesterday made the product better — and knowing which parts of that answer you should trust.