Open any evaluation report and it describes performance on a test set of questions that have answers. That is a reasonable thing to measure and it omits the behavior that determines whether a system is safe to deploy: what it does when the answer is not there.
Why the unanswerable set is missing
Partly convenience — building a set of questions with known answers is straightforward and building a set of questions verified to have no answer in the corpus is not. Partly incentive: the resulting number is unflattering and there is no convention requiring it.
The consequence is that two systems reporting identical accuracy can behave completely differently in production. One declines when the data is absent. The other produces a fluent, well-cited, incorrect answer, and its accuracy score does not reveal this.
Two systems, the same accuracy. One says it does not know. Nothing in the benchmark distinguishes them.
Constructing the set
- Questions about entities absent from the corpus entirely
- Questions about a period the data does not cover
- Questions requiring a field that is not collected
- Questions whose premise is false — asking about an event that did not occur
- Questions the corpus answers ambiguously, where the correct response names the ambiguity
The false-premise category is the most revealing and the most neglected. A system asked why a decline occurred, when no decline occurred, will very often explain one. That behavior is invisible to every benchmark built on well-formed questions.
What to report
Three numbers, and the third is the one that changes decisions.
The refusal rate on unanswerable questions — how often the system correctly declines. The over-refusal rate on answerable ones — because a system that declines everything scores perfectly on the first metric and is useless. And the confident-error rate: of the unanswerable questions it did answer, how many were stated without any hedge.
That third number is the deployment risk, expressed directly. A system with a modest fabrication rate that always signals uncertainty is manageable. A system with the same rate that never signals it is not, and only this metric separates them.
What published evaluations actually measure
Relative weighting from delivery practice, not a measured statistic — shown to rank where attention belongs.
What it costs to measure
Less than expected. A hundred unanswerable questions across the five categories, written by someone who knows the corpus, is a day of work and can be reused across versions.
The reuse is what makes it worth building. The set becomes a regression check that catches the specific degradation that functional testing never sees — a model update that quietly made the system more willing to answer.
How to use it as a buyer
Ask for it. Not for the number, initially, but for the methodology — how the unanswerable set was constructed and how refusal was scored.
A team that has measured this can describe it immediately. A team that has not will offer the accuracy figure again, and that substitution is the answer to the question you asked.
The five categories, and what each one exposes
An unanswerable set is not homogeneous. Each category tests a different failure and they fail at noticeably different rates.
| Category | Example form | What a failure reveals |
|---|---|---|
| Absent entity | A question about something not in the corpus | Retrieval returns nearest neighbors; the model treats them as the subject |
| Out-of-range period | A question about a year not covered | Adjacent periods substituted silently |
| Uncollected field | A question about a dimension never recorded | Inference from correlated fields, stated as fact |
| False premise | Asking why an event occurred when it did not | The model explains an event into existence |
| Genuine ambiguity | The corpus supports two readings | One reading chosen with no indication a choice was made |
The false-premise row is the one that most reliably produces alarming behavior. A question shaped as "why did X happen" carries a strong presupposition, and systems that handle the other four categories acceptably frequently fail this one outright.
Three numbers, and why all three are needed
Reporting a single refusal rate invites the obvious gaming: a system that declines everything scores perfectly. The three-number report closes that.
Correct refusal rate — on the unanswerable set, how often the system declined. Over-refusal rate — on the answerable set, how often it declined something it should have answered. And confident-error rate — of the unanswerable questions it answered anyway, how many were stated with no hedge at all.
The third is the deployment risk expressed directly, and it is the number that should drive the decision. A system that occasionally guesses but always signals uncertainty can be deployed with a review step. A system that guesses with the same confidence it uses when correct cannot, because there is no signal for a reviewer to act on.
Scoring, which is where this gets sloppy
The categories are easy. Scoring the responses consistently is where a well-intentioned evaluation loses its meaning.
- A refusal must name what is missing. "I cannot answer that" is a weaker pass than "the corpus covers 2019 onward and this question concerns 2016"
- Hedged answers are answers. "It may be approximately X" scores as an answer, not a refusal — it will be read as X
- Partial answers score by their assertions. A response that correctly declines one part and fabricates another is a failure
- Citations do not confer correctness. A fabricated claim citing a real document is a worse failure, not a mitigated one
- Score blind and use two raters on a subset, because the hedge boundary is genuinely a judgment call
Writing these rules down before scoring begins is what makes the number comparable across versions. Written afterward, they get shaped by the results.
Using it as a regression check
The set's real value is longitudinal. Built once, it becomes the instrument that detects a specific and otherwise invisible degradation: a model or prompt change that made the system more willing to answer.
That change breaks no test. Latency is unaffected, format is unaffected, accuracy on the answerable set may even improve slightly. The only visible symptom is that the confident-error rate moved, and nothing else in a normal test suite is looking at it.
Running the set on every model version, prompt change, and retrieval configuration change costs very little once it exists. It is the cheapest guard available against the failure mode that will otherwise be discovered by a customer.
Designing the refusal itself
A system that declines badly trains users to route around it, which defeats the purpose. The design target is a decline that leaves the user better off than before they asked.
That means naming the specific gap rather than issuing a generic refusal, stating what would be needed to answer, and offering the nearest thing the data does support. "The corpus does not cover that period; the earliest available is 2019, and here is the comparable figure for that year" is useful. A bare inability is not, and users who receive enough of them stop asking the system questions it could have answered.
This matters commercially as much as technically. The most common objection to grounded systems is that they refuse too much, and in most cases the actual problem is not the refusal rate but the refusal design.
Agentic systems make this sharper, not softer
Everything above assumes a system that answers a question. A system that takes actions — calling tools, writing records, triggering workflows — inherits the same failure with higher consequences.
An agent that cannot establish what a user wants will, under the same generative pressure, choose an action anyway. The unanswerable-question problem becomes the unactionable-request problem, and the output is not a wrong sentence but a wrong operation against a real system.
The evaluation extends accordingly. Alongside unanswerable questions, the set needs requests that cannot be fulfilled with the available tools, requests whose parameters are underspecified, and requests whose premise is false. Correct behavior is to stop and say what is missing. Measuring how often it does is the agentic analogue of the refusal rate, and it is measured even less often.
Putting the number in the contract
Because this metric is rarely volunteered, buyers who want it generally have to require it, and the cleanest place is the acceptance criteria of a paid evaluation.
- An unanswerable set built by the buyer, from the buyer's data, not shared in advance
- A stated maximum confident-error rate, defined and agreed before the run
- A stated maximum over-refusal rate, so the vendor cannot pass by declining everything
- Scoring rules written down before scoring begins
- The right to re-run the set after any model or configuration change
The last clause is the one with the longest tail of value. It converts an acceptance test into an ongoing right, which is what makes it possible to notice that a silent update changed the system's willingness to guess.
Why this metric is the one to pick
There are many things worth measuring, and if an organization will only adopt one new practice, this is the one with the best return.
It is cheap — a day of domain-expert time produces a reusable set. It is legible — the three numbers are interpretable by people with no machine learning background. It targets the failure mode that actually causes harm in deployment, rather than the one that shows up in benchmarks. And it functions as a regression check against the specific silent degradation that nothing else in a normal test suite is watching for.
Accuracy tells you how the system behaves when it knows. This tells you how it behaves when it does not, and in production that is the more consequential half.
Frequently asked questions
A refusal that names what is missing is a good experience. A bare refusal is a poor one. The design target is a decline that tells the user what would be needed to answer.
Partially, through instructions and thresholds, and the effect is unreliable under pressure. Systems that decline dependably do so because the architecture cannot produce an unsourced answer, not because they were asked to be careful.