The number the demo shows you
Almost every document-extraction demo lands on the same sentence: the system found 97% of the items. That is recall, and on its own it is close to meaningless. Recall tells you how much of what should have been found was found. It tells you nothing whatsoever about what else came out alongside it. A system that emits every candidate span in every document achieves perfect recall and is useless. Until someone gives you the second number, you cannot distinguish that system from a good one — and that is not a rhetorical flourish, it is arithmetic.
The second number is the false-extraction rate: of everything the system emitted, what share should not have been there. Some teams call it the wrong-item rate, some report its complement as precision. The name matters less than the fact that it is printed next to recall, computed on the same run, from the same matcher, broken out by category. When it is missing, the missing-ness is the finding.
Four counts, and the one that does not exist
Fix a corpus and a gold standard. G is the set of items that should be extracted. E is the set of items the system actually emitted. Three counts follow: hits (emitted items that match gold), misses (gold items with no emitted match), and false extractions (emitted items with no gold match). From those:
Recall = hits ÷ |G|. The share of what mattered that the system surfaced.
Precision = hits ÷ |E|. The share of what the system said that was right.
False-extraction rate = false extractions ÷ |E| = 1 − precision. The share of the output a reviewer has to throw away.
Extraction has no countable negative class
In classification you can count true negatives, so accuracy, specificity and ROC-AUC are all defined. In extraction over free text the negative class is every span the system could have emitted and did not — a quantity that depends on your tokenizer and is effectively unbounded. There is no honest true-negative count, so there is no honest accuracy figure. If a vendor quotes "99.2% accurate" for a document extractor, they have either silently redefined the task as classification over a fixed candidate list, or produced a number with no denominator. Either way, the right question is: accurate over what set?
None of this is new. The DARPA Message Understanding Conferences settled the bookkeeping more than thirty years ago, scoring extraction against categories of correct, partial, incorrect, missing and spurious, and defining overgeneration as spurious fills divided by actual fills — a false-extraction rate under an older name, alongside undergeneration as missing over possible. (Nancy Chinchor and Beth Sundheim, "MUC-5 Evaluation Metrics," Proceedings of the Fifth Message Understanding Conference, 1993; ACL Anthology M93-1007. The earlier MUC-3 and MUC-4 metrics papers, M91-1002 and M92-1002, define overgeneration the same way.) The field solved the reporting problem by 1993. Product demos un-solved it.
The arithmetic, worked
The following two systems are illustrative — invented to make the arithmetic legible, not measured on anything. Both are scored against the same corpus containing 100 gold items.
| Measure (illustrative) | System A | System B |
|---|---|---|
| Items emitted | 124 | 82 |
| Hits against 100 gold items | 95 | 78 |
| Recall | 95.0% | 78.0% |
| False extractions | 29 | 4 |
| False-extraction rate | 23.4% | 4.9% |
| Items emitted per gold item | 1.24 | 0.82 |
The headline writes itself: System A wins recall by seventeen points. Now read the same run as the person who has to check the output. A hands a reviewer 124 items to verify, 29 of which are wrong. B hands over 82, four of which are wrong. A's recall advantage is real — it found 17 true items B missed — but it purchased those 17 with 25 additional false ones and 42 additional items of review labor. The marginal precision of the recall lead is 17 ÷ 42, about 40%. That single derived figure is often more decision-relevant than either headline, and you cannot compute it without both numbers.
The last row is the fastest tell available to a buyer sitting in a demo. Items emitted per gold item — |E| ÷ |G| — takes one division and immediately reveals whether a recall lead came from better retrieval or from talking more. A ratio well above 1.0 with a strong recall number means volume is doing the work.
F1 does not rescue you, and neither does averaging
Collapsing the pair into one number is the obvious move and it destroys exactly the information you needed. On the illustrative figures above, System A scores an F1 of 84.8% and System B scores 85.7% — statistically indistinguishable for practical purposes, describing two systems with completely different operational profiles. A summary statistic that rates a 23.4% wrong-item rate and a 4.9% wrong-item rate as equivalent is not summarizing; it is hiding.
This failure mode is documented in the measurement literature. Chicco and Jurman show that accuracy and F1 can produce "overoptimistic inflated results," particularly on imbalanced data, because they ignore parts of the confusion matrix (BMC Genomics 21:6, 2020). Saito and Rehmsmeier make the companion point for plots: when negatives vastly outnumber positives, ROC curves flatter a classifier in a way precision-recall curves do not (PLOS ONE 10(3): e0118432, 2015). Extraction is the extreme case of imbalance — a handful of gold items inside thousands of candidate spans — so every one of these cautions applies with more force, not less.
Aggregate recall hides the category that carries the consequence
The second way a headline misleads is by averaging across categories of wildly different size and importance. Another illustrative run: 500 gold items across four categories — 200 appointments, 150 contacts, 120 restrictions, 30 dosage changes. The system hits 190, 140, 110 and 15 respectively. Overall recall is 455 ÷ 500, or 91.0%, and that is the number that goes on the slide.
Dosage changes are 6% of the gold set and the system finds half of them. The category that would actually hurt someone is the one the average is built to conceal, because it is small. Report per category or do not report. The same applies to the false-extraction rate: a system can be clean on the easy categories and noisy on exactly the one a reviewer trusts most.
The match rule is part of the metric
Neither number exists until you define what counts as a hit. Exact span match, normalized-value match, overlapping-span match and human adjudication produce materially different scores from the identical output. MUC carried a separate partial-match category precisely because correctness in extraction is not binary. So the match rule belongs in the published method, not in a footnote: change the matcher and both numbers move, which means an unpublished matcher makes a benchmark unfalsifiable no matter how many decimal places it carries.
Two further conditions decide whether the pair means anything. First, the split: a number computed on data the system was tuned against measures tuning, not generalization. Second, the provenance of the gold standard — who labeled it, under what written rule, and whether the labeling happened before the system saw the documents. We hold ourselves to labeling blind and leaving the engine untouched afterward, and we say so when reporting, because a reader has no way to check it otherwise.
Which error costs more is a mission question
Nothing above says low false-extraction is universally better. It says the trade has to be made deliberately, by someone who knows what the output feeds. A coverage sweep where a human reads everything and a miss is expensive should be tuned toward recall and will run a higher wrong-item rate; that is the correct answer for that job. An instruction consumed directly by an operator, or a field written into a record of decision, cannot afford the same posture. The metric pair does not make the choice — it makes the choice visible and auditable, which is the whole point.
That visibility is what an accreditation reviewer is actually asking for. Systems bound for an authorization boundary — the DoD Cloud Computing Security Requirements Guide published by DISA defines the Impact Levels those boundaries are expressed in — get read by people whose job is to find the claim that is not supported by evidence. The MEASURE function of the NIST AI Risk Management Framework (NIST AI 100-1, January 2023) calls for rigorous testing with stated uncertainty, comparison against benchmarks, and structured documentation of findings. One headline percentage satisfies none of that. A pair of numbers per category, a published matcher, and a blind split satisfy most of it.
We apply this to our own work and publish the result when it goes against us. In a benchmark we ran on safety-critical clinical text, a current frontier model out-recalled our deterministic extractor 97.9% to 68.2% — and emitted 657 items against 516 gold, at a 6.1% wrong-item rate versus our 2.6%. Both halves of that are in the write-up: the benchmark where our approach came second.
What we report, and what we refuse to
This is the reporting standard we hold ourselves to on extraction work, and the one we would ask any vendor to meet:
- Recall and false-extraction rate together, always, per category — same run, same matcher, never one without the other.
- Raw counts alongside every percentage — gold items, emitted items, hits, misses, false extractions.
- The match rule published in full, including how partial matches are treated.
- A blind split — data labeled before the system ran against it, with the system unchanged afterward.
- Items emitted per gold item, so a reader can see whether volume is doing the work.
- Repeat runs on identical input, because a system that answers differently twice does not have a measurable rate at all.
And the refusals, which matter more. We will not report a single headline accuracy figure for an extraction system — the denominator does not exist, and quoting one would be a small lie that makes every later number less trustworthy. We will not report recall without the wrong-item rate beside it. We will not tune against the set we then report on. We do not accept a model's self-reported confidence as a measurement; a confidence score is an output of the system under test, not evidence about it. We do not do human-subjects research and we are not a clinical or regulatory validation laboratory — a software measurement is not a safety claim, and we say which one we are making. And we will decline an evaluation engagement where no gold standard exists and no one in the program is willing to fund building one, because the resulting number would be decoration. We would rather say no than hand a customer a figure that cannot survive the first serious question about it.
Common objections
Isn't this just precision under a different name?
Numerically, yes — the false-extraction rate is 1 − precision. The reason to state it in the wrong-item direction is that it answers the question a reviewer actually has, which is "how much of this output is going to waste my time or mislead me?" People routinely nod at a precision figure and fail to notice that 76.6% precision means roughly one item in four is wrong. Framing changes what gets caught.
Our task has no gold standard. Can we still measure?
Partially, and you must say which part. Repeat-run agreement, schema validation against an external validator, and adjudicated spot checks on a sample all produce real evidence without a full gold set. What they do not produce is a recall figure, because recall requires knowing what was there to find. Reporting a recall number without a gold standard is the most common way these measurements go wrong.
What about generative systems where the output is prose, not items?
The same discipline applies once you decompose the prose into checkable claims and define what each claim must trace to. A claim with no supporting record is a false extraction regardless of how fluent the sentence is. The harder part is the decomposition rule, which then has to be published for the same reason the matcher does.
Doesn't a human reviewer make the wrong-item rate irrelevant?
It shifts the cost rather than removing it. Every false extraction is review labor, and past a certain density reviewers begin to skim, which is the failure mode that lets a wrong item through. A wrong-item rate is therefore also a prediction about how much attention the human layer will still have by the end of the queue.
Frequently asked questions
The share of the items a system emitted that should not have been emitted — false extractions divided by total emitted items. It is the complement of precision, and it is the number that tells a reviewer how much of the output is unusable.
Because it can be raised arbitrarily by emitting more. A system that outputs every candidate span reaches 100% recall while being worthless, and recall alone cannot distinguish it from a genuinely accurate system. Recall constrains only the misses; nothing constrains the noise until you also report the wrong-item rate.
Accuracy needs a true-negative count. In free-text extraction the negative class is every span that could have been emitted and was not, which is unbounded and tokenizer-dependent. Without a defensible denominator there is no defensible accuracy figure, which is why the MUC evaluations scored extraction with recall, precision and spurious-fill counts instead.
Four questions. How many items did the system emit, and how many gold items were there? What is the wrong-item rate, broken out by category? What match rule scored it? And was the evaluation data labeled before or after the system was tuned? A team that has done the work answers all four without hesitating.
No. For coverage sweeps where every item is human-reviewed and a miss is costly, higher recall at a higher wrong-item rate is the correct engineering choice. The requirement is that the trade be chosen deliberately and stated, not concealed behind a single number.