The moment the output stops being an answer
Most document-extraction work begins as a convenience. Someone is reading four-hundred-page packets by hand, a model reads them faster, and the output lands in a spreadsheet that a person still checks line by line. That arrangement is forgiving, because nothing downstream depends on the extraction being right — the human reviewer is the system. It stops being forgiving the moment the extracted value stops being a suggestion and becomes an entry: a field in a case management system, a figure in a determination, a number in a report that leaves the organization with somebody's name attached to it.
At that point the output is a record, and records are read by people who were not present when they were produced. In a federal setting the definition is not a matter of opinion: 44 U.S.C. § 3301 reaches all recorded information, regardless of form or characteristics, made or received by an agency under federal law or in connection with the transaction of public business and preserved as evidence of its operations. Once your pipeline's output meets that description, it inherits an entire apparatus — retention, production, review, and eventually somebody asking about it in a setting you do not control.
None of those readers will ask how the model works. An inspector general, an appeals officer, a contracting officer's representative, opposing counsel, or the program manager who takes over the system two years from now all ask the same question in different phrasings: where did this number come from? A system that cannot answer at the level of a specific sentence, in a specific version of a specific document, has not failed a technical benchmark. It has failed the only test that was ever going to be applied to it.
Why a confident summary fails and a located span does not
A generated summary is an assertion the system makes. It may well be correct. But its correctness is not checkable from the artifact itself: the reviewer holding the record has a sentence and a confidence number, and no path from either back to the page. Verifying it means redoing the extraction by hand, which is the work the system was bought to remove. The confidence score does not help, because a calibrated 0.94 tells you about the distribution of the model's outputs, not about this document.
The failure mode here is documented, and it is worse than outright fabrication. Magesh and colleagues, in "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools" (Journal of Empirical Legal Studies, 2025; preprint arXiv:2405.20362), evaluated commercial retrieval-backed research tools whose vendors advertised freedom from hallucination, and found incorrect output on between 17 and 33 percent of queries depending on the tool. The number is not the part worth carrying into a design review. The part worth carrying is the authors' observation that a citation pointing at a real source that does not actually support the claim is more dangerous than an invented one, precisely because it survives the check a busy reviewer performs.
The research framing for what a citation must mean predates the current tooling. Rashkin and colleagues set it out as Attributable to Identified Sources in "Measuring Attribution in Natural Language Generation Models" (Computational Linguistics 49(4), 2023): output about the external world should be verifiable against an independent source the system itself provides. A span-level citation is the operational form of that requirement. Not "this claim is supported somewhere in the corpus," but "this value is these characters, in this version of this document, at these coordinates."
What has to travel with the value
There is a ladder here, and most systems stop two rungs too early. Each rung survives a different kind of scrutiny.
| What travels with the value | What it survives | What defeats it |
|---|---|---|
| A confidence score | A demo | Any question beginning "where did this come from" |
| A document-level reference | A spot check on a short document | A four-hundred-page packet, or two documents that disagree |
| A page number | A reviewer with time and the same copy you had | Re-pagination, a re-scan, a different renderer, a redacted release |
| Character offsets into a named text layer | Schema changes and re-exports downstream | Re-running OCR or text extraction without pinning the version |
| Offsets, a coordinate box, and a hash of the text layer | A different reviewer, on a different machine, a year later | Deleting the source artifact — nothing else |
The bottom rung is the one that costs engineering time, and the cost is not in the citation format. It is in the immutability the format assumes. A locator is only durable if the thing it points into cannot change underneath it. Text layers change constantly by default: someone upgrades the OCR engine, someone swaps the PDF text extractor, someone adds a cleanup step. Treating the extracted text as a versioned, hashed artifact — produced once, stored, never regenerated on the fly — is the unglamorous decision that makes every citation above it mean something.
The chain is everything after the extraction
Nearly all the attention in this problem goes to the first step: finding the span. That step is the easiest one to demonstrate and the least likely to be where the chain breaks. The breaks happen in the three steps nobody demos.
Normalization. The characters on the page read 12/03/24. The record needs a date. Whether that becomes 3 December 2024 or 12 March 2024 depends on a convention that lives outside the document, and the same is true of currency, units, negation, and name forms. The correct pattern is that normalization emits a new value carrying a pointer to its input, and never overwrites the input. The record then holds both: the typed value the system computes on, and the characters a human can read and disagree with.
Coding. Mapping an extracted phrase to a controlled vocabulary — a diagnosis code, a NAICS code, a FAR clause reference, an agency's own internal code set — is an inference, not a lookup, and code sets are versioned and revised on a schedule. A record that carries only the code has lost the argument before it starts. What it needs is the code, the code system, the code system version, the span the code was inferred from, and what performed the inference. That is five fields where most schemas have one.
Export. The last step is where chains die quietly. A pipeline with excellent internal provenance writes a CSV, a JSON payload, or an HL7 or XML message into someone else's system, and the destination schema has no column for provenance. Every guarantee upstream of that boundary is now a guarantee about a system nobody will ever query. If the target schema cannot hold the locator, the honest options are to extend it, to write a companion evidence table keyed to the record identifier, or to say plainly that the output is a draft rather than a record. We wrote separately about what it takes to integrate into a system of record without corrupting it; the provenance column is the part that gets negotiated away first and missed worst.
Where the chain breaks in practice
Source-link durability by stage — editorial rating
Editorial weighting drawn from the published document-AI literature and practitioner reading — a judgment about relative difficulty, not a measured statistic.
- OCR — offsets point into the OCR text layer, not into the page. Change engine or version and the text changes, so every stored offset silently means something else. Carry word-level coordinates alongside the offsets, in a format like ALTO XML or hOCR, and pin the engine version as a build dependency.
- Tables — a cell's meaning comes from a header path that may sit several rows above it. A span capturing
4,210without its row and column headers is not a citation, it is a number. Header association has to be stored as part of the locator. - Merged cells — spanned headers are the specific case that breaks naive parsers. The PubTables-1M work of Smock, Pesala and Abraham (CVPR 2022) introduced a canonicalization step precisely because oversegmentation was a systematic source of ground-truth inconsistency in earlier table datasets.
- Re-wrapping — de-hyphenation, whitespace collapse, sentence splitting, HTML-to-text conversion. Any step that reflows text shifts every offset after it. Transformations must emit an explicit offset mapping rather than mutating in place.
- Unicode normalization — NFC and NFD, defined in Unicode Standard Annex #15, change the code-point sequence itself. Soft hyphens, non-breaking spaces, and ligature expansion do the same. Offsets computed before normalization are wrong after it, and nothing raises an error.
- Translation — there is no character-level alignment guarantee between a source-language sentence and its rendering in English. The citation must point at the source-language span, with the translation labeled as a rendering.
Translation deserves its own sentence because it is where well-run programs make the most expensive mistake. Translating a corpus to English and then extracting from the translation feels tidy and produces a chain whose last verifiable link is a machine artifact. Under that design, the reviewer who pulls the citation reads text no author ever wrote. The workable arrangement is the reverse order: extract in the source language, cite the source span, and present the translation as an aid that is explicitly marked as one, with the original always one click away. That is slower and it is the only version that holds up when somebody who reads the language looks at it.
The recall you give up, stated plainly
An engine restricted to values it can point at cannot state what the document does not contain. That is the guarantee, and it has a price that should be named before a contract is signed rather than discovered during acceptance testing.
The misses are systematic, not random. A total that has to be derived from three line items is not present as characters. A status carried by a checkbox with no adjacent text is not present as characters. A fact that lives across two documents — the date in one, the finding in the other — is not present as characters in either. A negation phrased as "no evidence of" when the field asks whether the condition was found requires reasoning the span cannot carry on its own. These are misses by construction. They are not bugs, and no amount of tuning removes them.
What you get in exchange is that the misses are visible. A blank field announces itself; a fabricated one does not. Recall you can measure is recall you can staff around, with a human review queue sized to the gap. Silent fabrication offers no such handle, and it is discovered by the person least equipped to absorb the discovery. That trade is the whole argument, and we have written it up with the arithmetic in the false-extraction rate and as a head-to-head comparison in a benchmark where the higher-recall system was still the wrong one to ship.
Test the chain, do not trust it
Provenance is the kind of property that appears to work until the day it matters. It should be tested in continuous integration on every document, the same way a schema is tested — not reviewed once at design time.
- Round-trip every citation. For each exported value, resolve its locator, pull the substring, and assert string equality with what the extractor claimed. This single test catches most offset drift.
- Re-render the cited region onto the page image and confirm the box lands on the value. Offsets can be internally consistent and still point at the wrong place on the page.
- Mutate the source. Delete the sentence or change the number, re-run, and assert the output changes or fails. A value that persists unchanged came from a cache or from the model's memory, and that is the failure you most want to catch.
- Run negative controls. Feed documents that genuinely lack the field. The correct output is "not present." Anything plausible returned here is invisible in aggregate accuracy and fatal in review.
- Pin and verify the text layer. Re-run extraction with the pinned engine version and assert a byte-identical text layer against its stored hash. If it differs, the layer was never pinned.
- Test from the destination, not the pipeline. Read the record back out of the system of record and resolve the citation from there. The only chain that matters is the one that survived the export.
- Time a real reviewer. Hand ten records to the person who signs and measure how long verifying one takes. If it is slower than doing the work by hand, the citation is decoration.
The requirement already exists in the receiving organization
Teams often treat span-level provenance as an extra their customer has not asked for. Usually the customer has asked for it, in a document written before anyone involved was thinking about extraction. NARA's Universal Electronic Records Management Requirements, Version 3 (June 2023), sets baseline lifecycle requirements derived from statute, regulation and standards for any system that produces or manages federal electronic records. NIST SP 800-53 Rev. 5 carries the same expectation in control language: AU-3 on the content of audit records, AU-10 on non-repudiation, and SI-7 on information integrity.
Where the record may end up in a proceeding, Federal Rules of Evidence 902(13) and 902(14) — added effective December 1, 2017 — allow electronic records to be self-authenticating on a certification by a qualified person, with the advisory committee notes describing hash comparison as the mechanism for establishing that a copy is identical to the original. That is a legal rule written around exactly the artifact a hashed, versioned text layer produces.
On the AI side specifically, OMB Memorandum M-25-21 (April 3, 2025), which rescinded and replaced M-24-10 and remains in force, sets minimum risk management practices for high-impact AI. Among them: pre-deployment testing, a completed AI impact assessment, ongoing monitoring for performance and adverse impact, human oversight and intervention, and consistent remedies or appeals. Not one of those is achievable in any meaningful sense unless the reviewer can reach the source. And in intelligence work the standard is older still: Intelligence Community Directive 206 requires disseminated analytic products to carry source reference citations and source summary statements describing the quality and credibility of what the analysis rests on. Sourcing has been the professional norm in that community for decades; the tooling is what is new.
What we build here, and what we do not
Precision Federal is an Iowa limited liability company — a small engineering firm with a few members. In this area we build the extraction pipeline itself: entity and relation extraction, classification, de-identification ahead of downstream processing, evaluation harnesses with strict and partial match scoring, and the serving and monitoring around them. The detail of that work sits on our NLP capability page, and the retrieval side of it on RAG systems. What we care about most in a build like this is the plumbing described above — versioned text layers, offset mappings that survive every transformation, and an export that carries the locator into the customer's schema rather than dropping it at the boundary.
There is a clear line around that, and it is worth stating before a first conversation rather than during one.
We do not make records determinations
What constitutes a federal record, which disposition schedule applies, and when something may be destroyed are decisions for the agency records officer under NARA authority. We build the system that carries the evidence and emits the metadata a schedule needs. We do not decide what is scheduled or for how long.
We do not give legal advice on admissibility
Rules 902(13) and 902(14) turn on a certification by a qualified person, and the qualified person is your declarant, working with your counsel. We can build the system so that the certification is straightforward to make and produce the hashes and logs it rests on. Whether a record comes in is not our call to make.
We do not assess or certify our own work
If we build the pipeline, we do not also write the independent evaluation that says the pipeline is sound. Independence is worth nothing once the assessor has a stake in the finding. We will hand an engineering review to a customer outright, and we will not follow it with a proposal to fix what we found.
We will not add a best-guess mode to a record-producing pipeline
The most common late request on this kind of build is to fill the blanks — have the model infer a value where no span was found, so the completion rate looks better. We decline it. A blank is a routing decision to a human. A quietly inferred value is an unlabeled assertion inside a record somebody signs, and it is unrecoverable once it is in there.
We do not perform classified processing on classified networks
We do not hold a facility clearance today. Under the National Industrial Security Program Operating Manual at 32 CFR part 117, a facility clearance is sponsored by a government contracting activity or a cleared prime against a bona fide requirement; a company cannot obtain one on its own initiative. We build to the constraints those environments impose so work transfers cleanly, and we are careful not to let designing to a pattern imply operating inside an accredited enclave.
Bottom line
The distance between a document-extraction demo and a document-extraction system that survives review is almost entirely in the parts that are not fun to build: an immutable text layer, offsets that survive every transformation applied to them, a coding step that keeps its inputs, an export that carries the locator across the boundary, and a test suite that proves all of it on every document rather than asserting it in a design document. The reward for doing that work is narrow and specific. When somebody asks where a number came from, the answer takes four seconds instead of four hours, and it is the same answer every time it is asked.
Frequently asked questions
It means each extracted value carries the exact location it came from: a document identifier, a version or hash of the text layer, character offsets into that layer, and ideally a coordinate box on the page. The test is whether a reviewer can resolve the locator and read the original characters without redoing the extraction.
Page numbers are a property of a rendering, not of the document. Re-pagination, a different renderer, a re-scan, or a redacted release all change them, and none of those events produces an error. A locator anchored to a versioned text layer plus a coordinate box survives all four.
It lowers recall, and the losses are predictable: derived totals, checkbox-only status, facts split across two documents, and negations that need reasoning rather than location. What you gain is that every miss is visible and can be routed to a person, rather than appearing as a confident value nobody flags.
Silent text mutation. De-hyphenation, whitespace collapse, Unicode normalization under Annex #15, and HTML-to-text conversion all shift offsets without raising an error, and an unpinned OCR engine version changes the text layer underneath every stored locator. Tables with spanned headers are the second common source.
In continuous integration, on every document: resolve each citation and assert the substring matches, re-render the cited box onto the page image, mutate the source and assert the output changes, run negative controls on documents that lack the field, and resolve citations from the destination system rather than from the pipeline that produced them.