What a document pipeline owes the system downstream of it
Document intelligence is the practice of turning paper and PDFs into values a computer can act on. That sounds like an accuracy problem, and accuracy is part of it. But the pipeline is never the last stop. Its output flows into a case management system, a payment file, a data warehouse, an eligibility determination, or a records repository that someone will be asked about years later. What the pipeline owes that next system is three things: a value, a location for where the value came from, and a number describing how much to trust it. A pipeline that returns only the first of the three has handed its consumer a problem instead of an answer.
Start there and the engineering work reorders itself. The hardest part is rarely reading the characters. Modern OCR on clean 300 ppi print is close to solved. The hard parts are deciding what counts as one record, keeping the page coordinates attached to every value through four transformations, agreeing what a date means when the form says 03/04/25, and knowing what the system should do when a field is genuinely absent from the page. Those are design decisions, and most of them get made informally in the first two weeks by whoever writes the first parser.
The second reordering is about where the money goes. On real programs, model inference is usually the small line. Annotation, exception handling, and the humans who review low-confidence output dominate the run rate. If the design does not aim at reducing human touches on the documents that matter, a better model changes very little.
Where automated extraction pays off soonest, by document class
Editorial weighting from public sources and practitioner reading. Illustrative, not a measured statistic.
Design backward from the receiving schema
The first real artifact of a document program is not a model. It is a written schema of the target fields, with a type, a normalization rule, a cardinality, and a stated policy for absence on every one. Vendor name is a string, trimmed, with internal whitespace collapsed, one per document, and null is legal. Invoice total is a decimal with two places, currency USD, exactly one per document, and null is not legal because a document without one is not an invoice. Writing that document takes a week and saves a quarter.
Ask the receiving system what it will do with each field before you build the extractor for it. Fields that trigger a payment, an eligibility decision, or a records entry need a different accuracy bar and a different review policy than fields that populate a search facet. Teams that treat every field as equally important spend their annotation budget uniformly and then discover the three fields that mattered are the three that are worst.
The same conversation settles the question of what an absent value means. There is a difference between "the form has a blank there," "the page was too degraded to read it," and "this document class does not contain that field." Collapsing all three into an empty string destroys information the downstream reviewer needs. Give absence its own vocabulary in the schema on day one.
The five decisions that are expensive to reverse
The unit of extraction. Is one record a page, a document, or a submission package that spans several documents? Choose page and multi-page tables break. Choose package and you inherit an assembly problem before any reading happens. Changing this after the database exists means rewriting ingestion, the schema, and every evaluation set you have built.
Schema versioning. Fields will be added, renamed, and split. If every extracted record carries the schema version and the pipeline version that produced it, you can re-run a subset and compare. If it does not, you will never be able to answer why last March's records look different, and re-processing becomes an all-or-nothing event.
Keeping the image and the coordinates. Store the rendered page image, the OCR token stream with bounding boxes, and the character offsets that back every extracted value. Storage is cheap. Re-deriving provenance for a record whose source PDF has been archived is not, and a value with no way back to its pixel is a value nobody can defend.
Document identity. Content hash, source-system identifier, or a printed control number? Duplicates are the norm in federal corpora, where the same memorandum arrives by three routes. Pick an identity rule early, make it deterministic, and record when two identities collapse into one.
Where the human sits. Reviewing before commit and reviewing after commit are different systems. Pre-commit review means a queue, a latency budget, and a staffing model. Post-commit correction means a change log, a re-notification path, and a story for records already consumed downstream. Deciding this late usually means building both badly.
Where ground truth comes from
Nothing in a document program is measurable without labeled documents, and labeled documents are the line item that gets cut first. Budget it explicitly. A usable starting set is a few hundred documents per class, drawn by stratified sampling across scanner sources, form versions, and date ranges rather than by whatever landed in the sample folder.
Label a portion twice with different annotators and measure agreement before trusting any model number. When two careful humans disagree on 6 percent of a field, no model will be scored above 94 percent on it honestly, and the right response is to fix the annotation guideline, not the model. Field-level guidelines should be specific enough to settle arguments: does the vendor name include the legal suffix, does a handwritten correction override the printed value, which of two dates on the page is the signature date.
Hold out a locked evaluation set and do not look at it during development. Keep a separate, small canary set that mirrors production input and gets re-labeled on a schedule. Those two sets do different jobs. One tells you whether a change helped. The other tells you whether the world moved.
What "good" means, in numbers
Vague accuracy claims are the norm in this space, and they hide the two things a buyer needs. Per-field numbers beat aggregate ones, because aggregates are dominated by the easy fields. And error rates need confidence intervals, because sample sizes in document programs are small enough for the interval to swamp the difference between two systems.
| Measure | What it answers | Reasonable target |
|---|---|---|
| Character error rate | How well the text layer was recovered before any field logic runs | Under 1% on clean modern print; 5% to 15% is normal on degraded microfilm |
| Per-field exact match | Whether the normalized value equals the labeled value for that specific field | Set per field by consequence, not one number for the whole schema |
| False-extraction rate | How often the system returns a confident value that is wrong | Near zero on decision-bearing fields; it is the number that causes harm |
| Document-level correctness | Share of documents with every field right | Compute it. At 99% on each of 20 fields, only 82% of documents are clean |
| Abstention rate | How often the system declines rather than guessing | A tuned lever, not a defect; it trades recall for trust |
| Review yield | Share of human-reviewed items that actually needed correcting | Above 30% means routing is working; below 10% means you are paying for nothing |
Two pieces of arithmetic settle most disputes. First, 0.99 raised to the twentieth power is 0.82, so a 20-field form at 99 percent per field still hands a reviewer an error on roughly one document in five. Second, sample size. Measure 200 documents, observe 95 percent, and the 95 percent Wilson interval runs from about 91 to 97 percent. To claim 95 percent give or take two points you need on the order of 450 documents. And if a system reports zero errors on 300 documents, the rule of three says the true rate could still be as high as 1 percent.
Confidence and the threshold that sends a page to a person
Every serious document system routes uncertain output to a human. The threshold that decides which output is uncertain should be derived, not guessed. It comes from three inputs: the cost of a missed value, the cost of a wrong value that gets committed, and the cost of a human minute. When a wrong value causes an improper payment and a review costs a dollar, the threshold sits far toward caution. When the field feeds a search index, it sits the other way.
That derivation only works if the confidence score is calibrated. Raw model scores usually are not. A field the model reports at 0.9 should be right about 90 percent of the time; check that with a reliability plot on held-out data and correct it with isotonic regression or temperature scaling if it is off. An uncalibrated score turns the routing threshold into a dial someone twists by feel.
Design the review interface as part of the system, not as an afterthought. Show the reviewer the cropped page region with the candidate value highlighted, pre-filled and editable, with a keyboard path through the queue. The difference between a good and bad review screen is often three to five times the throughput per reviewer, which on a large corpus is a larger cost lever than any model change.
The failure modes teams hit
- Tables that cross a page break. The header is on page four, the rows continue on page five, and the parser starts a new table.
- Checkboxes and hand marks. An X outside the box, a circled option, a struck-through line. Nearly always the worst-performing field type.
- Multi-value fields collapsed to one. Three payees, two contract line items, four dates. The schema said cardinality one and the document disagreed.
- Dates and money without a normalization contract. 03/04/25 is ambiguous, and 1,234.00 versus 1.234,00 has ruined more reconciliations than any model error.
- Skew, rotation, and double feeds. Scanner artifacts that shift the entire coordinate frame and quietly break every position-based rule.
- Redaction blocks read as content. A black box over text can yield garbage tokens that look like a legitimate value.
- Silent template changes. A form revision moves a field two inches and accuracy on it drops to zero with no error raised anywhere.
- Answering when the document has no answer. A generative extractor asked for a field the page does not contain will often produce a plausible one.
The last item deserves the most attention, because it is the failure mode that erodes trust fastest. A system that says "not present on this page" is more useful than one that is right 96 percent of the time and confidently invents the rest. Build abstention into the contract with the model and test it deliberately by feeding documents you know lack the field.
Latency, throughput, and what a page actually costs
Do the arithmetic before choosing an architecture. One million pages at three seconds a page is about 35 days of serial processing. At 50-way parallelism it is about 17 hours. That single calculation determines whether the design needs a job queue, a batch window, and idempotent retries, or whether a simple worker will do.
Cost varies more than most teams expect. Published list prices at the major cloud services span more than an order of magnitude, from roughly a dollar and a half per thousand pages for plain text detection up to roughly fifty dollars per thousand pages for full forms and table analysis. Vision-model calls that send a whole page image are priced by tokens and a tiled 300 ppi page typically consumes a few thousand input tokens, while the extracted text of the same page often runs under a thousand. Reading text once and reasoning over the text is usually far cheaper than sending the image to every model in the chain. Check current price pages before committing a budget; these numbers move.
Then compare all of that to the humans. A reviewer working sixty documents an hour at a sixty-dollar loaded rate costs a dollar a document. Route twenty percent of the corpus to review and the human line alone is twenty cents a document, which on most designs exceeds every machine cost combined. This is why routing precision, not model accuracy, is usually the dominant cost lever in production.
Monitoring a pipeline that has no labels in production
Production gives you documents and no answer key. Monitoring has to work without one. Four signals carry most of the value. Watch the per-field null rate, because a template change shows up as a field that suddenly stops being found. Watch the confidence distribution, because a shift in its shape precedes a shift in accuracy. Watch the input distribution itself: page counts, image dimensions, DPI, source system, share of pages with a text layer. And watch the review queue, because a rising correction rate on reviewed items is the earliest honest accuracy signal available.
Alongside those, run the canary set on every deployment and record the numbers in a versioned log. Re-label a small random production sample on a fixed cadence, monthly or quarterly, so the accuracy claim has a current basis rather than a launch-day one. Budget for it in the operations plan; it is the only thing that keeps a two-year-old number from becoming a fiction.
Log enough to reconstruct any decision: the document identity, the pipeline and schema versions, the model versions, the confidence, the coordinates, and the reviewer action if any. When someone asks in eighteen months how a specific value got into the record, that log is the whole answer.
The federal rules that shape the design
Federal document work has constraints that arrive from outside the engineering. NARA's electronic records rules at 36 CFR part 1236 govern how electronic records are managed and set minimum capture standards for digitization, including 300 ppi for textual records and required metadata that travels with the image. OMB memorandum M-23-07 pushed agencies to manage permanent records in electronic form and set June 30, 2024 as the point after which NARA stopped accepting analog transfers of permanent records, which is why so many legacy scanning backlogs exist at all.
On the security side, controlled unclassified information is defined by 32 CFR part 2002, protected on nonfederal systems under NIST SP 800-171, and contractually bound for defense work by DFARS 252.204-7012, which requires cloud services handling covered defense information to meet FedRAMP Moderate equivalency and requires cyber incident reporting within 72 hours. Defense workloads also carry impact-level constraints from the DoD Cloud Computing Security Requirements Guide. Any decision to send pages to a hosted model is a decision inside that boundary, and it belongs in the System Security Plan.
Two more rules shape design in ways teams miss. The Paperwork Reduction Act machinery at 5 CFR part 1320 requires federal collection forms to display a valid OMB control number and expiration date, which gives you a printed, machine-readable version key for template detection. And Section 508, codified at 29 U.S.C. 794d with standards at 36 CFR part 1194 incorporating WCAG 2.0 Level AA, means a scanned image with no text layer is not an accessible document, so OCR output frequently has an accessibility purpose in addition to a data one.
When a simpler method is the right answer
Sometimes the honest recommendation is to build less. If the PDFs are digital-native AcroForm or XFA files, the field values are already in the file structure and can be read directly with a library in an afternoon, at perfect fidelity, for no inference cost. Checking for that takes ten minutes and eliminates entire projects.
If the corpus is one stable template with fixed field positions, positional cropping plus OCR on the crop, with a template-version check driven by the printed control number, will beat a general model on both accuracy and cost. If the fields are strongly patterned identifiers such as a contract number, a CAGE code, or a UEI, a validated regular expression with a check-digit or format rule is more reliable than a model and its failures are inspectable. If the documents originate in an upstream system that has an API, the correct engineering answer is often to stop parsing its printouts and negotiate access to the source.
The general models earn their keep on variety: many templates, unpredictable layouts, natural-language passages, documents nobody controls. That is a real and common condition, and it is where we build. But the discipline of asking "what would this cost with rules" first is what keeps a program from spending six figures automating something a hundred lines of Python already did.
How we run this work
Our approach is consistent across programs. We write the target schema first and get it signed by whoever owns the receiving system. We build a labeled evaluation set with double annotation on the fields that carry consequence. We baseline with the simplest method that could work, so every later increment has to prove it earned its complexity. We report per-field numbers with intervals and a false-extraction rate, never a single accuracy figure. And we ship the review interface as part of the system, because that is where the operating cost lives.
Our engineers build these pipelines on federal cloud footprints with the boundary documented, the provenance preserved from pixel to record, and the monitoring in place on day one rather than after the first incident. That is what makes an extraction program something an agency can keep running rather than something it has to defend.
Common questions on scoping
Should we start with an off-the-shelf document service?
Usually yes, as a baseline. The hosted form and table services set a real bar quickly and tell you which fields are hard. Keep the interface behind an adapter so the engine can be swapped, and check the security boundary before any page leaves your environment.
Do we need a fine-tuned model?
Only after a baseline shows which fields are failing and why. Fine-tuning helps most on domain-specific layout and vocabulary with a few thousand labeled examples available. It rarely fixes bad scans, an ambiguous schema, or disagreement among annotators.
How long before the system is useful?
A scoped pilot on one document class with a written schema and a labeled evaluation set typically produces defensible numbers within weeks. What extends a timeline is corpus variety, access to representative documents, and the security review for the environment the pages will be processed in.
What changes if the documents are CUI?
The processing boundary becomes a design constraint rather than a deployment detail. Marking, storage, transmission, and the choice of any hosted model all sit inside the control set, and the boundary belongs in the security documentation before the first page is processed.
Frequently asked questions
Turning documents into structured values a system can act on. A working pipeline covers capture, text recovery, layout understanding, field extraction, normalization, confidence scoring, human review routing, and provenance. Reading the characters is one step of eight.
There is no single number, and any vendor offering one is hiding the distribution. Set targets per field by consequence, report an interval alongside each rate, and treat the false-extraction rate on decision-bearing fields as the number that matters most.
A few hundred per document class to start, drawn by stratified sampling across sources and form versions. To claim a rate within about two points at 95 percent confidence, plan on roughly 450 evaluated documents, plus a smaller canary set that gets re-labeled on a schedule.
When the PDFs carry form fields you can read directly, when the corpus is one stable template with fixed positions, when the target is a strongly patterned identifier with a validation rule, or when the source system has an API you could ask for instead.
Human review, in most designs. Machine costs per thousand pages are measured in dollars while a reviewed document is measured in dollars each. Improving which items get routed to a person is usually a bigger lever than improving raw model accuracy.