The errors that cost money do not look like errors
A pilot on fifty documents reaches ninety-something percent field accuracy in a fortnight and everyone concludes the problem is solved. Then the pipeline runs over sixty thousand documents from four hundred filers across nine years, and a number in a report is wrong by a factor of a thousand. Nobody notices for a month. When someone finally traces it, the character recognition was perfect: the table header said in thousands two rows above the cell, the parser attached the units of the previous table, and the extracted value was a clean, confident, well-formed lie. Every expensive failure in this domain has that shape. The output is not garbled. It is plausible, and plausible passes review.
That is why extraction accuracy on its own is a poor way to run this. The metric that predicts whether the system is trusted a year in is not how often a value is read correctly. It is how often a wrong value reaches a downstream consumer without anything catching it. Those are different quantities, and the second one is improved mostly by things that are not the extractor: unit propagation, period alignment, version handling, and arithmetic checks that the document itself supplies for free.
You are probably here because
- A pilot hit 94% on fifty documents and something went badly wrong at sixty thousand
- The same company appears twice with different figures for the same period and both came from your pipeline
- An analyst team is still re-keying the tables the pipeline was supposed to replace
- Nobody can say what a checked, trustworthy field actually costs you
The first two are unit, period and version problems. The third is a routing problem — review is not targeted where the risk is. The fourth is the number this article is mostly about.
Know the corpus before you choose the tooling
Financial documents are not one population. Profile yours before anyone picks a model, because the profile determines the architecture and the cost within a factor of ten.
Machine-readable versus scanned. A native digital document carries a text layer with exact glyph coordinates, and reading it is a parsing problem with no recognition error at all. A scan requires character recognition and inherits its error rate. In most corpora we see, the recent years are overwhelmingly native and the older years are scans, so the accuracy of the pipeline drops sharply at a date boundary — and a headline accuracy number averaged across the whole corpus hides that completely. Report accuracy by document class, always.
Templated versus free-form. Large filers reuse their own layout for years, so per-filer templates get you very high accuracy for a modest amount of work and are worth building for the head of the distribution. The tail is where general methods earn their cost. A useful early measurement: what fraction of your documents comes from the top fifty sources? If it is more than half, a hybrid beats a single general approach on both accuracy and unit cost.
Tables versus prose. In financial documents the payload is nearly always tabular, and tables are where extraction is genuinely hard: merged cells, multi-level headers spanning several periods, units declared once in a caption, footnote markers glued to digits, negative numbers in parentheses, and column alignment that exists visually but not structurally. Prose extraction is comparatively easy and comparatively rare in what people actually want.
| Error class | What it looks like downstream | What catches it |
|---|---|---|
| Unit scope thousands, millions, per-share | A value off by 10×, 1,000× or 1,000,000× | Magnitude checks against prior period; unit inherited explicitly, never by proximity |
| Sign convention parentheses, "less", credit balances | A cost that reduces a total instead of increasing it | Cross-footing the column; sign rules per line-item type |
| Period misassignment | Last year's figure filed under this year | Header period parsed as a first-class field, not a column index |
| Restated comparatives | Two different values for the same period, both correct | Store as-reported plus source document, resolve at query time |
| Amendment superseded | A figure the filer already corrected | Version chain per document; latest-version view |
| Character recognition | Digit transposition, a stray decimal | Arithmetic reconciliation; this is the class everyone plans for and the smallest |
Units and signs, treated as first-class fields
A number extracted without its unit is not data. Yet the common implementation attaches units by nearest-text heuristic, which works on the documents in the sample and fails whenever a page carries two tables at different scales — a per-share table under a thousands table is the standard trap.
Make the unit an explicit, required attribute of every extracted value, with a recorded source: which text, on which page, at which coordinates, declared it. If no unit declaration can be located, the value is not defaulted to anything. It is emitted with an unknown unit and routed to review. A null unit is a routable event; a wrongly assumed unit is a silent corruption, and the difference between those two is most of the value of the whole system.
Signs deserve the same discipline. Parentheses mean negative in almost all financial presentation, and the exceptions are real: some tables present expenses as positive under a header that says the total is a deduction, some present a bracketed figure as a reference rather than a value. Encode the rule per line-item type rather than globally, keep the raw string alongside the parsed number, and let a reviewer see (1,234) next to -1234 so the interpretation is visible.
Periods, restatements and the version chain
Period alignment is the second largest silent error class and it is entirely a modelling problem. Fiscal years do not align to calendar years. Retail calendars run fifty-two or fifty-three weeks, so one year in five has an extra week and any year-over-year growth calculation that ignores it is wrong by roughly two percent. Acquisitions create stub periods. A filer that changes its year end produces a transition period of unusual length that looks like a data error and is not.
Parse the period as a structured object — start date, end date, duration in months, fiscal label — and never as a column index. Column two is not "prior year" in any stable sense, and pipelines built on positional assumptions break the first time a filer adds a column.
Then there is the fact that the same period legitimately has more than one value. The current filing restates a prior year to reflect a discontinued operation, and both the original and the restated figure are correct as-reported statements from different documents. A pipeline that overwrites has destroyed information and will produce a history that no longer matches anything previously published. Store every extraction keyed by (document, document version, period, concept), mark the as-reported and the restated values distinctly, and let the consumer choose. Most analytical uses want the latest restated view; anything reconstructing what was knowable at a past date wants as-reported. Both must be answerable from the same store.
Amendments are the same problem one level up. An amended filing supersedes an original, sometimes months later. The extraction store therefore holds a version chain per document, and the default view resolves to the latest version while keeping every prior version retrievable. Teams that key on (company, period) instead of on the document discover this the first time an amendment lands, usually by finding two rows and no way to decide.
Every value keeps its coordinates
Store, with every extracted number: document id, document version, page, bounding box, the raw text as it appeared, the parsed value, the unit and its declaration source, the period object, the extractor and its version, and a confidence. It roughly doubles the size of the store and it is the difference between a system people trust and one they re-key around. When an analyst can click a figure and see the highlighted cell on the page it came from, disputes take thirty seconds instead of half a day, and a wrong value becomes a fixable bug rather than a reason to abandon the pipeline.
Architecture: idempotent per document, cached by content
At volume the pipeline design that survives is boring and specific. The unit of work is one document version, and processing it is idempotent: rerunning a document produces the same output and does not duplicate rows. Address documents by content hash, so the same file arriving twice through different routes is processed once and identity does not depend on a filename anyone can change.
Cache aggressively at the page level, again keyed by content hash. Reprocessing a corpus after a parser change is a routine event, and if page-level recognition results are cached, a rerun touches only the changed stages and costs a fraction of the first pass. Without that cache, every parser improvement carries the full compute bill again, which in practice means improvements stop being made.
Keep the stages separate and materialize between them: fetch, normalize, layout and text recovery, table structure, field extraction, validation, publish. Stage separation is what lets you re-run field extraction over three years of documents in an afternoon without re-recognizing a single page. Merge those stages into one function and every change becomes a full reprocess.
Backpressure matters more than throughput. These pipelines are bursty — a filing season, a bulk backfill — and the failure mode is a queue that grows until something falls over. Bound concurrency per stage, make every stage retryable with a dead-letter path, and keep a per-document status you can query, because "where is document X" is the question operations will ask every day.
Where the engineering time goes on a production build — our default split
Weights sum to 100. Our planning allocation for a first production build. Recognition is the smallest slice and the one that gets budgeted first.
Where a language model belongs, and where it is the wrong tool
The useful division is not model versus rules. It is deterministic where the document is structured, and a model where the document is ambiguous.
If a table has recoverable structure — ruling lines, consistent column geometry, a clean text layer — deterministic parsing is faster, cheaper by orders of magnitude, and exactly reproducible, which matters when someone asks why the number changed between two runs. Sending that table to a model is paying a lot for a worse property.
Where a model earns its cost is the ambiguous residue: a table whose structure did not recover, a footnote that qualifies a value, a concept that appears under a heading nobody has seen before, mapping a filer's idiosyncratic line-item name onto your standard concept. That last one is genuinely hard by rule and genuinely good by model.
The economics follow from the split. Route deterministically first, measure what fraction falls through, and send only that fraction to a model. In corpora we have worked with, the deterministic path typically handles the large majority of pages and the model handles a minority, which changes the blended cost per thousand pages by more than any model choice will. The number to hold in your head is not the price per token. It is the price per checked field, which is dominated by human review time.
Send twenty of your hardest documents and we will tell you what is achievable.
Email twenty documents you consider difficult, plus the fields you need from them, to contact@precisionfederal.com. You get back a written read on which are deterministic, which need a model, the accuracy band we would expect per class, and the error classes specific to your corpus. Two business days. No charge and no meeting.
contact@precisionfederal.comReconciliation is the cheapest ground truth you will ever get
Financial documents check themselves, and most pipelines throw that away. Columns foot to totals. Subtotals sum to totals. The balance sheet balances. Prior-period columns in this filing should equal the current-period columns of the previous filing, except where a restatement is disclosed. Cash flow reconciles to the change in cash.
Each of those is an arithmetic identity you can compute at no cost, and together they catch a large share of real extraction errors without a single label. Better, they catch exactly the class that confidence scores miss, because a model is often confident about a well-formed number that belongs in a different row.
Build reconciliation as a first-class validation stage with a materiality threshold and a residual, not a boolean. A column that foots to within rounding is fine; one off by a single line item points at the line item. Log the residual, route by its size, and you have a diagnostic that names the probable location of the error rather than merely flagging the document. In our experience this is the single highest-return component in the whole pipeline, and it is usually the last one built.
Measuring accuracy in a way that survives contact
Three rules make a measurement worth quoting.
Field-level, not document-level. "Ninety-four percent accurate" over documents is nearly meaningless when a document has three hundred fields of wildly different importance. Measure per field, weight by use, and report the fields that matter separately from the long tail.
Stratified by document class. Native versus scanned, templated versus tail, recent versus historic. A single average conceals a segment that is much worse, and that segment is where your first production incident comes from.
On a gold set built before you saw the output. Hand-label a few hundred documents drawn randomly within strata, and freeze it. A set built by correcting the pipeline's output is a set shaped around the pipeline's blind spots, and it will report an accuracy that field use does not confirm.
Then measure the number that actually matters: escape rate. Of the fields that were wrong, how many passed every validation and reached a consumer? That is the quantity a business cares about, it is usually one to two orders of magnitude below the raw error rate in a well-built system, and it is the one that improves when you invest in reconciliation rather than in a better extractor.
Review economics, stated plainly
No extraction system at scale runs without human review, and the design question is not whether but where. Route by expected cost of error, not by confidence alone: a low-confidence value in a field nobody uses can be accepted, and a high-confidence value in the field driving a published metric can still be worth a second look if it moved more than expected against the prior period.
Then measure the reviewer. Time per field, agreement between two reviewers on a sample, and the rate at which review changes a value. If review changes almost nothing, the routing threshold is too loose and you are paying for confirmation. If reviewers disagree with each other often, the definition of the field is ambiguous, and that is a specification problem no model fixes.
Honest planning numbers from builds of this kind: a reviewer working with a good side-by-side interface handles a few hundred fields an hour, and without one — flipping between a document viewer and a spreadsheet — a fraction of that. The interface is not a nicety. It is the largest single lever on the running cost of the system, and it is routinely descoped in favour of the extractor.
The mistakes we are called in to fix
- Units attached by proximity, so a per-share table under a thousands table is off by a factor of a thousand
- Rows keyed by company and period, so an amendment creates a duplicate nobody can adjudicate
- Restatements overwritten, destroying the as-reported history the pipeline was built to preserve
- One average accuracy number hiding a scanned-document segment that is far worse
- A gold set built by correcting the pipeline's own output, measuring the pipeline against itself
- Everything routed through a model when most pages parse deterministically for a thousandth of the cost
- No page-level cache, so every parser improvement costs a full reprocess and stops happening
- No reconciliation stage, discarding the arithmetic the document provides for free
An eight-week shape to a trustworthy first slice
First production slice, typical sequence
The sequence matters more than the durations. Schema before extractor, gold set before extractor, deterministic before model, reconciliation before scale-up. Every project we have seen go badly reversed at least two of those.
Before you put it in front of users
- Every value carries unit, period, version, page and bounding box
- An unlocatable unit produces a review event, never a default
- Documents are addressed by content hash and processing is idempotent
- Amendments form a version chain; the original stays retrievable
- As-reported and restated values coexist and the consumer chooses
- Reconciliation runs as a validation stage with residuals, not booleans
- Accuracy reported per field and per document class, never as one average
- Escape rate measured and tracked, not just extraction accuracy
- Page-level cache so a reprocess costs a fraction of the first pass
- A review interface that shows the source cell beside the parsed value
Bottom line
Extraction quality is a solved-enough problem for most financial documents; the remaining hard parts are meaning and bookkeeping. What decides whether the pipeline is trusted is whether a number knows its own unit, its own period, its own document version and its own place on the page — and whether the arithmetic the document already contains is being used to check the result. Teams that spend their budget on the extractor and treat the schema, the reconciliation and the review interface as follow-on work end up with a pipeline that is 94% accurate and 0% trusted, which costs more than the manual process it replaced.
Frequently asked questions
For different parts of the job. Where table structure recovers cleanly — ruling lines, consistent geometry, a real text layer — deterministic parsing is cheaper by orders of magnitude, faster, and exactly reproducible, which matters when someone asks why a number changed between runs. Models earn their cost on the ambiguous residue: broken structure, qualifying footnotes, and mapping an unfamiliar line-item name onto your standard concept. Route deterministically first and measure the fall-through.
It varies so much by document class that a single number is misleading, which is the real answer. Native digital, templated documents with clean tables sit far higher than historic scans with dense multi-level headers, and the gap between those two segments is usually larger than the gap between any two tools. Insist on accuracy reported per field and per document class, on a gold set built before the extractor existed, and treat any single headline figure as marketing.
Key extractions on document and document version rather than on company and period. An amendment becomes a new version in a chain, with the original still retrievable, and the default view resolves to the latest. Restated comparatives are a separate case: both the as-reported and the restated value are legitimate statements from different documents, so store both and let the consumer choose — analytics usually wants restated, and anything reconstructing a past view needs as-reported.
Unit scope. A value read perfectly and multiplied by the wrong scale produces a clean, confident, wrong number with nothing to flag it. Make the unit a required attribute with a recorded declaration source, and never default it — an unknown unit should route to review, because a null is a routable event and a wrong assumption is a silent corruption.
Enough that you should design for it rather than hope to eliminate it. Route by expected cost of error rather than confidence alone, then measure time per field, inter-reviewer agreement, and the rate at which review actually changes a value. If review rarely changes anything, your threshold is too loose; if reviewers disagree with each other, the field definition is ambiguous and no model will fix that.