Two different problems wearing one name
When someone says structured output is working, they usually mean the parser stopped throwing. That is the easy half, and modern constrained decoding solves it more or less completely. The hard half is whether the values in those fields are the values that were in the source, and nothing about a schema helps with that. We have seen a pipeline report a 99.7% success rate while roughly one invoice in nine carried a total that appeared nowhere in the document. Every one of those records was schema-valid.
So the useful framing is two separate questions with two separate answers. Is it parseable? Use the right mechanism and this goes to essentially zero failures. Is it right? That depends on schema design, on how you represent absence, and on measurement you have to build yourself. This article is mostly about the second question, because the first one is nearly free and almost everyone stops there.
You are probably here because
- Your extraction rate looks excellent and someone downstream says the data is wrong
- A date field sometimes contains an apologetic sentence explaining why there is no date
- You cannot tell a field that was absent from a field that was missed
- A retry-until-it-parses loop is quietly tripling the cost of every hard document
These are all the same root cause: a schema that can express a value but cannot express what the model actually knows.
Three mechanisms, and when each is right
There are exactly three ways to get structured data out of a model, and the differences matter more than the marketing around them suggests.
Constrained decoding. You supply a schema and the decoder is restricted at each step to tokens that can still lead to a valid document. The output is guaranteed to parse and to match the schema's types. This is the right default for anything with a fixed shape, and it eliminates the parse-failure class entirely rather than reducing it.
Tool or function calls. You define a function with a typed parameter schema and let the model call it. Mechanically this is the same constraint applied to the arguments, but it composes with everything else a model does in a turn, so it is the right choice when structure is one part of a larger interaction rather than the whole output. Turn on strict validation, and set the schema to reject unknown properties, or you will find extra fields appearing in arguments and being silently dropped by whatever consumes them.
Ask in the prompt and parse. Free generation with instructions to emit JSON, plus a parser and a repair loop. This is the approach to avoid when either of the others is available. It fails in the least convenient way, which is rarely and non-uniformly: hardest documents, longest outputs, most unusual inputs. Your failure rate is therefore correlated with exactly the cases you most wanted handled.
| Mechanism | Parse failures | Good for | What it costs |
|---|---|---|---|
| Constrained decoding | Effectively none | Fixed-shape records, batch extraction, anything a program consumes | Schema features are restricted; deep nesting and open unions are awkward |
| Typed tool call | Effectively none with strict mode on | Structure inside an agent turn; several possible actions | More moving parts; the tool description is prompt text and changes behavior |
| Prompt and parse | Low single digits, concentrated in hard inputs | Prototypes; shapes the constraint system cannot express | A repair loop, extra latency and cost on precisely your worst cases |
One caution that applies to all three. A constraint system enforces shape, not sense. Ask for a field typed as a number and you will get a number every time, including on documents where no such number exists. The constraint has converted a visible failure into an invisible one, which is a real improvement in uptime and a real regression in trustworthiness unless the schema gives the model somewhere honest to put "not present."
Schema design is where your error rate is decided
Most extraction quality problems are schema problems, and they are cheap to fix before launch and expensive afterward. Five choices carry most of the weight.
Keep it flat. Three levels of nesting with arrays of objects inside objects is harder for a model to fill correctly than the same information in a flat record, and it is harder for you to score. If the domain is genuinely hierarchical, extract in two passes rather than asking for the whole tree at once.
Enumerate wherever you can. A field typed as a free string will contain seventeen spellings of the same concept within a month. A closed enum of eight values will not. When the real world has a long tail, add an explicit other member with a companion free-text field, so the tail is captured rather than mangled into the nearest legal value.
Name fields the way a person would. Field names are instructions. counterparty_legal_name extracts better than name2, and the description attached to each field is read and used. This is the single highest-return edit available and it takes an hour.
Avoid unions and polymorphism. A field that is sometimes a string and sometimes an object is a field you will spend a year defending. Split it into two nullable fields.
Give every field somewhere to say "I could not." This is the one that matters most, and it gets its own section.
Effect on field-level accuracy — our ranking of the levers
Relative impact as we rank it across extraction work. Judgment, not a benchmark; the ordering is the part worth borrowing.
Absence is not null, and null is not one thing
Take an invoice with no purchase order number. There are at least four different states hiding behind an empty field, and they call for four different downstream behaviors.
The document genuinely has no purchase order, which is normal and requires nothing. The document has one but the page is illegible, which requires a re-scan. The model found something it believes is the purchase order but is not confident, which requires a human glance. Or the extraction failed for a reason unrelated to the document, which requires a retry. Encode all four as null and you have thrown away the only information that tells anyone what to do next.
The fix is small. Every field carries a value and a status, where status is a closed enum along the lines of found, not_present, unreadable, and uncertain. It roughly doubles the size of the record and it is worth it every time. It converts an unquantified error rate into a routing decision, and it lets a reviewer see the twenty records that need a look rather than the two thousand that do not.
Do not accept the alternative that appears when the schema has no room for this, which is prose in a typed field. A date field containing "the document does not appear to specify a delivery date" is a parser bug on some future Tuesday, and it happens because the model was given no honest option and did the most helpful thing available.
Require a source span for anything consequential
For each extracted value, also ask for the exact text it came from, and where possible the offsets or the page. Two things happen. Reviewers stop opening the source document to check a number, which is where most of the review time goes. And you gain a free automated check: if the returned span does not appear in the input, the value is suspect regardless of how plausible it looks. That single assertion catches a large share of confident-but-invented values at nearly no cost.
Validate in layers, and put business rules last
Schema validation is the first layer and the weakest. Three more sit above it.
Type and format checks beyond the schema. A string typed as a date should parse as one; a currency amount should be a decimal with a plausible magnitude; an identifier should match its known pattern and, where one exists, its checksum. Constraint systems are not expressive enough to carry all of this, so it lives in code.
Cross-field consistency. Line items sum to the subtotal. The end date follows the start date. The declared currency matches the symbol in the extracted total. These checks find the errors that are invisible field by field, and they are the highest-yield validation you can write in an afternoon.
Grounding. The returned source span occurs in the input document. Numeric values appear somewhere in the text. This is the check that separates extracted from generated, and it is the one that most pipelines lack.
What happens on failure should be decided per field, not globally. A malformed optional field can be dropped with the record retained and flagged. A failed cross-field check on a total should stop the record and route it to a person. A failed grounding check on a consequential value should never be silently retried until it passes, because a retry loop with a validity gate will eventually produce something that passes the gate and is still wrong.
The repair loop, and its real cost
When validation fails, the reflex is to send the output back with the error and ask for a correction. It works often enough to be tempting and it has three properties worth being explicit about.
It doubles or triples the cost and latency of the affected requests, and those are your hard cases, so the average hides it while the tail gets much worse. It converges on passing the validator rather than on being correct, which for a semantic error means the second attempt is a plausible value that satisfies the rule. And it makes your metrics ambiguous unless you record attempts, because a first-pass rate of 82% repaired to 97% is a very different system from one that gets 97% directly, and only the former has a latency problem you have not measured.
Use it, but bound it: one retry, not a loop; record the attempt count on every record; alert on the first-pass rate rather than the final rate; and never repair a grounding failure, because the correct response to "this value is not in the document" is to mark the field, not to ask again more insistently.
Send us the schema and thirty documents.
Email your extraction schema and a sample of real inputs to contact@precisionfederal.com. You get back a written note on the fields we would restructure, the checks we would add, and where we would expect the error rate to sit. One business day, no charge, no meeting.
contact@precisionfederal.comMeasure per field, not per document
The number most teams report is the share of documents that produced a parseable record. It is the least informative number available. Replace it with four.
| Metric | What it tells you | Why the aggregate hides it |
|---|---|---|
| Field-level accuracy, per field | Which fields are actually working | A record with nineteen right fields and one wrong total scores 95% and is unusable |
| False-extraction rate | How often a value is produced where none exists | The most damaging error, and invisible to any completeness measure |
| Abstention rate, per field | Whether the model is using the status enum or guessing | A rate of zero means the honest option is not being taken |
| First-pass validity | Cost and latency you are paying in repairs | Post-repair success rates conceal it entirely |
Building the ground truth for this is the actual work, and there is no way around it. A hundred documents labelled carefully by someone who knows the domain is enough to make every number above meaningful, and it is a two-day task that gets deferred for months. Label the awkward ones deliberately: the multi-page, the badly scanned, the ones in a second language, the ones a colleague argued about. A test set of clean examples will tell you the system is excellent right up until it meets your customers' documents.
Schemas are public contracts once anything consumes them
The moment a schema is written to a database or read by another team, it has the same change discipline as an API. Adding an optional field is safe. Adding an enum member breaks any consumer that switched exhaustively on the old set unless you declared the enum open on day one. Tightening a field from optional to required breaks a producer. Renaming anything breaks everyone.
Two habits make this manageable. Version the schema and store the version on every record, so a query can distinguish "this field was empty" from "this record predates the field." And keep a golden set of inputs with expected outputs that runs against every schema change, every prompt change and every model change, because all three alter the output and only one of them is visible in a diff.
What we see go wrong
- One nullable field standing in for four different states, so nobody downstream can route anything
- A number type on a field that is often genuinely absent, converting a blank into a fabrication
- Success measured as "it parsed", with no per-field accuracy and no ground truth at all
- An unbounded repair loop that converges on satisfying the validator rather than on being right
- Free-text where an enum belonged, producing a normalization project six months later
- No source span, so every review requires opening the original document
- Prose leaking into typed fields because the schema offered no honest way to say "not present"
- Schema changes shipped without a golden set, so a rename silently empties a column
A week to a defensible extraction pipeline
Extraction hardening sprint
Step one is the step that gets skipped, and skipping it makes every later step unmeasurable. Two people, two days, a hundred documents. After that you can tell whether a change helped, which is the entire difference between engineering and adjusting a prompt until the samples look better.
Common objections
Constrained decoding guarantees the schema. Why do we still need validation?
Because the guarantee covers shape and type, not meaning. A field typed as a date will contain a valid date; nothing says it is the date in the document, or that it is before the end date, or that it exists in the source at all. Constraint handles syntax. Format checks, cross-field consistency and grounding handle everything that actually costs you money.
Should we fine-tune a model for extraction instead?
Usually not first. Schema redesign, better field descriptions and a handful of well-chosen examples typically recover more accuracy than fine-tuning, at a fraction of the effort, and they remain valid when you change models. Fine-tuning earns its place when a domain has genuinely unusual conventions, the volume is high enough for the economics to work, and you already have labelled data — which means you did the labelling step anyway.
Is asking for a confidence score worth it?
A self-reported confidence number is weakly informative and poorly calibrated, and treating it as a probability of correctness will mislead a threshold. A coarse status enum with three or four levels is more honest and more useful, because it maps to a decision rather than to a false precision. If you publish a numeric confidence to customers, you owe them a definition and a calibration curve, and both change when the model changes.
Our documents are messy scans. Does any of this apply?
All of it, and the status enum matters more, not less. On degraded inputs the distinction between "not present" and "unreadable" is the difference between a record you accept and a page you re-scan. Grounding checks get weaker, because the text layer itself may be wrong, so lean harder on cross-field consistency and on routing anything uncertain to a person.
Before it goes near a database
- Structure comes from constrained decoding or a strict typed tool call, never a parser
- Every field carries a status alongside its value
- Consequential values carry a source span that is checked against the input
- Enums are closed with an explicit other member and a free-text companion
- Cross-field consistency rules exist and route failures per field
- A hand-labelled set of at least 100 real documents exists
- Field-level accuracy, false-extraction and abstention are reported separately
- Repair is bounded to one attempt and the attempt count is stored
- The schema is versioned and the version is written on every record
- A golden set gates schema, prompt and model changes with a per-field diff
Bottom line
Choose a mechanism that makes parse failures impossible, then spend your remaining effort on the part that mechanism cannot help with. Design the schema so the model can be honest about what it did not find. Check the values against the source rather than against the schema. Measure per field against data a human labelled, and report the false-extraction rate next to the success rate, because that pair is the only honest summary of an extraction system. A pipeline that says "I do not know" on the eight percent of records where it should is worth more than one that is confidently wrong on three percent and silent about which three.
Frequently asked questions
Constrained decoding restricts the decoder at every step to tokens that keep the output valid against your schema, so a parse failure is not possible. Prompting for JSON and parsing the result relies on the model choosing to comply, which it usually does — and the exceptions cluster in long outputs and unusual inputs, meaning your failures land on the documents you most needed handled.
With an explicit status alongside the value, not with null. Distinguish at minimum: found, not present in the source, present but unreadable, and found with low confidence. Those four states lead to four different downstream actions, and collapsing them into an empty field discards the information that tells anyone which action to take.
Because success is usually measured as "a valid record came back," and a constraint system makes that nearly always true. It says nothing about whether the values match the source. Measure field-level accuracy against hand-labelled documents and, separately, the rate at which values are produced for fields that were actually absent — that second number is the one that damages trust.
One bounded retry, with the attempt recorded, is reasonable. An unbounded loop is not: it multiplies cost and latency on exactly your hardest inputs, and it optimizes for satisfying the validator rather than for being correct. Never retry a grounding failure — if a value does not appear in the source, asking again produces a different value that also does not appear in the source.
Around a hundred, chosen deliberately rather than sampled at random, is enough to make field-level accuracy and false-extraction rates actionable. Weight them toward the difficult end: poor scans, unusual layouts, multi-page records, and the cases your team has argued about. A clean random sample will report excellent performance and tell you nothing about production.