Retrieval pipelines that survive real documents

Every retrieval demo works

A demo built on forty clean markdown files answers every question you throw at it. That is why these projects get funded, and why so many stall two months later. The same pipeline is pointed at the real corpus: 60,000 files, a large slice of them PDFs produced by a copier, another slice exported from spreadsheets, the rest email threads and slide decks. Recall falls off, answers start citing pages that do not contain the fact, and the first instinct is to swap the model. The model is rarely the reason. The chunk it was handed is.

This article is about the layer between a file on disk and a retrievable unit of text. It is the least interesting part of a retrieval system to talk about and it sets the ceiling on everything above it. No embedding model recovers a table flattened into prose. No reranker repairs a chunk that starts mid-clause because the extractor read the second column first. No prompt makes a citation correct when the page number came from a counter that reset at every section break.

The work splits into six stages that fail in their own ways: census, parse, normalize, chunk, index, gate. What follows is how we build each one.

Take the census before you write any pipeline code

The first deliverable is a census, and it takes about two days. Walk the estate and produce one table: extension, file count, total bytes, median page count, fraction with a usable text layer, median words per page, date range, and the top twenty producing applications from PDF metadata. That last column is worth more than the rest combined. A corpus dominated by Adobe PDF Library and one dominated by a copier's firmware string are two different projects with two different budgets.

Three numbers come out of it and they drive every later decision. What fraction of pages carries real text rather than an image. What fraction comes from the top ten producing applications, because parser work is per-producer and a concentrated corpus is cheap. And what fraction is a near-duplicate, because that is the difference between indexing 60,000 documents and indexing 19,000.

What a PDF actually is, and why reading order is a real problem

A PDF page is not a document. It is a list of drawing instructions: place this glyph, in this font, at this coordinate. There is no paragraph, no column, no heading, and no reading order unless the producer emitted tagged structure, and most did not. Every text extractor is guessing, and the libraries guess differently. pdfminer.six runs layout analysis and sorts by position. PyMuPDF returns blocks roughly in content-stream order. A scanner writing one text block per line hands you the lines in whatever order its engine walked the page.

The failures this produces are specific and repeatable. A two-column page read in raster order interleaves the columns line by line, so every resulting sentence is half of one argument and half of another. Repeated headers and footers land inline, and a chunk boundary falls between "the tolerance shall not exceed" and "Confidential, page 12 of 340". Hyphenated line breaks survive as config- uration, or as a soft hyphen at U+00AD the tokenizer keeps. Ligatures arrive as U+FB01 and U+FB02, so "identify" carries a character the lexical analyzer does not fold, and the keyword half of a hybrid search quietly stops matching a word that appears on every page.

All of it is fixable and none of it is fixed by default. Normalize to NFKC, fold the ligature block, and join hyphenated line breaks when the following line starts lowercase. Detect the repeated header and footer bands by finding text at the same vertical position on more than roughly 30 percent of pages, and strip them. Run column detection before reading anything. Then measure: sample 30 pages, render them as images, and read the extracted text beside the picture. That half hour is the highest-value half hour in the project.

Tables are where the confident wrong answers come from

A table flattened into a text run is the most dangerous artifact in a retrieval corpus, because it produces answers that are specific, well formatted, and false. The header row ends up separated from its values by two hundred characters of other cells. A question about the 2024 figure retrieves a chunk where the 2024 label sits above a column the extractor read out of order. The answer comes back with a number and a page citation, and nobody catches it, because catching it means opening the source page, which is the work the system was built to avoid.

Treat tables as a separate object class from day one. Detect them, extract them with something that understands ruling lines and whitespace alignment rather than raw text position, and serialize each one deliberately. A markdown pipe table reads well and breaks on merged cells. One row per chunk with the header repeated inline, as in Region: West | FY2024 revenue: 4.1M | Variance: -3%, is verbose and retrieves very well because every row carries its own column names. A structured record stored beside the text, with a short description in the index, scales to large tables and answers aggregate questions with a query instead of a language model.

We ship the row-per-chunk form for tables under roughly 40 rows and the structured form above that. Two rules save grief either way. A chunk boundary never falls inside a table, and a table fragment never enters the index without its header, which means carrying the header forward when the table crosses a page break.

Document class	What breaks	What we do about it
Born-digital single-column PDF	Headers, footers, hyphenation, ligatures	Band detection, NFKC normalization, hyphen joining. Cheap, and the default path.
Two-column or technical PDF	Column interleaving, footnotes merged into body, figure captions orphaned	Column detection before extraction; footnotes captured as their own block linked to the page.
Scanned PDF, no text layer	Nothing extracts; or a bad text layer extracts garbage that looks real	Characters-per-page discriminator, then OCR with confidence stored on every chunk.
Spreadsheet exported to PDF	Row and column structure gone; numbers detached from headers	Find the source workbook if it exists. If not, table detection and row-per-chunk serialization.
Slide decks	Fragmentary bullets, substance in speaker notes, text inside images	Slide plus notes as one unit; OCR any slide whose text extraction is under about 20 words.
Email threads	Quoted replies duplicate the same text five times; signatures dominate the index	Split on quote markers, keep the newest message body, strip signature blocks at parse time.

The scanned fraction, and what OCR really costs

Every real corpus has a scanned tail, and the question is not whether to run OCR but which pages get it. Run the cheap discriminator first: characters recovered per page from the existing text layer. A born-digital page of prose returns roughly 1,500 to 3,000 characters, and a pure image page returns zero. The dangerous case is the page returning 50 to 300, which usually means a poor text layer from an old scanner, or a form where only the printed labels are real text and every entered value is an image. Those pages look processed and are not.

On the recognition itself: 300 DPI is the long-standing floor, and 400 rarely pays for itself on typewritten text. Deskew and despeckle first, keep per-word confidence, store the page mean. Tesseract is adequate on clean typewritten pages and cheap at volume, PaddleOCR handles rotation and low contrast better, and the managed document services from the large cloud providers are materially better on forms and tables at real money per page. That price is why the discriminator matters.

Store OCR confidence on the chunk and use it downstream. A chunk whose page scored a mean word confidence of 62 percent should stay retrievable, but the answer layer should not quote it as though it were typed.

Chunking, and why token count is the wrong unit

Fixed-size chunking with overlap is the default in every framework, and it is why so many systems retrieve fragments that read like they were cut with scissors. Five hundred and twelve tokens with fifty of overlap has no idea where a section begins. It separates a numbered clause from its number and a definition from the term being defined.

Chunk on structure first and size second. The order we use: split on the document's heading hierarchy where one exists, split on paragraph boundaries inside a section, merge short paragraphs forward until the chunk clears a floor of roughly 200 tokens, cap between 600 and 800, and never split inside a table, a list, or a code block. Then prefix every chunk with its heading path, so a chunk from a maintenance manual carries 4 Hydraulic system > 4.3 Bleed procedure > 4.3.2 Cold-weather variant in its own text. The prefix costs 15 to 30 tokens and does two jobs: it gives the embedding model context the body lacks, and the keyword index terms the body never mentions.

Make the overlap structural rather than arithmetic. Instead of copying the last fifty tokens of the previous chunk, carry the last complete sentence plus the heading path. The text stays readable, and it removes the duplicate-fragment problem where one sentence is retrieved twice from adjacent chunks and the answer layer reads two copies as two sources.

Chunking strategy	Unit	Where it wins	What it costs
Fixed size with overlap	Tokens	Uniform prose with no structure; fastest to stand up	Cuts across clauses and tables; citations land on the wrong section
Heading-aware	Section, then paragraph	Manuals, policies, specifications, anything with a numbering scheme	Needs a real heading extractor; degrades on documents with none
Row-per-chunk for tables	Table row plus header	Numeric lookups where the header must travel with the value	Verbose; inflates chunk count on wide tables
Parent-child	Small chunk indexed, larger parent returned	Precision on retrieval with enough context for the answer	Two stores to keep consistent; re-ingest touches both
Whole document	File	Short documents under about 800 tokens; memos, tickets, notes	Above that the vector averages too many topics to be specific

The metadata that has to survive to the answer

The chunk record we ship carries a stable document identifier, a content hash, the source path, the page range, the heading path, the document title, an effective date and a superseded-by pointer where versions exist, a permission tag, the extraction method and its version, OCR confidence where the text came from an image, and a flag marking table-derived text. Each field answers a question somebody asks in the first month.

Two fields get skipped and then retrofitted painfully. Extraction method version lets you re-run only the pages a parser fix touches instead of reprocessing the estate. Effective date stops the system quoting a policy replaced two years ago, which is the failure that ends pilots, because a subject-matter expert notices it inside ten minutes of first use.

Duplicates, versions, and the boilerplate tax

Exact duplicates are easy. Hash the extracted text, collapse matches to one document, and keep the list of paths so you can answer why a file is not in the index. Near-duplicates are the real work: the same document saved five times with small edits, email threads where each reply quotes everything before it, forms where most of every instance is the template. Shingle the text, compare with MinHash or SimHash, keep the newest, and point the others at it rather than deleting them.

Then there is the boilerplate tax, the one people miss. An 80-word confidentiality notice appearing on 12,000 pages becomes 12,000 near-identical vectors in one region of embedding space, and any query brushing that vocabulary retrieves a wall of the same paragraph. Strip boilerplate at the parse step, where it is a layout problem with a clean solution, not at the retrieval step, where it becomes a ranking problem you fight forever.

Retrieval on a corpus like this

With the ingest layer honest, retrieval configuration is comparatively routine. Run hybrid: a lexical index with a tuned analyzer alongside dense vectors. Lexical carries exact identifiers, part numbers, and rare terms embeddings smooth away; dense carries paraphrase. Fuse with reciprocal rank fusion, which needs no score calibration between the two and works well at the usual constant of 60. Rerank the top 50 with a cross-encoder and hand 5 to 8 to the answer layer.

One interaction is worth stating plainly, because it catches good teams. A model that accepts 8,191 tokens will happily embed a 6,000-token chunk, and the resulting vector averages so many subjects that it sits near everything and is specific to nothing. Long context windows are for the answer layer. The index wants chunks small enough to be about one thing.

The parse-quality gate

Make parse quality a pipeline stage with thresholds and a failing exit code, not something somebody eyeballs once. Files that fail go to a quarantine bucket with a reason attached. That bucket is what lets you answer "why did it not find that document", which otherwise has no answer and destroys trust faster than a wrong answer does.

The reading-order check is the only one that needs a person, and it needs one for about twenty minutes per parser change. Render a stratified sample, put the extracted text beside the image, and have someone read it. Automating that check has never paid off on a build we have run, because the failure it catches is the kind a machine scores as fine.

Build the evaluation set out of the ugly cases

Between 150 and 300 questions, each with the answering passage marked at the chunk level, is enough to run a real build. Where they come from matters more than how many. Sample deliberately from the tail: pages with tables, pages off a scanner, two-column layouts, documents with a superseded version in the corpus, questions whose answer appears twice with different dates. A set drawn only from clean pages reports that everything is fine while a quarter of the corpus is unusable.

Score three things separately. Recall at 20 says whether retrieval found the passage at all, and it is the number the ingest layer moves. Normalized discounted cumulative gain at 10 says whether ranking put it where the answer layer will see it. Citation correctness, checked by asking whether the cited page actually contains the asserted fact, is the one an executive cares about and the one nobody tracks. Break all three down by document class, because a single average hides the class that is broken.

The set and its scoring script live in the repository next to the pipeline code, not in a notebook on somebody's laptop. It is the asset that survives every other decision. Models get deprecated, vendors get acquired, chunkers get rewritten, and the evaluation set still tells you within an hour whether a change made things worse.

Re-ingest is a first-class command

Parsers get fixed. Embedding models get replaced. Chunking strategy changes about three times before it settles. Each is a reprocessing event, and a pipeline that can only run from empty turns every one of them into a weekend.

Store three layers, not one: the raw file, the parsed output with page coordinates and structure preserved, and the chunks. Storage is cheap next to a second OCR pass, and keeping the middle layer means a chunking change is a re-chunk and a model change is a re-embed, neither of which touches the parser. Key everything on a content hash so an incremental run processes only what moved, and make it idempotent so running twice produces the same index rather than duplicates.

Rough magnitudes: a born-digital PDF parses at tens of pages per second per core, so a million-page corpus is hours rather than days. Local OCR runs around half a second to three seconds per page per core, which is why a discriminator that skips most pages is worth writing. Embedding a few million chunks with a current small model is a modest cloud bill. The dominant first-pass cost is OCR plus human review of the quarantine bucket, and afterward it is re-embedding, which makes the embedding model a two-year commitment worth a comparison run first.

The mistakes we find most often

• Judging the pipeline on questions the team wrote rather than questions users ask, which skips every hard document in the corpus
• Chunking on token counts with no structure signal, then blaming the embedding model
• Letting tables into the index as flat text, which converts a missing answer into a confidently wrong one
• Skipping the character-per-page discriminator, then either OCR-ing everything or trusting a text layer that is noise
• Post-filtering on permissions, which returns a short result list to some users and leaks the existence of documents to all of them
• No effective date on the chunk, so the system quotes a superseded version with total confidence
• An ingest that only runs from empty, making every parser fix a reprocessing event nobody schedules

A checklist before you call ingestion done

• A census table exists, with producing applications and the scanned fraction
• Thirty sampled pages have been read next to their rendered images
• Headers, footers, and signature blocks are stripped at parse time
• No chunk boundary falls inside a table, and no table fragment travels without its header
• Every chunk carries source, page, heading path, effective date, and permission tag
• The parse-quality gate fails the build, and quarantined files carry a reason
• The evaluation set covers scanned, tabular, and superseded documents, and lives in the repository
• Re-chunk and re-embed run without re-parsing, and an incremental run is idempotent

Thirty days to a system you can measure

Thirty days works because everything expensive here is measurable inside it. Whether the corpus parses is measurable. Whether the scanned tail is 6 percent or 40 percent is a query. Whether retrieval finds the passage is a number the evaluation set produces the day it exists. What takes six months is running without any of those and finding out in month five that a third of the pages were never readable.

Bottom line

Retrieval quality on real documents is decided before the first vector is computed. Parse correctly, keep the structure, keep tables intact, triage the scanned pages, chunk on the document's own boundaries, and carry enough metadata that a chunk can account for itself. Put a gate in front of the index and an evaluation set behind the pipeline. Then the model choice becomes a tuning decision with a measurable answer, instead of the thing everyone argues about while the real problem sits in the ingest layer.

Frequently asked questions