The demo corpus and the real corpus are different problems
Retrieval demos work. Take a couple of hundred well-formed documents, embed them, and questions get good answers within an afternoon. That success is what gets the project funded, and it is also what sets an expectation the real corpus cannot meet. The real corpus is spread across a document management system, three shared drives, an email archive and a wiki. It contains the 2021 policy, the 2023 revision, the 2024 revision with tracked changes still in it, and a summary deck someone made about all three. It contains spreadsheets where the answer lives in a cell, and scanned faxes where it lives in an image. And some of it must not be visible to the person asking.
None of those problems are about embeddings, and none of them are solved by a better model. They are solved by the parts of the system that nobody demos: extraction quality, structure preservation, version and time handling, access enforcement at retrieval, and a measurement that separates a retrieval failure from a generation failure. What follows is where we spend the effort and roughly what it costs.
You are probably here because
- The pilot on a hundred documents was good and the rollout to forty thousand was not
- It answers from a superseded version and sounds completely certain doing it
- It cannot find things by their exact identifier — a part number, a contract code, a person’s name
- Someone asked what happens if a user retrieves a document they should not see, and the room went quiet
Those are four different subsystems — parsing, versioning, lexical matching and access enforcement — and none of them is the model.
Six ways a real corpus differs
Near-duplicates dominate. In corpora we have indexed, a large share of documents are versions, drafts or copies of another document. Naive retrieval returns five chunks that are the same paragraph from five revisions, which wastes the context budget and leaves the model to guess which is current. It usually guesses by position.
Documents have effective dates, and questions have implicit ones. "What is the reimbursement limit" almost always means today's limit, but "what was the limit when this claim was filed" is the same question with a different answer, and both are asked by the same people in the same interface.
Permissions are not uniform. Some material is restricted by team, by client, by matter, by region. An index that flattens the corpus has flattened the access model with it, and retrieval becomes a way to read documents through a search box.
The answer is often in a table. Rate cards, thresholds, schedules, mappings. Table structure carries meaning that disappears when a parser flattens it to a line of numbers, and the resulting chunk is confidently wrong rather than obviously broken.
The vocabulary is internal. Every organization has acronyms, product codenames and a word that means something different inside than outside. General-purpose embeddings have never seen them. Someone asks about "the Q3 uplift" and the retriever has no idea that is a pricing adjustment.
The formats are a zoo. Native PDFs, scans of printed pages, slide decks where content is in text boxes, email threads with quoted history five levels deep, spreadsheets with merged cells, and a handful of files whose extension is a lie.
Parsing sets the ceiling on everything downstream
This is the least glamorous part of the work and the one that decides whether the system is any good. If extraction drops a table, mangles a multi-column layout, or interleaves a repeated header into the middle of a sentence, no retrieval strategy recovers it. The chunk is wrong before it is embedded.
Budget accordingly. On projects with heterogeneous document sets, ingestion and extraction routinely take a third to a half of the total engineering effort, and teams that budgeted a week for it are the ones who end up six weeks late. Measure it directly rather than assuming: pull a stratified sample of a hundred documents across formats, extract them, and have a person check whether the text is faithful and the tables survived. That number is your ceiling, and it is usually lower than anyone expects on the first pass.
| Document type | What breaks | What we do | Rough cost |
|---|---|---|---|
| Native text PDF | Multi-column reading order; repeated headers and footers inside paragraphs | Layout-aware extraction; strip repeating page furniture before chunking | Low |
| Scanned pages | Recognition errors in exactly the numbers that matter | OCR with a per-page confidence score; flag low-confidence pages rather than indexing them silently | Moderate, plus ongoing |
| Tables and spreadsheets | Structure flattened; merged cells; the header row lost two pages back | Keep the table as a unit, carry the headers into every row, render as markdown for the model | High — usually the single biggest line item |
| Slide decks | Text boxes extracted in creation order, not reading order; content only in the image | Position-ordered extraction; a vision pass on slides with little extractable text | Moderate |
| Email threads | Quoted history duplicated across dozens of messages | Thread reconstruction; index the newest message plus a thread summary | Moderate |
| Everything else | The five percent nobody mentioned in scoping | Route to a quarantine queue with a visible count, never silently drop | Small, if it exists at all |
The quarantine queue in the last row matters more than its size suggests. Documents that fail extraction have to be visible, counted and reviewable. The alternative is a system that answers "I could not find anything about that" because the only relevant document failed to parse eight months ago and nobody was told.
Chunk on structure, not on character count
Fixed-size chunking with overlap is the default in every tutorial and it is the wrong default for documents that have structure. It cuts tables in half, separates a heading from the paragraph it governs, and splits a numbered list across two chunks so that item four arrives without the sentence that gives it meaning.
Chunk on the document's own boundaries — sections, clauses, table rows, slides — and fall back to size limits only inside a section that is genuinely too long. Then attach the structural path to every chunk, so a fragment carries the document title, the section heading above it, and the effective date, in text the retriever sees. A chunk that reads "The limit is $50,000." is useless. The same chunk prefixed with its document, section and date is answerable.
Two additions pay for themselves. Keep a small overlap so a sentence spanning a boundary is not lost, but do not use overlap as a substitute for structure. And index at more than one granularity when queries vary: a paragraph-level index for specifics and a section-level index for questions about a whole topic. Deduplicate at retrieval time so both do not return the same text twice.
Where retrieval quality comes from on a real corpus — our default weights
Weights sum to 100. Our starting allocation of effort, not a measurement. Note the last row: embedding choice is real and it is rarely the constraint.
Hybrid retrieval, because exact terms are half the questions
Dense vector search is good at meaning and unreliable at exactness. Ask for invoice INV-2024-0847 and a pure vector search returns thirty invoices that look similar. Ask for a policy by its code, a part by its number, a person by surname, and the same thing happens. Meanwhile classical keyword search handles all of those perfectly and fails completely on a question phrased differently from the document.
Run both and fuse the results. Reciprocal rank fusion is a reasonable default because it needs no score normalization between two systems whose scores mean different things. On corpora full of identifiers, adding lexical search to an existing vector system is frequently the single largest improvement available, and it takes days rather than weeks.
Then rerank. Retrieve a wide candidate set — fifty to a hundred is typical — and use a cross-encoder to reorder it, passing only the top handful into the context. This is where you convert recall into precision, and it is the step that lets you send five passages instead of twenty. A rerank pass adds latency measured in tens to low hundreds of milliseconds and is almost always worth it.
Version and time belong in the query, not in a filter added later
Handle this at ingestion or you will handle it forever in support tickets. Every document gets a family identifier grouping its versions, a version number, an effective date range, and a superseded flag. Retrieval defaults to current-as-of-today and only reaches into history when the question asks for a date or when the user chooses to.
Then make it visible. Every citation shows the document version and effective date, and when a superseded version is used the answer says so. This single piece of interface honesty resolves an enormous share of the trust problems in these systems, because the failure users hate most is not a wrong answer — it is a confident answer from a document that was replaced last spring.
Detecting version families in a corpus that never labelled them is real work. Filename conventions get you part of the way, content similarity gets you further, and neither is sufficient. Expect to build a review queue where a person confirms the ambiguous groupings, and expect it to be worth the time.
Access enforcement is a retrieval-time property
The index is a copy of your documents, and unless you do something deliberate it is a copy with no access model. Filtering results after retrieval is not enough on its own: it makes the answer depend on documents the user cannot see, and it leaks through counts, summaries and phrasing even when the text itself is removed.
Filter before scoring, using the caller's actual permissions, and re-check every retrieved chunk against the source system's access rules before it enters the context. Keep permission metadata on the chunk and refresh it, because permissions change more often than documents do and a stale copy is exactly the kind of quiet defect nobody notices until it matters.
Test it deliberately. Create accounts at each permission level, ask questions whose answers live in restricted documents, and confirm both that the content does not appear and that the system does not reveal the existence of what it withheld. This takes an afternoon and it is the test most likely to be skipped.
Log the retrieved chunk ids on every answer
Store which chunks were retrieved, which were passed to the model after reranking, and their scores. When a user reports a bad answer, this converts an argument into a two-minute check: was the right passage retrieved and ignored, retrieved and outranked, or never retrieved at all? Those are three different bugs in three different subsystems, and without the log every one of them gets attributed to the model.
Send us twenty documents and ten questions and we will tell you what will break.
Email a representative sample — including the ugly formats — and the questions people actually ask, to contact@precisionfederal.com. You get back a short written note on the extraction problems we can already see, whether your questions need lexical matching, and what we would build first. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comMeasure retrieval separately from generation
A single end-to-end quality score cannot tell you which half is broken, and the two halves have completely different fixes. Split the measurement.
Retrieval, measured alone. Build a set of questions with the correct source chunks labelled, and report recall at the number of passages you actually pass through, plus the rank at which the correct chunk appears. If the right chunk is not in the top five, no amount of prompt work will save the answer.
Generation, measured with retrieval held fixed. Give the model the correct passages and check whether the answer is faithful to them, whether every claim is attributable, and whether it abstains when the passages genuinely do not contain the answer. That last case needs its own examples, because a system that always answers scores perfectly until the day someone asks about something that is not in the corpus.
Keep a small set of questions whose answers are deliberately absent. It is the cheapest possible test of the behavior that damages trust fastest.
When retrieval is the wrong shape
Worth saying plainly, since it costs us work. Retrieval over chunks is one architecture and it is regularly applied to problems it does not fit.
If the answer lives in structured data, query the database. "How many claims over fifty thousand were filed last quarter" is a SQL question. Chunking a report about claims and hoping the number surfaces is a worse version of a solved problem.
If the question spans a whole document, do not chunk it. "Does this contract permit assignment" requires reading the whole agreement, including the clause that modifies the clause you found. Put the document in the context and ask, or extract a structured summary per document at ingestion and reason over those.
If the corpus is small, skip retrieval. A few hundred pages fits in a modern context window. It is cheaper to engineer, easier to test, and better on accuracy than a retrieval pipeline over the same material. Add retrieval when the corpus outgrows the window, not before.
If the question requires aggregation, precompute. "What is our average payment term across all vendor agreements" cannot be answered from five retrieved passages. Extract the field from every document once, store it in a table, and query the table.
A six-week build on a real corpus
Retrieval Build
Weeks one and two are the ones under pressure to compress, and compressing them is how projects end up rebuilding ingestion in month four. The fifty labelled questions in particular are what turn every later argument into a measurement.
The mistakes we are called in to fix
- Fixed-size chunking over documents with structure, cutting tables and clauses in half
- Vector search only, so no exact identifier can be found
- No version model, so superseded documents answer as though current
- Chunks with no document, section or date attached, leaving fragments unusable
- Failed extractions dropped silently, producing confident "nothing found" answers
- Permissions filtered after retrieval, or not represented in the index at all
- Tables flattened to text, so the rate card is wrong in a way that looks fine
- One end-to-end score, so nobody can say whether retrieval or generation failed
- No questions whose correct answer is that the corpus does not say
Before you roll it out
- Extraction fidelity measured on a stratified sample by a person
- Tables preserved as units with headers carried into rows
- Chunks follow document structure and carry title, section and date
- Failed documents land in a visible queue with a count, never dropped
- Version families identified; retrieval defaults to current
- Every citation shows version and effective date
- Permissions filter before scoring and are re-checked at the source
- Hybrid lexical and semantic retrieval, fused, then reranked
- Recall at k measured against labelled questions, separately from generation
- Abstention cases exist and are scored
- Retrieved chunk ids logged on every answer
Bottom line
Retrieval on a real corpus is mostly a document-engineering problem wearing an AI label. The embedding model is the part everyone debates and one of the smaller levers. Extraction quality sets the ceiling. Structure-aware chunking with context headers determines whether a retrieved fragment is usable. Lexical search recovers the half of questions that are about exact identifiers. Version and permission handling determine whether anyone can trust the answers. And separate measurement of retrieval and generation is what turns a vague complaint into a specific fix. Do those and a modest model performs well. Skip them and the best model available produces a confident answer from a policy that expired two years ago.
Frequently asked questions
Because scale brings in near-duplicates, superseded versions, formats the pilot did not contain, and permission boundaries. On a hundred clean documents almost anything works. At forty thousand, the retriever is choosing among five copies of the same paragraph from five revisions, and roughly a third of the corpus is in a format the pilot never tested.
Both, fused. Vector search handles paraphrase and fails on exact identifiers; keyword search does the reverse. Reciprocal rank fusion combines them without needing to normalize scores between two systems, and on corpora full of part numbers, codes and names, adding a lexical index to an existing vector system is often the largest single improvement available.
On a heterogeneous corpus, commonly a third to a half of total engineering effort. Tables are usually the largest single line item. Measure it before committing to a schedule: extract a stratified sample of a hundred documents and have a person check the output. That number is the ceiling on everything downstream, and it is normally lower than the team expects.
Group documents into version families at ingestion, record effective dates and a superseded flag, and make current-as-of-today the default retrieval scope. Then show the version and date on every citation, and state it plainly when a superseded document was used. Detecting the families in a corpus that never labelled them takes a mix of filename rules, content similarity and a human review queue for the ambiguous groups.
When the answer is in structured data — query the database instead. When the question needs a whole document read, such as whether a contract permits assignment. When the corpus is small enough to fit in a context window, where retrieval adds cost and error for nothing. And when the question requires aggregation across many documents, which needs extraction into a table you can query, not five retrieved passages.