Retrieval over documents nobody cleaned

What the estate actually contains

Take one policy document and follow it through a twenty-year-old organization. There is a word processor original on a shared drive. There is a PDF export of that original, made for distribution. There is a scan of the signed copy, which is the only one with the signature and the only one with no text layer. There are three copies in mail archives, sent as attachments to three distribution lists. There is a slide deck that quotes two paragraphs of it without saying so. And there is a superseding version, issued four years later, filed by a different office under a different naming convention. Six of the seven artifacts are genuine. Exactly one is in force. Nothing on any of them says which.

That is the ordinary case, not the pathological one. When a team decides to point retrieval-augmented generation at a document estate like this, the technical conversation usually starts at the retriever: which embedding model, which vector store, what chunk size. Those decisions matter and they are the cheapest part of the build. The expensive part, and the part that determines whether the system is trusted or quietly abandoned six months in, happens upstream in the corpus.

This piece is about that upstream work: what an uncleaned estate does to a retrieval system, why “clean the data first” is a plan that never finishes, and what to build instead. It assumes a reader who is deciding whether to fund this work and wants to know where the money goes.

Four failures that come from the corpus, not the model

Supersession. The system answers using a document that was rescinded, and it answers correctly. The quotation is verbatim, the citation resolves, the record is genuine and the office that issued it existed. Every check a well-built provenance layer performs comes back clean, because span-level provenance proves that a sentence came from a record, not that the record is still in force. This is the most expensive failure on an old estate and the one least likely to be caught in a demo, because demos use questions whose answers have not changed.

Duplicate crowding. The retrieved context is a fixed budget. If eight of the ten passages handed to the model are the same three paragraphs from five near-identical copies, the system paid for ten documents and received the information content of two. Two published results make this worse than it sounds. Position inside the context is not neutral: Liu and colleagues found that model performance is highest when the relevant material sits at the beginning or the end of the input and degrades when it sits in the middle, even for models built for long contexts (Lost in the Middle, TACL 2024). And near-miss material is actively harmful: Cuconasu and colleagues report that the retriever's highest-scoring documents that do not actually contain the answer degrade the model's effectiveness (The Power of Noise, SIGIR 2024). A duplicate-heavy estate produces exactly that population — documents that score high because they are textually close to the right answer, and are the wrong copy.

Permission flattening. An index is a new access path, and it inherits nothing. The original folder had a permission group on it. The mail archive had a mailbox owner. The exported extract had whatever the export destination had. Once all of it is chunked into one store, the only access control that exists is the one deliberately built into the query path. Teams discover this when someone asks a plain question and gets back a paragraph from a personnel file.

Latent sensitivity. Obscurity was doing quiet work. Material sat in a folder nobody browsed, in a scan with no text layer, in an attachment nobody indexed. Retrieval removes that protection in a single afternoon. In a federal context this has a specific edge: Controlled Unclassified Information is defined by category and authority in the registry the National Archives maintains as Executive Agent under 32 CFR Part 2002, and a document's status does not depend on whether anyone remembered to mark it. Marking is a duty under policy such as DoD Instruction 5200.48; it is not the definition. An unmarked scan of a document containing CUI is still CUI, and the retrieval system will surface it just as readily as anything else.

Why “clean the data first” never finishes

The instinct is reasonable and it fails for a structural reason. Cleaning is defined by an end state nobody can specify — there is no point at which a shared drive is done. Meanwhile the estate keeps accreting, the people who know what the 2011 folder names meant keep retiring, and the budget that was supposed to fund the AI system is consumed by a filing project with no visible output. We have watched this specific arc more than once, and the pattern is described in more general terms in our note on why AI projects fail at the data-access step.

The alternative is to stop trying to fix the files and start describing them. An inventory pass computes a small set of facts about every artifact, records them beside the artifact, and never modifies the original. It is deterministic, cheap, runs in a batch, and produces something a records officer can read — which is often the artifact that unlocks the access approval in the first place. Decisions about what goes into the index are then made against classes of files rather than individual ones, which is what makes the work finite.

None of this is new discipline. Litigation support has processed hostile, unstructured, mixed-format collections for twenty years, and the vocabulary is settled: the processing stage of the Electronic Discovery Reference Model covers hash de-duplication, near-duplicate grouping and email threading as routine steps before anyone reads anything. Even the practice of stripping known system files has an authority behind it — the reference hash sets published by NIST's National Software Reference Library. A team building retrieval over a messy estate is solving a problem that a neighboring profession already named. Borrow their steps.

What the inventory pass computes

These are the fields worth carrying. Each one is cheap, each one is used later by either the retriever, the access layer or the evaluation, and none of them requires a judgement call at inventory time.

• Canonical-text hash — hash the extracted text, not the file bytes. The same document re-exported by a different tool has different bytes and identical text, and byte hashing will miss the match every time.
• Near-duplicate cluster — shingle the canonical text and cluster it, so the copy with a changed footer, a stamped date or an added cover page lands in the same group as its siblings.
• Extraction class — born-digital text, scanned with usable OCR, scanned with no usable text, spreadsheet, mail item, presentation. These behave so differently downstream that mixing them in one pipeline is the source of most silent quality loss.
• Extraction confidence, per page — carried forward as data, with a per-region unquotable flag. A page the extractor barely read should not be allowed to supply a quotation.
• Three separate date fields — filesystem, embedded document metadata and dates asserted in the content. Collapsing them into one “date” column destroys the only evidence that resolves version conflicts.
• Authority signal — did this come out of a system of record, or is it a copy someone kept? This single bit does more work in version resolution than any timestamp.
• Access class — the permission group of the source location, captured as data at inventory time. Reconstructing it later is far harder than recording it now.

The output is a table, not a cleaned drive. It answers questions the sponsor has never been able to answer before — how much of the estate is scanned, how much is duplicated, how much has no date evidence at all — and those answers usually reshape the scope before any model is chosen.

Dates lie in three different ways

Version resolution runs on date evidence, and most teams reach for the wrong date because it is the easy one. The three sources are not interchangeable and should never be merged into a single field.

A scan made in 2019 of a memo signed in 2003 carries a 2019 file date, a blank document date and a 2003 content date. Sort by file date and the system will confidently present a twenty-three-year-old memo as the newest thing it has.

Supersession is a records problem with a legal vocabulary

The law is ahead of most estates here. The Federal Records Act's definition of records at 44 U.S.C. § 3301 covers recorded information regardless of form, and it explicitly excludes duplicate copies of records preserved only for convenience. The distinction between the record and the convenience copy is already drawn in statute. What is missing is any mechanism that applied it to the files, one at a time, over twenty years.

There is precedent for classifying by provenance instead of by reading everything. Under NARA's General Records Schedule 6.1, email managed under the Capstone approach is dispositioned by the role of the account holder rather than by the content of each message, with agencies required to obtain NARA's verification before implementing it. That is the same move an inventory pass makes: decide by class and provenance, because deciding by content at estate scale is not affordable.

Two more anchors are worth knowing before scoping this work. OMB and NARA's memorandum M-23-07 set a government-wide deadline of June 30, 2024 for agencies to manage records electronically, after which NARA stopped accepting analog transfers and accepts records only in electronic format with appropriate metadata (NARA's announcement). And NARA's Universal Electronic Records Management Requirements, currently at Version 3 released in June 2023, is a published baseline a buyer can hand a vendor instead of drafting requirements from scratch.

The engineering consequence is that supersession should be modeled as a graph, not inferred from a sort. Edges come from three places: explicit language in the text (supersedes, cancels, rescinds, revision and change numbering), identifier and version parsing where a numbering convention exists, and near-duplicate clusters resolved by content date. Where the graph cannot decide, the right behavior is to return both documents and say plainly that they conflict. That is a product decision, and it is a better product than a system that picks one silently.

The index is a new access path

Two designs work and one does not. Separate indexes per access population work when the populations are few, stable and coarse. Query-time filtering works when they are not, provided the authorization decision is made outside the retriever by something that already knows the user, and provided the filter is a hard constraint on the candidate set rather than a scoring hint. What does not work is filtering the generated answer after the fact. By then the protected text has already been read into a context window, and the only thing standing between it and the user is a model's cooperation.

The control language for this is ordinary. NIST SP 800-53 remains at Revision 5, updated by patch release 5.2.0 on August 27, 2025 (NIST's summary of changes), and its access control family carries the requirements an assessor will ask about. On the CUI side there is a live discrepancy worth stating plainly rather than glossing: NIST published SP 800-171 Revision 3 in May 2024, but Revision 2 remains the version DoD contractors are assessed against for DFARS and CMMC purposes as of this writing. If a vendor tells you the answer is simply “Rev 3,” they have not checked recently. Our note on CUI handling for federal AI covers the boundary questions in more detail.

Where the hours actually go

The chart below is our own allocation of engineering effort across a retrieval build over an uncleaned estate — the share of hours, not of infrastructure cost, and it sums to the whole job. It is here to make one point: on this class of work, the retriever and the generator together are a minority of the effort, and a proposal that inverts this shape is describing a demo rather than a system.

Retrieval design that tolerates a messy corpus

Hybrid lexical and dense retrieval, fused. An estate full of form numbers, contract identifiers, office symbols and OCR misspellings punishes a pure embedding approach: those strings carry meaning that similarity does not preserve. A lexical index catches the identifier, a dense index catches the paraphrase, and the two ranked lists are combined by reciprocal rank fusion, which Cormack, Clarke and Buettcher showed outperforms both the individual systems and Condorcet fusion (SIGIR 2009). It has one parameter and needs no training data, which is why it is the right first move rather than the sophisticated one.

Cluster-aware top-k. This is the single highest-yield change on a duplicate-heavy estate. Retrieve wide, collapse the result set to one representative per near-duplicate cluster, then refill the budget with genuinely different documents. The representative carries the list of its siblings so a user can still see that six offices hold the same memo. Nothing else recovers as much usable context for as little code.

Metadata filters as hard constraints. When the question is about current policy, “in force” is a filter on the candidate set, not a feature in a ranking function. Rerankers are worth their cost but they do not solve supersession, because a cross-encoder scores relevance and a rescinded document is extremely relevant. The choice of store matters less than most vendor conversations suggest; our comparison in vector database selection holds, with the addition that filtered search performance under a large filter set is the property to test on this kind of corpus.

Do not let the generator arbitrate. When two retrieved documents conflict, a model asked for one answer will produce one answer. Conflict detection belongs upstream, in the retrieval layer, where it can be surfaced as a conflict. If the underlying question is really about numbers in tables and registries rather than prose, the honest answer may be that retrieval is the wrong tool entirely.

Evaluating a corpus nobody has read

The standard advice is to build a gold question set. On an uncleaned estate that advice has a hole in it: a gold set is built from documents someone has read, and the whole premise here is that nobody has read them. Two adjustments make evaluation possible anyway.

First, build the question set from the work rather than from the corpus. Collect real questions people actually asked over the last quarter, have a subject-matter expert answer each one the way they always have, and record which record they used. That produces labels grounded in the organization's actual information need, and it produces them at a rate the organization can afford.

Second, stratify by the inventory classes. If a fifth of the estate is scanned material, then a fifth of the evaluation questions must depend on scanned material, or the reported numbers describe the easy part of the corpus and nothing else. This is the failure that makes pilots look better than production.

Two metrics belong beside recall on this kind of corpus, and neither appears in standard retrieval benchmarks. Wrong-version rate is the share of correct-looking answers whose support comes from a superseded record — it is the number that predicts whether anyone will still be using the system next year. Duplicate share of context is the fraction of the retrieved budget consumed by near-duplicates of one another, which is a direct measure of how much of the model's attention you are wasting. Both are cheap once the inventory exists. The broader method for separating retrieval measurement from generation measurement is in our piece on evaluating retrieval quality, and the companion metric for extraction work is the false-extraction rate.

The scanned fraction, and a budget argument worth making

Scans are usually where the estate's most consequential documents live, because the signed copy is the one that was scanned. They are also where retrieval quality collapses quietly: an image-only PDF contributes nothing to an index and produces no error. Our notes on OCR pipelines for legacy documents and document AI for federal PDFs cover the extraction side.

The part worth raising with a sponsor is the budget argument. A scanned PDF with no text layer fails two separate obligations at once. It fails the retrieval requirement, and it fails accessibility: under the Section 508 standards at 36 CFR Part 1194, electronic content must conform to WCAG 2.0 Level A and AA, with PDF/UA-1 (ISO 14289-1) available as the applicable route for PDF documents. Adding a real text layer with correct reading order serves both. Format targets are settled as well — PDF 2.0 is ISO 32000-2:2020, and the preservation profile built on it, PDF/A-4, is ISO 19005-4:2020.

For federal agencies there is a third obligation in the same neighborhood. NARA's rule on digitizing permanent records, 36 CFR Part 1236 Subpart E, effective June 5, 2023, sets standards for digitizing permanent paper and photographic records and specifies the metadata that has to be captured and embedded. The metadata that rule requires overlaps substantially with the metadata a retrieval system needs. One remediation program can satisfy the records obligation, the accessibility obligation and the retrieval requirement together, and that is a far easier case to fund than any of the three alone.

What we would do in the first two weeks

• Run the inventory over the whole estate before selecting any subset — the counts change the scope, and they change it in the sponsor's favor more often than not.
• Report duplication, scanned share and date-evidence coverage as three separate numbers — these are the three that determine cost.
• Identify the systems of record — the fraction of the estate with real provenance is the fraction where version questions have answers.
• Capture the source permission group for every file, as data — this is far cheaper now than after the first index is built.
• Screen for sensitive categories before indexing, not after — the default for an unresolved class is exclusion.
• Assemble fifty real questions from the last quarter's actual work — with the record that answered each one, from the person who answered it.
• Stand up lexical and dense retrieval together, fused, on day one — hybrid is the baseline to beat, not an optimization to reach later.
• Measure wrong-version rate before measuring anything else — it is the metric that decides whether the system gets trusted.

Common objections, and where this argument breaks

Bottom line

Retrieval over an uncleaned estate is a corpus engineering problem wearing an AI project's clothes. The retriever configuration is a week of work with well-understood defaults; the inventory, the version resolution, the access model and the evaluation set are the job. The organizations that get durable systems out of this are the ones that fund the upstream work explicitly, name wrong-version rate as a metric before anything is built, and accept that the deliverable includes a description of the estate as well as a search box. The ones that skip it get a demo that answers the questions someone already knew the answers to.

Frequently asked questions