Search over your own documents

Four products, one sentence

The first useful thing to do is decide which of these you are asking for, because they differ by a factor of five in effort and they fail in different ways.

Find the document. Somebody knows the thing exists and cannot find it. This is classic search, the cheapest to build, and often the highest-value per dollar. A good full-text index with sensible filters solves it, and no model is required.

Find the passage. The document is two hundred pages and the answer is one paragraph. This needs retrieval down to the section level and it is where semantic matching starts to earn its cost, because people ask in their words and the document uses its own.

Get an answer with citations. A written answer assembled from several passages, each one linked back to its source. This is the product most people picture, and it is the one with the failure mode — it can produce a fluent answer that no source supports.

Turn every document into a row. Not search at all. This is extraction: pull the same fields out of ten thousand contracts and put them in a table. Different build, different measurement, and usually more valuable to a business than the chat interface people ask for first.

Ask which questions people actually need answered today, in their words, and the category usually becomes obvious in ten minutes. It is worth those ten minutes, because building the third when you needed the first is the single most expensive mistake in this area.

The corpus is the project

Here is the part that surprises buyers. In a typical engagement of this kind, the retrieval design and the model integration are a minority of the work. The majority goes into getting the documents into a state where retrieval is possible at all.

Scanned pages. A meaningful share of most corpora is images of text rather than text. Text recognition on a clean modern scan is very good; on a faxed page from 2009 with a stamp across it, it is not. You will need a quality check that flags pages where recognition failed, because a page that silently extracts as gibberish is worse than one that is missing — the missing one gets noticed.

Tables. Most extraction pipelines flatten a table into a line of text, which destroys the relationship between a row label and its number. If the answers people need live in tables, that has to be handled deliberately, and it is one of the larger line items.

Structure. Headings, numbered sections, appendices, and the fact that a defined term on page 3 governs a clause on page 180. Preserving structure during ingestion is what allows an answer to cite “Section 7.2” rather than “somewhere in this file.”

Duplicates. The same document exists as a draft, a signed version, a scan of the signed version, and an email attachment. Left alone, the top five results are five copies of one thing, and the user concludes the search is broken.

Budget accordingly. Ingestion and cleanup is commonly half to two-thirds of the total effort on a first build, and a vendor whose estimate does not reflect that has either not looked at your documents or is planning to discover it later on your schedule.

Versioning is where the wrong answers come from

This deserves its own section because it is the failure that damages trust fastest and it has nothing to do with model quality.

Your document set almost certainly contains contradictions. The 2022 policy and the 2025 policy both say what the reimbursement limit is, and they disagree. A retrieval system with no notion of recency or authority will find both, and the answer it produces will confidently state one of them, sometimes the old one, with a citation that looks perfectly legitimate. The user checks the citation, sees that the document really does say that, and has no way to know it was superseded eighteen months ago.

There are three defenses and a good system uses all three. Attach effective dates and supersession relationships as metadata during ingestion, so the retriever can prefer the current version. Show the date and source prominently in every answer, so a person can see they are reading a 2022 document. And when the retrieved passages disagree, say so rather than picking — “two documents address this and they differ” is a genuinely useful answer and it is the honest one.

If your corpus has no reliable dates and no supersession record, that is not a retrieval problem, it is a records problem, and it should be fixed at ingestion or accepted openly as a limit of the system.

How the retrieval part actually works

In plain terms, without the vocabulary that usually surrounds this.

Chunking. Documents are split into pieces small enough to retrieve precisely. Split too small and a piece loses the context that makes it meaningful; too large and the answer is buried in noise. Sensible defaults land somewhere in the range of a few hundred to a thousand words with some overlap, and the right answer depends on your documents rather than on a rule. Splitting along the document's own structure — sections, clauses, headings — beats splitting by length nearly every time.

Two ways of matching, used together. Keyword matching finds exact terms: a part number, a person's name, an unusual acronym. Semantic matching finds meaning: someone asks about “time off” and the policy says “annual leave.” Each fails where the other works, and running both and merging results is the standard design because it is reliably better than either alone. Systems that use only semantic matching lose exact identifiers, and that failure is very visible to users.

Reranking. The first pass returns perhaps fifty candidates fast; a slower, more careful model reorders them and keeps the best handful. This is one of the highest-value additions available and it is often skipped. It typically adds a modest amount of latency and improves what reaches the answer step substantially.

Answering with citations. The kept passages are given to a model with an instruction to answer only from them and to cite. Two behaviors matter more than fluency: the answer must be traceable to a specific passage, and the system must be willing to say it did not find anything. A system that never says “not found” is not confident, it is unfalsifiable.

Permissions, and the leak that is easy to build

If everyone who will use the system can already see every document, skip this section. Almost nobody is in that position.

The mistake is retrieving first and filtering after. Two things go wrong. Restricted content can appear in a generated summary even when the source link is hidden, because the model already saw the passage. And a user who is filtered down to nothing gets an empty result that still tells them the document exists, which is itself a disclosure in some settings.

The correct design is to filter at query time, using the same permission source the document system uses, so that a restricted passage is never a candidate in the first place. That has a cost: permissions change, and a stale copy of who-can-see-what is a leak waiting to happen. Plan for permission data to refresh on a schedule, or be read live, and test it with a real account that should not see certain things.

Salary documents, personnel files, board material, deal rooms and anything involving a named individual are the common cases. It takes one incident to end a rollout.

How to tell if it works

Insist on this before any build starts, because it is what converts “the demo felt good” into something you can hold a vendor to.

Build a question set. Fifty to a hundred and fifty real questions, collected from the people who will use the system, each with the correct answer and the document it comes from, written down by a person who knows the material. This takes a few days of somebody's time and it is the most valuable artifact in the project. It also outlives any vendor: it is how you compare a replacement to the incumbent.

Measure two things separately. First, did the right passage make it into the candidates at all — retrieval either found it or it did not, and if it did not, no model can save the answer. Second, given the right passage, was the answer correct. Teams that measure only the final answer cannot tell which half is broken, and spend weeks changing prompts when the problem is chunking.

Measure the refusals. Include questions the corpus genuinely cannot answer. The system should say so. A system that scores well on answerable questions and invents answers to unanswerable ones is not ready, and this is exactly the property a demo will never reveal.

Watch what people actually ask. Log queries, log the ones that returned nothing, and read them weekly for the first two months. Real queries are shorter, stranger and more specific than anything anyone imagines during design, and the zero-result log is the best backlog you will get.

Timelines and costs

Rough shapes, assuming the documents are accessible and someone on your side can answer questions about them.

Two to four weeks gets a useful pilot over a bounded corpus — one department, one document type, a few thousand files — enough to tell whether retrieval quality is there and whether people will use it. Do this first, always.

Six to twelve weeks is a realistic range for a production system over a messy real corpus with permissions, versioning, an evaluation set and an ingestion pipeline that keeps running. Longer if the documents are mostly scans, or if the answers live in tables.

Recurring cost has three parts: hosting the index, which for corpora in the tens or low hundreds of thousands of documents is usually modest; embedding new and changed documents, a small ongoing amount; and per-query cost, which is the one that scales with adoption. A query that retrieves several passages and generates an answer typically costs a fraction of a cent to a few cents depending on model choice and how much text is sent. That is cheap per query and worth watching per month, because success is what makes it grow.

When you should not build this

Three honest cases.

If your documents already live in a system with decent search — and several document platforms have added competent question-answering — try that first. It will not be as good as a purpose-built system over a properly prepared corpus, and it may be good enough, and it costs a fraction of a project.

If the corpus is a few hundred documents, a good full-text index plus a sensible folder structure will beat a retrieval system on both cost and reliability. Semantic search earns its complexity at scale and at variety.

And if the questions people are really asking are about numbers — how many, what was the total, which customers — then the answer is not document search. The answer is getting that data into a database, because a question with an exact answer deserves an exact one.

The mistakes we see most

• No evaluation set, so quality is judged by whoever demoed it last
• Semantic matching only, so exact identifiers and names stop being findable
• No versioning, so superseded documents answer current questions
• Permissions filtered after retrieval rather than at query time
• Fixed-length chunking that cuts clauses and tables in half
• Answers without citations, which cannot be checked and therefore are not trusted
• No refusal behavior, so the system invents rather than reporting nothing found
• One-time ingestion with no pipeline, so the index is stale within a quarter
• Scanned pages never quality-checked, so a portion of the corpus is silently unreadable

Before you start

• You know which of the four products you are building
• Someone counted the documents, the file types, and the share that are scans
• An evaluation set of real questions with known answers exists
• Every document has a date, and superseded versions are marked
• Permission filtering happens at query time, tested with a restricted account
• Keyword and semantic matching are both in the design
• Every answer carries a citation to a specific passage
• The system has a defined way to say it found nothing
• Ingestion is a running pipeline, not a one-time load
• Query logs, including zero-result queries, are captured from day one

Bottom line

Decide which of the four products you need before anyone writes code, because they are not the same project. Expect most of the effort to go into the documents rather than the model. Handle versioning explicitly, or the system will answer current questions from superseded files and lose the room's trust in a week. Filter permissions at query time. Build the evaluation set first and measure retrieval and answering separately. Start with a bounded pilot in two to four weeks. And if the question people keep asking has an exact numerical answer, put the data in a database instead — that is a better outcome and a cheaper one.

Frequently asked questions