The complaint is never that search is slow
When search inside a product goes wrong, the ticket almost never mentions speed. It says: I searched for the thing, I am certain it exists, and it did not come back. That is a correctness failure, and no amount of faster infrastructure or a larger embedding model repairs it. It gets repaired by deciding what a correct result is, writing that down as a set of graded examples, then building a retrieval stack that satisfies them. Teams reliably do those steps in the opposite order, which is how a search project spends five months on infrastructure and ships something users route around by asking a colleague in chat.
Search is unusual among features because every user grades it, every time, in under a second. They type, they read three rows, and form a permanent opinion. A recommendation engine can be mediocre for a year before anyone proves it; a search box cannot hide for an afternoon. That is also why the work is unforgiving: a change that lifts the average and breaks the twenty queries your power users type every morning gets reported as an outage.
In the systems we have reviewed, the ranking function is rarely the problem. The failures sit in text processing that differs between index time and query time, permission filters applied at the wrong stage, an indexing pipeline that never learned about deletes, and no way to tell whether a change helped. Those are ordinary engineering problems, and they go unworked because "improve relevance" gets scoped as a modeling task and handed to whoever is most interested in models.
You are probably here because
- Someone swears a document is in the index, searches the exact phrase from its title, and gets an empty page back
- A search rebuild shipped, and now part numbers, error codes and invoice IDs no longer find their own record
- Nobody can say whether last week’s change made search better or worse, so every review is settled by whoever brought the best example
None of these is a ranking problem — they come from text processing that differs between index time and query time, and from having no number that says whether a change helped, which is what the sections on the metric, the judgment set and the analyzer are for.
Define what "works" means as a number, before touching the index
Every search conversation improves the moment somebody writes down the metric. Without one, each review is a contest of anecdotes and the loudest anecdote wins. Pick a primary metric, two guardrails, and diagnostics you watch but never optimize directly.
Success at 3. The share of judged queries with a relevant result in the top three. Our default primary: it matches how people read a result page and a non-engineer can reason about it.
NDCG at 10. Graded relevance discounted by position. Better resolution, and sensitive to reordering inside the visible page. Track it across changes.
Recall at 100. The share of judged queries with a relevant document anywhere in the candidate set before reranking. The ceiling on everything downstream, and the number most teams never compute.
Zero-result rate. A guardrail, not a target, and the highest-value backlog in the system.
p95 latency, per stage. The other guardrail. End-to-end p95 hides one stage growing 200 ms while another shrank.
Reformulation and abandonment. Diagnostics. If a user issues a second query seconds after the first, the first one failed.
The query log is the specification
Before anyone proposes an architecture, pull a month of query logs, deduplicate, and sort by frequency. The shape is the same in every system we have looked at: a small set of queries carries most of the volume, then a very long tail seen once and never again. There are usually a few hundred head queries, and if the top twenty return the right thing most of your users are having a good day. The tail is where semantic retrieval, spelling correction and synonyms earn their keep.
Then the most common gap we find. Most systems log the query string and nothing else, which makes every later analysis impossible. Log the query, the parsed form, the filters applied, the result IDs in order, the position of anything clicked, and whether the user reformulated. Without IDs and positions you cannot compute click-through by position, cannot correct for position bias, and cannot reconstruct what a user saw when they complained.
Build the judgment set, and put it in the repository
Two hundred to three hundred queries is enough to be useful and small enough to finish. Sample proportionally across the frequency distribution, then over-sample what breaks retrieval stacks: exact identifiers, abbreviations, misspellings, multi-word entity names, and every query attached to a support ticket. Grade 0 to 3 rather than binary, because "sort of relevant" is a real category. Have two people grade an overlapping subset and write the labeling guide that resolves their disagreements. The guide is worth as much as the labels.
Judgment Set Composition — 250 Queries
A starting distribution. Shift toward whichever category your support queue is loudest about.
Store it as a plain text file beside the code, not a spreadsheet in someone's drive, and write the scoring script to print NDCG at 10, success at 3 and recall at 100 in one line. Wire it into continuous integration. Once a relevance number appears next to every pull request, the argument about whether a change helped stops being an argument.
Start with lexical retrieval, and tune it properly
BM25 is not a legacy technique you tolerate until the embeddings arrive. It is a strong baseline that most teams never tune, then replace on the grounds that it underperformed. Lucene, and therefore Elasticsearch and OpenSearch, defaults to k1 of 1.2 and b of 0.75. Those are reasonable and not right for every corpus: b controls how hard document length is penalized, so if your index mixes 80-word product blurbs with 40-page manuals, b is doing something significant that nobody has looked at.
Field boosts are the other underexamined dial. Boosting title heavily makes exact title matches win and quietly destroys recall for anything discussed only in the body. The fix is the judgment set, not intuition: sweep the boost, plot the metric, pick the value, record why. One more specific worth knowing: in a sharded index, term statistics are computed per shard by default, so the same document scores differently depending on where it landed.
What vectors add, and what they quietly break
Dense retrieval solves a real problem: the user's words and the document's words are different. Someone searches for "cannot log in after password reset" and the article is titled "Authentication loop following credential rotation." No amount of term matching bridges that, an embedding model does, and on tail queries the gain is obvious.
The failure mode is just as sharp, and it lands on the queries users are most confident about. Embeddings encode similarity, and identifiers are similar by construction. A model places ERR-4021 and ERR-4012 close together because they differ by one character, and does the same for part numbers, customer IDs and invoice numbers. A pure vector system demos beautifully and fails the first time a support engineer pastes in a code.
Chunking is where the rest of the value leaks. Split a document into fixed 512-token windows and the middle chunks lose the subject: a paragraph reading "this must be enabled before the migration runs" is meaningless without the title above it. Prepend document title and section heading to every chunk before embedding, align chunks to structural boundaries, and store the parent document ID so you deduplicate at result time instead of showing six fragments of one page.
Budget the memory honestly. Five million chunks at 1,024 dimensions in float32 is roughly 20 GB of raw vectors before the graph index adds overhead, and int8 quantization cuts that close to four times at a recall cost you should measure. The search-time breadth dial on an HNSW index, usually ef_search, trades recall against latency directly, and leaving it at the default before blaming the embedding model is one of the most common wrong diagnoses here.
Hybrid, and the fusion step that makes it work
The answer is almost always both arms. Run BM25 and vector search in parallel, take the top 50 to 100 from each, and merge. Score normalization puts both scores on a common scale and adds them, which is fragile: BM25 scores are unbounded and query-dependent, cosine similarities sit in a narrow band, and the min-max normalization everyone reaches for is computed over whatever landed in this result set.
Rank fusion ignores scores and uses positions. Reciprocal rank fusion, from Cormack, Clarke and Buettcher's 2009 work, gives each document a score of 1 divided by (k plus its rank) in each list and sums across lists, with k conventionally 60. It needs no calibration and is hard to beat without heavy tuning. The dial that matters more than the fusion method is candidate width: if each arm returns 10 documents, fusion has nothing to work with. Retrieve wide, fuse, then narrow, and measure recall at 100 to see whether the widening bought anything.
| Strategy | Catches | Misses | Cost profile |
|---|---|---|---|
| BM25 only | Exact terms, identifiers, codes, rare words, quoted phrases | Paraphrase and any query whose words are not in the document | Cheap, low latency, no GPU, trivial to operate |
| Dense vectors only | Paraphrase, conceptual similarity, long natural-language queries | Identifiers, negation, exact phrases, anything too rare to have been learned | Index memory is the constraint; embedding cost on every write |
| Hybrid with rank fusion | Both of the above, with no score calibration to maintain | Fine-grained ordering inside the merged candidate set | Two retrievals in parallel: the slower arm, not the sum |
| Hybrid plus cross-encoder rerank | Ordering, and moving the right answer from rank 30 to rank 1 | Anything retrieval never returned; recall is unchanged | The dominant latency line; scales with candidates scored |
Reranking, and the ceiling it cannot break
A cross-encoder reads the query and a candidate document together and scores the pair, which is why it beats any embedding comparison and why it cannot be precomputed. That forces the architecture: retrieve wide and cheap, rerank a small set expensively, usually fifty to one hundred candidates. Then the part that gets missed most often. A reranker reorders what retrieval returned, and cannot conjure a document that was never in the candidate set.
Reranking is also where the latency budget goes, so write the budget before you build. Below is the shape we start from for an interactive search box, where 300 ms at p95 keeps the interface feeling immediate. Treat it as an allocation to defend rather than a measurement, because the reranker will consume whatever is left over.
Latency Allocation — 300 ms Budget at p95 (milliseconds)
The two retrievals run in parallel, so the arithmetic is the slower arm plus the serial stages, with headroom left deliberately.
The analyzer is where relevance is really decided
Here is the bug we find more than any other, and it never appears in a design document. The index-time analyzer and the query-time analyzer disagree. One strips hyphens, the other keeps them, and every hyphenated product name silently stops matching. The system returns nothing, nothing is logged as an error, and the ticket that arrives says "search is broken for European customers."
Decide these deliberately, per field, and test them. Stemming collapses "universities" and "university," which you want, and also collapses pairs you did not want collapsed, which the judgment set tells you about. Numbers and units need their own handling, because "10gb," "10 GB" and "10 gigabytes" are one concept to a user and three tokens to a tokenizer. Identifiers want a separate unanalyzed field so an exact match scores exactly. Autocomplete wants its own prefix-token index, not a boost bolted onto the main query.
Assert token output in CI, and remember every analyzer change is a reindex
Both major search engines expose an analyze endpoint, and Postgres shows the output of to_tsvector directly in psql. Write a test that feeds twenty deliberately awkward inputs through the real analyzer and asserts the exact token list: a hyphenated part number, an accented name, a plural, a unit with and without a space, an all-caps acronym, a string with an apostrophe. It takes an hour and catches the class of bug that otherwise takes a week to find. Note the operational consequence: an analyzer change alters how documents were tokenized at write time, so it needs a full reindex, not a config reload.
Synonyms deserve one specific piece of advice: do not start from a general thesaurus. Mine them from your logs. Find sessions where a user searched X, clicked nothing, then searched Y and clicked a result. That pair is a synonym in your domain, discovered from real behavior, and a list built this way is short and high precision.
Send it over and we will tell you what we would change.
Email your analyzer and field mapping configuration, plus ten searches that should have matched and did not, to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comPermissions belong in the filter, not the post-filter
Search indexes are the most common place a permission model leaks, because search shows titles and snippets of documents a user cannot open. The mechanism is usually the same: retrieve the top 10 by relevance, then remove what the user is not allowed to see. Now the page shows three results, the count is wrong, and the removed titles were rendered somewhere before being filtered.
Apply the access filter as part of retrieval so ranking only ever operates on the permitted set. In a vector index that means filtered approximate search, with one caveat: as the filter gets more selective, graph-based search explores further to find enough permitted neighbors, and recall degrades before latency does. Test at your most restrictive realistic filter, not the average one, and write a test asserting a user cannot see a title they are not entitled to open. In a multi-tenant product the tenant identifier goes in the filter and in every cache key, without exception.
Indexing is a pipeline, and it fails like one
Writes must be idempotent and keyed by a stable document ID so a replay after failure does not create duplicates. The delete path needs to exist and needs a test, because "records the user deleted last month still show up in search" is a bug we find in a large share of the systems we review, almost always because deletion was a soft delete in the database that nobody propagated to the index.
Reindexing should never happen in place under live traffic. Build the new index beside the old one, run the judgment set against it, compare, then move an alias atomically for a one-second rollback. Know your visibility lag too: a write becomes searchable only after a refresh interval commonly set to a second, so if a screen creates a record and immediately searches for it, that second is a bug report. Treat freshness as an objective with a number on it.
Choosing the engine
The engine matters less than the six sections above, which is why it comes last. The framing is fit, plus where each option runs out.
| Option | Fits when | Where it runs out |
|---|---|---|
| Postgres full-text (tsvector + GIN) | Data already lives there, corpus in the low millions, one fewer system to operate | Ranking is not BM25, per-field analysis is coarse, heavy faceting gets slow |
| Elasticsearch / OpenSearch | The default workhorse: per-field analyzers, filters, facets, aggregations, mature tooling | Operational cost is real; cluster management and reindexing are ongoing work |
| Typesense / Meilisearch | Product search where typo tolerance and a fast start beat deep control | Complex analysis chains, unusual ranking needs, very large multi-tenant corpora |
| Vespa | Ranking as a programmable layer, tensors, lexical and vector in one engine at scale | The steepest learning curve here; overkill below a certain size and team |
| Vector store (pgvector, Qdrant, Weaviate) | The vector arm of a hybrid stack, or a Postgres you would rather extend than replace | They are not lexical engines; pair them with one rather than pretending otherwise |
Signals beyond the text, and the feedback trap
Click data is the most tempting and most dangerous signal in the system. The top result gets clicked partly because it is relevant and partly because it is on top. Train on raw clicks and you build a model whose main skill is preserving the current ordering, and it will look like it is improving while it slowly freezes. Using clicks as labels requires position-bias correction or randomization in the top positions. For most teams the better first answer is to use clicks as a diagnostic and keep the judgment set as the label source.
Measuring online without fooling yourself
The judgment set gates every change before it ships. Online measurement decides whether the offline gain was real, and interleaving comes first: it blends two rankers into one list for the same user on the same query and attributes clicks back to whichever ranker contributed the item. Because the comparison happens within a query rather than across users, it detects a difference with far fewer impressions than a split test, which matters at thousands of searches per day rather than millions.
A/B testing comes after, with the sample size computed before the test starts rather than watched until it crosses a threshold. Keep the offline harness authoritative for regressions: an online test says whether users liked a change on average, while the judgment set says which specific queries you broke, and the broken query is what produces the angry message from the person who uses your product eight hours a day.
Default Weighting When We Scope a Search Engagement
Weights summing to 100. Reranking sits low on purpose: it is the last lever, not the first.
The mistakes we find most often
- No judgment set, so every relevance decision is settled by whoever produced the best anecdote in the meeting
- Index-time and query-time analyzers that disagree, found when hyphenated or accented terms stop matching
- Lexical retrieval replaced entirely by embeddings, losing every part number, error code and invoice ID query at once
- Permissions applied after ranking, producing short pages, wrong counts, and titles the user cannot open
- Reindexing in place under traffic instead of building a second index and swapping the alias once the numbers check out
- Raw click-through used as a relevance label, training the system to preserve whatever is already on top
- Deleted records still searchable, because the delete path from the source system to the index was never built
Zero results, and the long tail behind them
An empty result page is the worst thing your search can produce, because it gives the user nothing to work with and no path forward. Build a fallback ladder and take the rungs in order: relax strict term matching, apply spelling correction built from your own corpus vocabulary rather than a general dictionary, run the vector arm alone, then fall back to a browse surface with the closest categories.
Then treat the zero-result log as a product backlog, because it is one. Every query in it is a user telling you in their own words about something they expected to find. Some are content gaps, some are vocabulary gaps that become synonym entries, and some are real retrieval bugs. Reviewing that list weekly with an owner assigned to the top entries produces more improvement per hour than almost anything else on this page.
A checklist before you call it done
- A judgment set of at least 200 graded queries in the repository, with the labeling guide that produced it
- A scoring script in CI printing NDCG at 10, success at 3 and recall at 100 for every change
- Analyzer tests asserting exact token output for hyphens, codes, units, accents, plurals and acronyms
- Query logging that captures result IDs, positions, filters applied, clicks and reformulations
- A weekly zero-result review with owners assigned to the top offenders
- Permission filtering applied during retrieval, with a test proving an unauthorized title never renders
- An alias-swap reindex path exercised at least once under production-like load
- A written per-stage latency budget, with p95 measured per stage rather than end to end
A thirty-day plan that gets you a measurable system
Search Improvement Sprint
Thirty days works because the expensive unknowns in search are all measurable inside it. Whether your analyzer is losing terms is a day of work. Whether recall is your ceiling is one script away. Whether a reranker fits the budget is answered by running it on your hardware with your candidate count. What takes six months is arguing about which model to use before anyone has established what the system currently gets wrong.
Bottom line
Search is an evaluation problem wrapped around a text-processing problem with a permissions problem inside it. Build the judgment set first, make the lexical baseline honest, add vectors for the tail without letting them near your exact-match queries, fuse by rank, and rerank only what retrieval actually found. Do that and you get a search box people trust, which is the only measure that ends up mattering.
Frequently asked questions
Build a judgment set of 200 to 300 real queries with graded relevance labels, then track NDCG at 10 as the engineering metric, success at 3 as the reported number, and recall at 100 as the ceiling check. Guard with zero-result rate and per-stage p95 latency.
Not necessarily. A properly tuned lexical engine handles a large share of real queries, and vectors mainly help on the tail where the user's words differ from the document's. If you add them, run both arms and merge by rank rather than replacing term matching, because a pure vector system fails on identifiers and exact phrases.
Usually the analyzer. If text is tokenized one way at index time and another way at query time, matches disappear silently, and hyphens, accents, casing, units and acronyms are the usual culprits. The other frequent cause is a permission filter applied after ranking.
Buy the engine, build the relevance. Nobody should be writing an inverted index or an approximate nearest neighbor library. What no product supplies is your judgment set, your analyzer configuration, your synonym list mined from your logs, your permission model and your ranking rules, and those decide whether the results are right.