Why federal RAG deployments fail quietly
A federal retrieval-augmented generation system rarely fails with an error message. It fails with a fluent, confident, well-formatted answer that happens to be wrong, because the retrieval step missed the passage that held the truth and the language model wrote around the gap. On a commercial help desk that produces an annoyed customer. In a federal program it produces a defensible-looking response to a Congressional inquiry, a benefits determination, or an analytic summary that no one questions until someone checks the source. The generator is very good at hiding a retrieval miss, and that single fact is the reason retrieval quality has to be measured directly rather than inferred from how good the answers sound.
When a demo goes well, the retrieved passages were probably good and the model had what it needed. When a demo goes badly, everyone blames the model. Both instincts are wrong often enough to matter. A large share of the production RAG failures worth naming trace to retrieval, not generation: the right document was in the corpus, the query never surfaced it, and the model filled the hole with plausible language. You cannot fix what you are not measuring, and a single end-to-end "does the answer look right" score measures the wrong thing. A retrieval miss the generator papers over is the most expensive failure in the system, because no one files a bug against an answer that sounds right.
Retrieval evaluation and answer evaluation are two measurements
The most common mistake is to run one evaluation that grades the final answer and call it a retrieval evaluation. Those are separate measurements with separate failure modes, and collapsing them hides the failures that cost the most. Retrieval evaluation asks a narrow question: given a query, did the system surface the passages that actually contain the answer, and did it rank them where the model would use them? Answer evaluation asks whether the generated response is correct, complete, and appropriately hedged. A system can retrieve perfectly and still answer badly through a weak prompt or a distracted model. More dangerous, a system can answer well on easy questions while retrieving poorly, because the model's own parametric knowledge covers the gap until a question gets specific enough that only the corpus holds the answer.
Separate the two and a regression becomes localizable. If answer quality drops after an embedding-model change and retrieval recall drops in lockstep, the fix lives in retrieval, not the prompt. Keep them fused and every regression turns into a guessing game. The discipline is to score retrieval on retrieval terms first, then score the answer conditioned on what retrieval actually returned.
Where retrieval evaluation should spend attention
Relative editorial emphasis, not benchmark results: where evaluation attention tends to pay off on federal corpora.
The three families of metrics answer three different questions, and a mature evaluation reports all three rather than blending them into one number that reassures without informing.
| Metric family | What it measures | The failure it catches |
|---|---|---|
| Retrieval metrics | Whether the right passages were surfaced and ranked high enough to be used | The silent miss: the answer exists in the corpus but never reaches the model |
| Answer metrics | Whether the response is correct, complete, and calibrated to what the passages support | Fluent wrongness, over-claiming, and ducking a question that was answerable |
| Citation metrics | Whether each claim traces to a passage that actually supports that specific claim | Invented or mismatched citations that look authoritative and are not |
Building a gold question set you can trust
Everything downstream depends on a labeled set of questions paired with the passages that should answer them. Synthesize as little of it as possible. A gold set assembled by asking a language model to imagine user questions measures how well the system answers imaginary questions. Build it from three real sources and keep them balanced against each other.
- Real user queries. Pulled from production logs and scrubbed of sensitive data, these are the questions people actually ask, in the phrasing they actually use. Most representative, least tidy.
- SME-authored questions. Written by the people who would read the output, such as analysts, adjudicators, and investigators. They cover the important cases that logs under-represent because users learned not to ask them.
- Adversarial phrasings. The same information need expressed with wrong vocabulary, acronyms, misspellings, and oblique wording. Retrieval that only works on well-formed queries fails the real user.
- Unanswerable queries. Questions the corpus cannot answer. The correct behavior is a clean refusal, and refusal cannot be measured without them.
- Multi-document questions. Answers that require stitching two or more passages from different documents. They expose coverage gaps that single-passage questions never reach.
- Superseded-document traps. Questions whose answer changed when a policy was updated, to check that the system serves the current passage and not the retired one.
Each item carries the passages that count as relevant, a difficulty label, and a version. Label with people qualified to judge relevance; a policy-QA gold set labeled by someone who has never read the policy is not a gold set. Keep it large enough to detect a regression and small enough to run on every change. A few hundred to low thousands of items, stratified by difficulty and by source, is the working range.
Retrieval metrics that matter in practice
The literature offers dozens of retrieval metrics. Four carry most of the weight for federal RAG, and each answers a plain question a program manager can hold in their head.
Recall at k
The plain question: did the answer reach the shortlist the model actually sees? Of the passages that should have been retrieved, how many land in the top k. If this is low, no prompt engineering can help, because the model never saw the answer. This is the first number to watch.
Reciprocal rank
How near the top was the first relevant passage? One divided by its rank, averaged across the set as mean reciprocal rank. Models lean hardest on the earliest passages, so a relevant passage sitting at position nine often behaves like a miss even though recall counted it as a hit.
Multi-document coverage
For questions that need several passages, the fraction of the required passages actually retrieved. A system can look strong on single-passage recall and quietly fail every question that needs synthesis across documents.
Correct refusal rate
On the unanswerable subset, how often the system declines instead of inventing an answer. Report it on its own so a system that refuses everything cannot hide inside a single blended score.
Report these per difficulty stratum and per query source. A blended average across easy and hard questions hides the regressions that matter, because the easy questions dominate the count while the hard questions are where deployments break.
Groundedness and citation accuracy
Retrieving the right passage is necessary, not sufficient. The answer still has to use it. Two checks close that gap. Groundedness asks whether every sentence in the answer is supported by a retrieved passage; a sentence with no supporting passage is a hallucination whether or not it happens to be true. Citation accuracy asks whether each citation points to a passage that backs the specific claim, not merely a passage on the same subject.
Both are checkable mechanically before any person reads the output. Attach a stable identifier to every retrieved passage, require the model to cite those identifiers, then validate every citation token against the retrieved set. A citation to a passage that was never retrieved is either a model error or an injection, and either way it should not render to the user as grounded. For groundedness, an entailment check (does this passage support this sentence) can run as an automated pass with a second model, with disagreements escalated to a person. The target states simply: every factual sentence traces to a passage a reader can open.
The wrong-answer budget
No production RAG system has a zero hallucination rate, and pretending otherwise is how programs get surprised in front of a customer. The honest move is to set a wrong-answer budget with the mission owner before launch. What rate of unsupported or incorrect answers is tolerable for this use, given the cost of a mistake and the human review in the loop? A drafting assistant a lawyer reviews line by line can absorb more than an autonomous determination that ships without a second set of eyes.
The budget is a number the mission owner signs, not a number the engineering team picks quietly. Once set, the evaluation measures against it: this release produces unsupported answers at or below the agreed rate on the gold set, or it does not ship. The budget also drives refusal tuning. A lower tolerated wrong-answer rate means the system should refuse more aggressively at the margin, trading a few answered questions for fewer wrong ones. Naming the budget turns an argument about model quality into a measurable release gate.
How chunking and metadata shape what retrieval can find
Retrieval quality is decided before a query is ever run, in how documents were split and labeled. A chunk that lost its section heading retrieves poorly because the heading carried the vocabulary a query would match. A chunk split through the middle of a table answers no question about the table. When retrieval recall is low, the chunking is often the cause rather than the embedding model everyone reaches for first.
Metadata is the other half. Effective dates, document version, source authority, and classification are not decoration; they are filters that keep retrieval honest. A question about current policy should not retrieve a superseded passage at all, and the way to enforce that is a metadata filter applied at query time. Evaluate chunking and metadata by their effect on the retrieval numbers. Change the chunking, re-run recall at k and multi-document coverage on the gold set, and keep the change only if the numbers move the right way. Chunking is an empirical decision, not a default to accept.
Drift and the discipline of re-baselining
A retrieval evaluation is a photograph, and the subject keeps moving. Three kinds of drift erode it, and each has a control.
Corpus drift. New documents arrive, old ones are superseded, and the gold set slowly stops representing the live corpus. Diff the corpus on a schedule and re-review any gold item whose source document changed.
Embedding-model drift. Swapping or upgrading the embedding model re-embeds the whole corpus and can move recall in either direction. Treat an embedding change like a code change: re-run the full retrieval evaluation and compare against the current baseline before promoting it.
Query drift. What users ask shifts as the system is adopted. Sample real queries each month, route them through the harness, and compare against the gold-set baseline. A widening gap means the gold set needs fresh items.
Re-baselining is the deliberate act of accepting a new reference point after a change, with the old and new numbers recorded side by side. A baseline nobody updated is worse than no baseline, because it reports green while the system has moved underneath it.
Evaluation as an authorization artifact
In federal work the evaluation is not only an engineering tool; it is evidence for the authorization package. A documented gold set, versioned results tied to specific model and index versions, and a harness anyone on the team can re-run map directly onto the assessment and continuous-monitoring controls an ATO expects. The retrieval and groundedness numbers become the measured evidence behind the claim that the system does what the security plan says it does.
Build the harness once so every result is tagged with the model version, index version, gold-set version, and run date, and the same run that gates a release also produces the artifact an assessor reads. Repeatability is the property that matters most. An evaluation a reviewer cannot reproduce is an assertion; an evaluation that runs on demand and returns the same numbers is evidence.
The honest limits of LLM-as-judge
Scoring thousands of answers by hand does not scale, so a second language model scores the first. Used inside its competence, an LLM judge is fast and consistent on subjective dimensions: is the answer clear, is the tone appropriately cautious, does it stay inside the retrieved passages. Used outside its competence it fails quietly, in the same way the system under test does. A judge is weak exactly where the subject is specialized and ground truth is contested, which is where a great many federal questions live.
Two rules keep the judge honest. Calibrate it against a few hundred human-labeled items and track judge-human agreement as its own metric; when agreement falls, the judge, not the system, may be the thing that drifted. And route the cases that carry real consequence to human adjudication: safety-relevant refusals, contested correctness on specialized matter, and anything sitting near the wrong-answer budget line. The judge triages at scale; a qualified person decides where being wrong is expensive. Automated scoring earns trust by knowing what it should not try to score.
Frequently asked questions
Retrieval evaluation measures whether the system surfaced and ranked the passages that hold the answer. Answer evaluation measures whether the generated response is correct, complete, and calibrated. They fail for different reasons and have to be measured separately, because a fluent answer can hide a retrieval miss.
Recall at k (did the answer reach the shortlist), reciprocal rank (how near the top the first relevant passage sat), multi-document coverage (did synthesis questions get all their passages), and correct refusal rate on unanswerable queries. Report them per difficulty stratum rather than as one blended average.
From real scrubbed user queries, SME-authored questions, and adversarial phrasings, balanced with unanswerable and multi-document cases. Label with people qualified to judge relevance, version every item, and keep the set large enough to catch regressions but small enough to run on every change.
An acceptable rate of unsupported or incorrect answers, set with the mission owner before launch based on the cost of a mistake and the review in the loop. The evaluation measures each release against that agreed rate, and the rate drives how aggressively the system should refuse at the margin.
For subjective dimensions with a tight rubric, calibrated against human labels, it scales well. For contested correctness on specialized matter and anything safety-relevant, it does not. Route consequential cases to human adjudication and track judge-human agreement as a first-class metric.
A documented gold set, versioned results tied to model and index versions, and a repeatable harness map onto the assessment and continuous-monitoring controls an authorization package expects, turning evaluation output into audit evidence with no extra work.
Where this fits in our practice
We build federal RAG systems end to end and the evaluation loop that keeps them honest: gold-set construction, retrieval and groundedness measurement, the wrong-answer budget, and the harness that gates each release and doubles as authorization evidence. See our RAG architecture and LLM evaluation posts for how measurement fits with the rest of the stack, and our agentic AI capability for the broader platform context.