RAG vs fine-tuning: the federal decision tree.

The question people are actually asking

"Should we fine-tune the model or use RAG?" is almost never the real question. The real question is some mix of: "how do we get this model to be accurate on our domain," "how do we keep our data inside our boundary," "how do we make it cite sources," "how do we make it sound like us," and "how do we afford it at production scale." Each of those sub-questions has a different answer, and the architecture falls out of answering them honestly.

This post is the decision tree we walk with federal programs. It is not a theoretical survey. It is what we actually tell agencies who ask us whether they should stand up a fine-tuning pipeline or invest that effort in retrieval.

What each technique actually does

RAG keeps the knowledge outside the model. At query time you retrieve relevant passages from a vector or hybrid store, pass them to the model as context, and the model answers from that context. The model itself is unchanged. The knowledge lives in the index.

Fine-tuning pushes knowledge, behavior, or both into the weights. Supervised fine-tuning (SFT) teaches the model patterns from input-output pairs. Instruction tuning teaches it to follow specific instruction formats. Preference tuning (DPO, KTO) teaches it to prefer certain answer styles. LoRA and QLoRA train a small adapter layered on top of a frozen base model, which is the only fine-tuning variant that makes economic sense on federal programs outside of very large primes.

The two techniques solve different problems. RAG solves "the model does not know this specific fact" and "we need to cite where we got it." Fine-tuning solves "the model does not respond the way we want" and "we need a consistent pattern of output."

Decision tree, top level

flowchart TD
    A([Your federal LLM use case]) --> B{Answers from\na specific corpus?}
    B -->|Yes| RAG1[RAG — document retrieval\nis mandatory]
    B -->|No| C{Citations required\nback to source?}
    C -->|Yes| RAG2[RAG — fine-tuned models\nfabricate citations]
    C -->|No| D{Corpus changes\nmore than quarterly?}
    D -->|Yes| RAG3[RAG — fine-tunes\ngo stale immediately]
    D -->|No| E{Access is user-dependent\nor clearance-gated?}
    E -->|Yes| RAG4[RAG — cannot bake\nper-user access into weights]
    E -->|No| F{Narrow classification,\nextraction, or transformation?}
    F -->|Yes| FT1[Fine-Tune — 7B specialized\nbeats frontier and cheaper]
    F -->|No| G{Specific format or\ntone base model resists?}
    G -->|Yes| FT2[Fine-Tune adapter —\nprompt-forcing is fragile at scale]
    G -->|No| H[Evaluate further —\nRAG plus adapter hybrid often wins]
    style RAG1 fill:#3b82f6,color:#fff,stroke:#3b82f6
    style RAG2 fill:#3b82f6,color:#fff,stroke:#3b82f6
    style RAG3 fill:#3b82f6,color:#fff,stroke:#3b82f6
    style RAG4 fill:#3b82f6,color:#fff,stroke:#3b82f6
    style FT1 fill:#7c3aed,color:#fff,stroke:#7c3aed
    style FT2 fill:#7c3aed,color:#fff,stroke:#7c3aed
    style H fill:#0d9488,color:#fff,stroke:#0d9488
        

1. Does the task require answering from a specific document corpus? Yes: RAG is mandatory. The only question left is whether you also need fine-tuning on top.
2. Does the task require citations back to source? Yes: RAG. Fine-tuning cannot honestly cite; it compresses the training data and will fabricate plausible-looking citations.
3. Does the corpus change more than once a quarter? Yes: RAG. Any fine-tune is stale the week it ships.
4. Is access to the corpus user-dependent (clearance, need-to-know)? Yes: RAG. You cannot bake user-variable access into weights.
5. Is the task a narrow classification, extraction, or transformation? Consider fine-tuning. A specialized 7B model trained on 10K-50K labeled examples often beats a frontier model on narrow tasks, runs cheaper, and is easier to self-host.
6. Does the output need a specific format, tone, or style the base model resists? Consider a small fine-tuned adapter. Prompting for format is cheap until prompt size dominates latency.

When RAG is the right call

• Policy lookup. Agency policies, regulations, SOPs, CONOPS. Churn is continuous, provenance is required, and users have differing authorizations.
• Case file analysis. Investigators, adjudicators, and analysts asking questions about specific records.
• Research assistance. Technical libraries, NIST publications, journal archives, patent databases.
• Compliance assistance. Answering "what control applies" or "what is the reporting requirement" from a living standard.
• Contracting support. FAR/DFARS lookups, past-performance retrieval, clause analysis.

In every one of these, fine-tuning would be actively counterproductive. A fine-tuned model forgets yesterday's policy update, cannot cite its source, and cannot be partitioned by user clearance.

When fine-tuning is the right call

The "both" case: adapter plus RAG

The pattern we most commonly ship on mature federal programs: a small LoRA adapter on an open-weight base model for tone and format, plus a RAG pipeline for facts. The adapter makes the output sound like the agency without prompting for it. The RAG pipeline keeps the content current and cited.

Cost math that is honest

Rough 2026 numbers for planning, not procurement.

Those numbers exclude compliance and ATO work, which dominate federal project budgets and are independent of the approach.

Fine-tuning pitfalls that federal programs hit

RAG pitfalls that send teams toward fine-tuning for the wrong reasons

Teams often reach for fine-tuning because their RAG is failing, when the actual problem is a bad RAG pipeline. Fix the pipeline before changing the approach:

• Poor chunking produces fragmented context. Fine-tuning will not fix retrieval.
• Single-model embeddings on a domain they were not trained on. Swap the embedder before touching the LLM.
• No hybrid search means exact matches on case numbers or statutes fail. Add BM25.
• No reranker, so the top-5 is noisy. Add a cross-encoder.
• Retrieval works but the prompt lets the model drift. Tighten the prompt before tuning the model.

Tooling honest review

The defensibility argument

On a federal program, "why did the model answer this way" is a question you will be asked. RAG makes it answerable: "because we retrieved these passages from these documents and presented them as context." Fine-tuning makes it harder: "because the model learned a pattern during training." For any application where an answer will be read by a decision-maker, a court, an IG, or Congress, the RAG story holds up better. This is not a technical preference; it is a defensibility preference that matters uniquely in federal work.

A recommended starting posture

1. Build a solid RAG pipeline first. Hybrid retrieval, reranking, grounded citations, real eval harness.
2. Measure what actually fails. Is it retrieval? Is it tone? Is it format?
3. If retrieval is the failure, fix retrieval.
4. If tone or format is the failure and prompt engineering is expensive or unreliable, train a small LoRA adapter.
5. Only consider full fine-tuning when narrow-task performance, self-hosting economics, or latency floors force the issue.
6. Whatever you ship, gate it on a regression eval that mirrors real usage.

Where this fits in our practice

We build RAG platforms, train LoRA adapters, and design the eval harnesses that tell you which to use when. See our RAG architecture and LLM evaluation posts for the rest of the stack.

FAQ