The comparison that actually decides the build
Most people frame this as "is a small model good enough?" That is the wrong axis. On a fixed corpus with a fixed output contract, the frontier model usually reads better and the small model usually deploys better. The decision is which of those two properties your program is actually short of — and that is a question about your environment, your data, and your obligations, not about a leaderboard.
Two definitions first, because both terms get used loosely. By small reasoning model we mean a purpose-built system: an open-weight base model, commonly in the single-digit-to-low-tens-of-billions parameter range, adapted to a specific corpus and wired into a retrieval and citation harness, running on hardware the customer controls. By frontier model we mean the largest general-purpose models, normally reached through a provider's API, and in a small number of cases hosted inside a government cloud region.
The honest short version: the frontier model has more general capability and you will feel it immediately. The small model has properties the frontier model cannot give you through an API — you can freeze it, host it where the data already lives, pin the inference stack, and re-derive last year's output byte-for-byte. Whether those properties are worth a capability gap depends entirely on what the output is used for.
The accreditation boundary is the first filter, not the last
In regulated environments this is usually decided before anyone measures accuracy. The DoD Cloud Computing Security Requirements Guide (CC SRG), published and maintained by DISA, defines the Impact Levels that govern what data may be processed where. IL5 covers higher-sensitivity controlled unclassified information and unclassified national security systems; IL6 covers classified information up to SECRET and, per the CC SRG, must be processed in a DoD or federal government community cloud. Whether a given cloud service offering carries a FedRAMP authorization — and at which FIPS 199 baseline — is checkable on the FedRAMP Marketplace, which GSA operates.
That produces a structural asymmetry worth naming plainly. A set of open weights you host is an artifact inside your enclave; it inherits the enclave's authorization the same way any other binary does. A model reached over an API is a separate service, and the authorization has to belong to the provider at the level your data requires. If your data sits above what the provider is authorized for, no accuracy result changes the answer. That single fact decides a large share of these projects, and it is answerable from documents in an afternoon.
Three documents settle the deployability question
The DISA CC SRG for what may be processed at which Impact Level (read the current revision — the control baselines moved with the transition from NIST SP 800-53 Rev 4 to Rev 5). The FedRAMP Marketplace for what a given cloud service is actually authorized to do. Your own enclave's System Security Plan for what you are allowed to add inside the boundary. None of these are AI documents, and all three outrank the benchmark.
The caveat runs the other direction too, and vendors skip it: hosting open weights inside an authorized boundary does not make the system compliant. You inherit the enclave's controls and you own everything you added. Container hardening against the applicable DISA STIGs, audit records that satisfy the NIST SP 800-53 Rev 5 Audit and Accountability (AU) family, configuration management for the weights as a supply-chain artifact, and a documented answer for how model updates are approved. "We run it on-prem" is the beginning of the security story, not the end of it.
What a small purpose-built model actually buys
Reproducibility. Served LLM endpoints are not reliably deterministic even at temperature zero. Thinking Machines Lab's September 2025 write-up "Defeating Nondeterminism in LLM Inference" traces the dominant cause not to floating-point non-associativity, as is usually assumed, but to the fact that production servers batch unrelated requests dynamically, so the batch your request lands in varies run to run. Their remedy is batch-invariant kernels — which you can only apply if you control the server. When you own the stack you can pin weights, kernels, batch composition, and seed. When you call an API you cannot, and no amount of prompt discipline substitutes.
Frozen weights. A model you host does not change under you. A hosted frontier model is a moving target by design: versions are deprecated, defaults shift, safety behavior is tuned. Anything you validated against a version you no longer have access to is a claim you can no longer reproduce. In a program where the accreditation package cites the evaluation, that is a real liability rather than an inconvenience.
Data residency and egress. The strongest argument for a hosted small model is often not technical at all — it is that the corpus never leaves the enclave. That removes an entire category of review, and it removes the recurring conversation about what a provider does with inference traffic.
Cost, stated carefully. The structural difference is per-token variable pricing versus amortized fixed hardware. Which one wins is a function of your sustained volume, your utilization, and whether you already own the accelerators. We are not going to publish a crossover point, because anyone who does is describing their own workload and calling it yours. Build the spreadsheet with your real request volume; it is the one comparison in this article you should not take on anyone's authority.
Latency, with the caveat. Local inference removes the network round trip and the queue you do not control, which usually helps tail latency more than mean latency. But a small model on modest on-prem hardware can absolutely be slower than a large model served from a very large fleet. "Small" does not mean "fast." It means "yours."
Where the frontier model is genuinely the right answer
We would be arguing against our own evidence if we claimed otherwise. There are clear cases where the large general model is correct and a purpose-built one is a waste of a year:
- The corpus is not fixed — if the inputs keep changing shape, generality is exactly what you are paying for, and a fine-tune will chase a moving target.
- Low volume, high variety — the engineering to build, host, and re-validate a small model has to amortize against sustained request volume; at low volume across heterogeneous requests, it does not.
- Broad world knowledge is load-bearing — when correct output depends on facts outside your corpus, retrieval over your corpus cannot supply them.
- Drafting with a human on every line — where a person reviews the output before it counts, the reproducibility argument weakens sharply and raw capability wins.
- Coverage sweeps — when over-production is cheap and a miss is expensive, higher recall is worth a higher false-positive rate.
- No one to operate it — a model your team cannot run, patch, and re-accredit is worse than a model you rent. This is the most common reason a technically correct on-prem recommendation is the wrong recommendation.
And the plainest case of all: if the data is public or sits comfortably at a low impact level, most of the deployability argument evaporates. Use the better model.
Cost, control, and what each side actually gives up
| Dimension | Small purpose-built model, self-hosted | Frontier model via provider API |
|---|---|---|
| Deployability at IL5/IL6 | Weights are an artifact inside your boundary; inherits the enclave's authorization | Requires the provider to hold authorization at your data's level; check the FedRAMP Marketplace and CC SRG posture first |
| Run-to-run reproducibility | Achievable — you control weights, kernels, batching and seed | Not controllable from the client side; dynamic batching varies output |
| Version stability | Frozen until you choose to change it | Provider-controlled; deprecations and defaults shift |
| General capability | Narrower; strong only where you adapted and grounded it | Substantially broader, and it shows on anything open-ended |
| Cost shape | Fixed hardware plus operations; favors sustained high volume | Variable per token; favors low or spiky volume |
| Operating burden | You own patching, monitoring, hardening, re-accreditation | Provider owns the stack; you own the boundary and the logs |
Traceability is an engineering property, not a model property
The line we care most about is this one: a system that reads a body of data and produces a written conclusion, with every statement traced back to the exact record it came from. That is not something you get by choosing a bigger model, and it is not something you get by asking a model to cite.
The public evidence is direct about this. ALCE, the citation-evaluation benchmark from Gao, Yen, Yu and Chen (EMNLP 2023, arXiv:2305.14627), measures whether generated statements are actually supported by the sources the model cites. The authors report that on their long-form ELI5 subset, even the best systems they tested left roughly half of generated statements without complete citation support. Citation-shaped text is not the same thing as attribution.
The design consequence is architectural. Do not ask the model to produce a citation as text. Carry the retrieved span through the pipeline as a structured object, render the sentence from that span, and make an uncited sentence something the system is structurally incapable of emitting. Then the audit question stops being "did the model hallucinate" and becomes "here is the record, go read it." That harness is model-agnostic — it works over a frontier model too — but it is far easier to certify when the model underneath it is frozen and deterministic. Related reading: logging, auditability, and explainability for federal AI.
How we run the comparison
When we are asked to make this call for a customer, the protocol is short and deliberately unglamorous:
- Write the output contract before the evaluation, not after. If nobody has written down what a correct output looks like, no model choice rescues the project.
- Build a blind held-out set, labeled before either system runs against it. Otherwise the benchmark measures how well you tuned, not how well you generalize.
- Run identical inputs at least three times through each system and report the disagreement rate. A system that answers differently to the same question does not have a recall problem; it has an undefined output.
- Report recall and false-extraction rate together, per category. Recall alone is a marketing number — a system can buy recall by simply saying more.
- Run the accreditation path in parallel with accuracy. A winner that cannot deploy in the target environment is not a winner.
- Price both at your real sustained volume, including the engineering hours to operate the self-hosted option.
What we will not do
Scope stated plainly is more useful than claimed breadth, so here is ours on this subject.
We will not tell you the small model wins when your corpus is not fixed. The advantages described above are real on bounded, self-consistent corpora with a stable output contract. On open-ended work they mostly evaporate, and we will say so before the statement of work is written rather than after.
We are not a cloud service provider and we do not hold a DISA provisional authorization. We build inside the customer's authorized enclave. The boundary, the ATO, and the accreditation decision belong to the customer and their authorizing official. Any firm that tells you it can hand you an Impact Level is describing something other than what the CC SRG describes.
We will not run an evaluation whose held-out set we also wrote unblind, and we will not report a recall figure without the false-positive rate beside it. If the honest result is that the approach we proposed came second, that is the result we hand over.
We do not ship a model without a documented way to reproduce its output. If reproducibility is genuinely not required for your use case, say so and we will happily recommend the frontier model, which will probably do the job better.
Common questions on where the line sits
Won't frontier models just get small enough to run locally?
Capable open-weight models keep getting smaller for a given quality level, and that trend genuinely helps this argument rather than hurting it — it lowers the cost of the self-hosted option. But the deployability question is not primarily about size. It is about who holds the authorization and who controls the version. Those are governance facts, and they do not resolve themselves with better hardware.
What about a frontier model hosted in a government cloud region?
That is a real option and it changes the calculus for some programs. It is also the one claim in this space most worth verifying rather than accepting: check the specific service offering, at the specific Impact Level, on the FedRAMP Marketplace and against your component's CC SRG posture. Region availability is not the same as authorization, and authorization at one level is not authorization at the next.
Doesn't fine-tuning make reliability worse?
It can, particularly if you fine-tune to fix something that was actually a retrieval problem. Our default is to exhaust grounding and harness design first and adapt weights only for output format and domain vocabulary, which is where adaptation reliably pays. We treat the choice between the two as a real decision with evidence on both sides — see our RAG versus fine-tuning decision tree.
What about open-weight licensing?
Read the license, not the benchmark. Open-weight models ship under materially different terms — some under standard permissive open-source licenses, others under bespoke community licenses with use restrictions and redistribution conditions. The license, together with any government data-rights terms in your contract, determines what you can actually deliver. This is a contracts review, and it belongs early in the project rather than at delivery.
Frequently asked questions
In practice: an open-weight base model, adapted to a specific corpus, and wired into a retrieval and citation harness that constrains what it can output. The "reasoning" happens in the harness as much as in the weights — the model reads retrieved records and produces a structured conclusion, and the system carries the source span through so every statement stays attached to a record.
When the corpus is bounded and stable, the output contract is written down, the volume is high enough to amortize hosting, and the output has to be reproducible or has to stay inside an accreditation boundary the provider cannot reach. Remove any one of those and the case weakens.
That depends on the specific cloud service offering and its authorization, not on the model. The DISA CC SRG defines what may be processed at each Impact Level, and the FedRAMP Marketplace shows current authorizations. Verify the offering you intend to use, at the level your data requires, rather than reasoning from the provider's brand.
No. It removes the data-egress and third-party-authorization problems, which are often the hardest ones. You still owe container hardening against applicable STIGs, audit records meeting the NIST SP 800-53 Rev 5 AU family, configuration management for the weights, and an approval path for model updates.
Structurally, not through prompting. Retrieved spans are carried as objects through the pipeline, output sentences are rendered from those spans, and the system cannot emit a statement with no record attached. Then pin the inference stack so the same input yields the same output, and retain machine-written logs for every run.