The demo is not the measurement
Most people who end up buying an AI system did not set out to become experts in model evaluation. They have a product, a customer, a date, and a requirement that arrived late — an impact level nobody mentioned in the first call, an air-gapped enclave, a CUI marking, an accreditation boundary the customer's security officer treats as settled and everyone else is hearing about for the first time. Then a vendor gives a demo, the demo goes well, and the decision is now yours. The uncomfortable part is that a demo is a performance, and a performance is designed. What you watched was a system running on inputs someone chose, scored by a rule someone wrote, with the failures edited out by the ordinary process of preparing to be watched. None of that requires bad faith. It is simply what a demo is.
The good news is that you do not need to become a machine-learning engineer to buy well. You need six questions. Each one has the same property: a vendor who has actually done the engineering can answer it in a few minutes, from documents that already exist, while a vendor who has not will need a week and will come back with an answer that is longer and vaguer than the question. That asymmetry is the whole instrument. You are not testing the model in these conversations. You are testing whether anyone has ever tested the model.
One: what exactly was the evaluation set, and who wrote it?
Every accuracy claim is a claim about a specific set of examples. The number means nothing until you know what that set was. So ask for it in plain terms: how many items, where they came from, who labeled them, what happened when two labelers disagreed, and — the question that matters most — whether the people who built the model had access to those items while they were building it.
The last one is not an accusation. Contamination is usually ordinary rather than careless: the team pulls a sample from the same corpus they tuned on, or the evaluation items were written after someone had already looked at where the model struggled. A set written by the vendor is calibrated to the vendor, in the same way a test written by a teacher is calibrated to that teacher's course. That is why a number from an instrument the vendor built is a different kind of number than one from an instrument they never saw — a distinction we work through in detail in testing on someone else's instrument.
Public benchmarks do not settle this either. They tell you something about general capability and very little about your documents, your vocabulary, and your customer's edge cases. The documentation forms that answer these questions already exist and are widely used: Gebru and colleagues' "Datasheets for Datasets" (Communications of the ACM, 2021) for the data, and Mitchell and colleagues' "Model Cards for Model Reporting" (Conference on Fairness, Accountability, and Transparency, 2019) for the model. Asking a vendor for a datasheet and a model card is a normal request with a standard shape, and the response tells you a great deal before you read a word of it.
Two: what is the wrong-answer rate, not the accuracy rate?
Accuracy is one number covering two very different failures, and the two rarely cost the same. A system that misses something important costs you an escape. A system that raises something that was never there costs you review time, and review time is the budget that actually runs out. Ask for both counts separately, broken out by category, not blended into a single figure and not averaged across categories where one category carries all the consequence.
The structural trap is the base rate. When the thing you are hunting is rare, the size of the review queue is set by the false-positive rate rather than by the headline, and the headline can stay high while the queue becomes unusable. This is the arithmetic behind it:
Why a high accuracy figure can still produce an unusable queue
Take 10,000 documents in which 200 — two percent — actually contain the item you care about. Suppose the system finds 90 percent of them, and separately raises a flag on 5 percent of the 9,800 documents that contain nothing. It returns 180 true findings and 490 false ones: 670 flags, of which roughly 27 percent are worth opening. Every figure here is chosen to show the shape of the problem, and none of it describes any product. The point is structural — as the target gets rarer, the false-positive rate governs the workload, and no accuracy headline will tell you that.
So the follow-up question is a workload question, not a statistics question: at your volume and your base rate, how many items will a human open in a day, and how many of those will be nothing? A vendor who has deployed the system somewhere real will have a view on this. A vendor who has not will change the subject back to the model. We wrote a longer treatment of the second number and why it is so often absent in the false-extraction rate.
Three: what happens when the model is unsure?
This is the question that separates a product from a demo. Every model encounters inputs it should not answer confidently. The design question is whether the system knows that, and what it does about it.
Start with whether there is a confidence score at all, then ask whether it has been calibrated — whether a score of 0.9 actually corresponds to being right about nine times in ten. It usually does not by default. Guo and colleagues showed in "On Calibration of Modern Neural Networks" (ICML, 2017) that modern networks are systematically overconfident, that the problem got worse as networks got larger even while classification error fell, and that post-hoc correction such as temperature scaling is needed to bring the numbers back into line. A raw softmax value is not a probability of being right, and a vendor who treats it as one has not looked.
Then ask what the system does with low confidence. Does it abstain? Does it route to a person? Does it say something quieter, or does it say the same thing in the same tone as when it is certain? For generative systems this is the central failure mode: NIST's Generative AI Profile (NIST AI 600-1, July 2024), the companion document to the AI Risk Management Framework, names confabulation — confidently stated output that is not true — as one of its twelve risk categories, and asks organizations to measure it rather than assume it away. Ask the vendor to demonstrate the unsure path on purpose, with an input they did not select. Watching a system fail well is more informative than watching it succeed.
Four: where does the data go — every copy of it
Ask for a diagram, not a sentence. The sentence is always "your data stays secure." The diagram has to show every place a copy exists: what leaves your network at inference time, what is written to logs and how long they are retained, what is captured for debugging, what telemetry goes back to the vendor, whether any of it is used to improve the model, and who at the vendor can read any of it. Vendors who have built for regulated customers produce this from a drawer. Others draw it live on a call, which is itself the answer.
If a federal customer is anywhere in your chain, this question changes character. OMB Memorandum M-25-22, "Driving Efficient Acquisition of Artificial Intelligence in Government" (April 3, 2025), which rescinded M-24-18 and applies to any contract awarded under a solicitation issued on or after September 30, 2025, directs agencies to address ownership of government data where that data is used to train, fine-tune, or develop an AI system, and to restrict vendors from using non-public agency inputs and outputs to further train publicly or commercially available models without explicit agency consent. It also directs agencies toward terms that prevent vendor lock-in and support interoperability. Its companion, M-25-21, covers agency use and governance and defines a category of high-impact AI carrying additional obligations. Both carve out national security systems, so a defense customer's requirements arrive from a different regime — the impact-level framework rather than the civilian memoranda, which we lay out in FedRAMP versus DoD impact levels in plain language and IL4, IL5, and IL6 for AI workloads.
The practical version for a commercial team whose customer just said "IL5" is narrower than it sounds. Ask: can this component run entirely inside the customer's boundary, with no outbound call, no license check, no telemetry, and no model download at start-up? That single constraint eliminates a surprising share of the market, and it is better to learn it in week one than in the security review. The engineering pattern that survives it is described in headless model deployment into a customer sandbox.
Five: what will the vendor put in writing?
Everything above is a conversation until it becomes contract language. The translation step is where most AI purchases quietly go wrong, because the demo claims and the signed obligations turn out to describe different systems. The federal side has converged on a useful pattern here that commercial buyers can borrow directly: M-25-22 encourages performance-based techniques built on a statement of objectives, quality-assurance surveillance plans, the ability to test systems before award, and ongoing testing and monitoring rights afterward. Stated plainly — define the acceptance test before you sign, and keep the right to re-run it.
- A named acceptance test — the metric, the threshold, the data it runs on, and who runs it, written before award rather than negotiated after delivery.
- A set the vendor does not hold — you keep an evaluation sample the vendor has never seen, and the contract permits you to score against it.
- Explicit data terms — no training, fine-tuning, or model improvement on your inputs or outputs without written consent, with retention periods stated in days.
- Re-test rights over time — the right to re-run acceptance after any model, prompt, or dependency change, because a silent upstream update is a real event.
- An exit that works — what you receive if you leave: artifacts, formats, configurations, and whatever is needed to run or replace the component without a rewrite.
- A remedy that bites — what happens if the number is missed, expressed in something other than a promise to try harder.
Six: what does this vendor refuse to do?
Ask directly what the vendor turns down and why. A firm with no refusals has either not been in business long or is not telling you the truth, because every genuine method excludes something. The answer also tells you whether they understand the regulatory terrain: any vendor claiming a component arrives pre-accredited is describing something that does not exist, since accreditation attaches to a system in an environment under an authorizing official, not to a piece of software in a box.
Independence is the other place to press. If the same firm writes your acceptance criteria, builds the system, and then grades it, you have bought a self-assessment with extra steps. Assessment regimes take this seriously for a reason, and a vendor who volunteers the conflict before you notice it is showing you how they will behave when something goes wrong later.
What we do here, and what we do not
We should be concrete about our own side of that line. Precision Federal builds evaluation harnesses, bias and fairness testing, explainability outputs, privacy engineering, monitoring, and the governance documentation that goes with them — the work described on our responsible AI page. We build the small, deployable models and the headless components that sit inside someone else's boundary, and we will write the acceptance test with you before anything is built.
What we do not do: we do not issue ATOs, and no vendor can — that authority belongs to an authorizing official at the accrediting organization. We are not a C3PAO and cannot assess anyone for CMMC. We will not hold both sides of an evaluation: if we write your independent assessment, we are not also the firm that remediates its findings, and we will tell you which role we are taking before the engagement starts. We hold no facility clearance today and do not perform classified work on classified networks. We are not a training vendor, and we do not write proposals for other firms. And we will never tell you a component arrives pre-accredited, because nothing does. Our fuller scope statement is what we build, and what we decline.
The short version
If you take one page from this into a vendor call, take this one. The middle column is not a sign of dishonesty; it is a sign that the work has not been done yet, which is a different and more recoverable problem — but you should know which one you are buying.
| What to ask | An answer that should slow you down | An answer worth having |
|---|---|---|
| What was the evaluation set? | "We tested it thoroughly." | Item count, source, labeling process, disagreement rule, and whether the build team saw it. |
| Who wrote it? | "Our team, using industry benchmarks." | A named author, a datasheet, and a willingness to be scored on a set you supply. |
| What is the wrong-answer rate? | A single accuracy figure. | Misses and false alarms counted separately, per category, at your base rate. |
| What happens when it is unsure? | "Confidence scores are available." | A calibration method, an abstention threshold, and a demonstrated hand-off to a person. |
| Where does the data go? | "Your data stays secure." | A flow diagram covering inference, logs, retention, telemetry, and internal access. |
| What goes in the contract? | "That is a legal question." | Acceptance test, held-back set, data terms, re-test rights, exit path, remedy. |
Common objections you will hear
"Our evaluation set is proprietary."
Sometimes true, and usually workable. You do not need their items. You need the shape of the set — size, source, labeling process — and the ability to score the system on items you provide. If both are refused, you are being asked to accept a number with no way to check it, which is a decision you can make knowingly but should not make by default.
"We benchmark against the standard public evaluations."
Public benchmarks measure general capability and are useful for ranking candidates. They do not measure performance on your corpus, your vocabulary, or your customer's rare cases, and they are the exact material most likely to have been seen during training. Treat a public score as a filter for the shortlist, never as acceptance evidence. We go deeper on building task-specific harnesses in LLM evaluation for federal use cases.
"A model reviews the outputs, so quality is checked automatically."
Automated judging is a legitimate technique, and it has documented failure modes. Zheng and colleagues, in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS, 2023), reported strong judge models agreeing with human preferences at rates comparable to human-human agreement, while also identifying position, verbosity, and self-enhancement biases. The right question is whether the judge was validated against human judgment on your task, and how often that validation is repeated.
"We will tune it on your data as we go."
Often the correct engineering answer, and always a contract question. If your data improves their model, decide deliberately whether that improvement is yours, theirs, or shared, and write it down. If a federal customer is downstream, the consent and data-ownership provisions described in OMB M-25-22 will shape what is permissible anyway, so it is better handled at signature than at audit.
Frequently asked questions
"Who wrote the evaluation set, and had the build team seen it?" Every performance claim rests on that set. If the instrument was written by the party being measured, the number describes the instrument as much as the system. It is also the fastest question to ask and the hardest to answer vaguely.
Not for these six questions. They are procurement questions about evidence, documentation, and contract terms. You will want technical review before you accept a delivery, but the questions that separate a real capability from a demo are answerable by anyone willing to ask for the artifact behind the claim and to notice when it does not arrive.
Architecture, mostly, and earlier than you expect. The binding question is whether the component can operate entirely inside the customer's authorization boundary with no outbound call at runtime. Many commercial AI products cannot, because licensing, telemetry, or model fetch assumes internet access. Establish that constraint before selection rather than during the security review.
No. An authorization to operate attaches to a system in a specific environment, granted by an authorizing official at the organization accepting the risk. A component can be built to inherit controls, ship with documentation, and reduce the work substantially — but any claim that it arrives pre-accredited is describing something that does not exist.
Not for anything you intend to present as independent. Building, grading, and remediating with one party produces a self-assessment regardless of how careful everyone is. Split the roles, and expect a serious vendor to raise the conflict before you do.