The question changes when somebody has to sign
For most of a project's life, evaluation answers an engineering question: is this version better than the last? Then the system moves toward a governed workflow — a clinical step, a benefits determination, a filing that gets examined, an assessment delivered under contract — and the question changes shape without anyone announcing it. It is no longer "is the model good." It is "can the person accountable defend the decision to use it, to somebody unsympathetic, a year and a half from now, with the original team gone." That second question is mostly about records.
A compliance owner — quality, legal, privacy, risk, a security officer, an agency AI governance office — is usually not trying to slow you down and is not equipped to audit your architecture. They are establishing something narrower: that the result was produced under rules written before anyone knew what the result would be, on data resembling what the system will actually see, scored by a procedure a stranger could re-run.
Write the pass criteria before you run anything
The largest difference between an evaluation that survives review and one that does not is whether the acceptance threshold existed before the number did. A threshold chosen after seeing the results is not a threshold; it is a description. That is why "we reached 94%" reliably draws the question "against what target, set when, by whom?"
Writing criteria first forces decisions the team has been deferring. What decision does this output feed, and who acts on it? Which error direction carries the consequence — regulated workflows are rarely symmetric, since a missed adverse finding and a spurious one land on different people at different cost, so each needs its own threshold. And what counts as a hit: an exact value, an overlapping span, the right field populated, or a human judging two outputs equivalent? Loosening that match rule raises every number without changing the system.
| Element of the criteria | Fixed before the first run | Why the reviewer asks for it |
|---|---|---|
| The decision the output feeds | Named step in the workflow, and the role that acts on it | Establishes what a wrong answer costs and who absorbs it |
| Directional thresholds | A separate acceptable rate for each error direction | A blended score can hide the error the regime cares about |
| Population | The sampling frame, collection window, and exclusions | Tells the reviewer which cases the number applies to |
| Slices that must pass | Named in advance, with the required count in each | Distinguishes "reported" from "passed" — different claims |
| Match rule | Hit, miss, partial, and not-applicable, as implemented in code | Loosening it moves every metric with no change to the system |
| The failure path | What actually happens to a case the system gets wrong | The control that makes the residual error rate tolerable |
That last row is not decoration. For federal deployments, OMB Memorandum M-25-21 (April 3, 2025), which replaced M-24-10, sets minimum risk management practices for what it defines as high-impact AI — among them pre-deployment testing, an AI impact assessment, ongoing monitoring, adequate human training and oversight, remedies or appeals, and feedback from users and the public.
The average is a summary; the slices are the evaluation
Aggregate accuracy is the number most likely to be true and least likely to be useful, because no workflow experiences the average. And slices are not only demographic; treating them as only demographic is how teams miss the ones that bite. In document-heavy workflows the slices that separate performance are structural: born-digital versus scanned files, records from before and after a form redesign, the source system a record arrived through, unusually long or terse inputs, and the rare class that carries most of the harm.
The clearest template comes from medical devices. In the joint FDA, Health Canada and MHRA guiding principles for Good Machine Learning Practice (October 2021), Principle 4 requires training and test datasets to be appropriately independent, with all sources of dependence — including data acquisition and site factors — considered and addressed. Principle 8 asks for statistically sound test plans generating performance information independently of the training set, with attention to important subgroups, the operating environment, measurement inputs and confounding factors.
Illustrative arithmetic — worked example, not a measured result
Then comes the question that sets the labeling budget: how many items per slice. Suppose a slice holds 40 items and the system fails none. That is not a zero failure rate. Hanley and Lippman-Hand's rule of three (If nothing goes wrong, is everything all right? Interpreting zero numerators, JAMA 1983;249:1743–1745) holds that when none of n observations show the event, you can be about 95% confident the true rate is at most 3/n — here, roughly 7.5%.
A spotless run on 40 items is therefore consistent with a failure rate near one in thirteen. Pushing that bound to about 1% takes on the order of 300 items in the slice with zero failures.
Comparison has the same problem from the other side: to claim this version beats the last, the set must resolve a difference that size. Card and colleagues found underpowered experiments widespread, with small test sets unable to detect the differences being claimed (With Little Power Comes Great Responsibility, EMNLP 2020, arXiv:2010.06595). A slice too small to pass is also too small to compare.
Adversarial cases, including the boring ones
Two categories belong in the set, and teams usually build only the first. The hostile category is what people picture: inputs crafted to steer the model, including instructions smuggled inside retrieved documents — a live concern for anything retrieving over a corpus you do not fully control (the defensive side is its own subject).
The second is duller and produces more real incidents: the out-of-scope question the system should decline, two source documents that contradict each other, a record with the critical field missing, a superseded policy beside the current one, the ambiguous case a trained human would escalate. Appropriate abstention is a measured property here, not a fallback — a system that answers everything has no abstention behavior to score, and that is a defect. For retrieval systems the measurable form is refusal appropriateness alongside answer faithfulness and citation accuracy, which is its own body of technique.
Whatever the set holds at launch, it should grow in one direction only: every production failure becomes a permanent case, dated, with its circumstances attached. That regression set is the cheapest institutional memory a program has, and it shows a reviewer that the team learns in writing.
Human review is an instrument, and instruments need calibration
Most regulated evaluations end with people reading output and rendering judgment, and that is where much quietly weak evaluation happens, because the reviewer's judgment gets treated as ground truth without ever being measured. Three practices fix most of it. Blind the reviewer to the source wherever format permits. Double-annotate part of the sample and measure agreement — Cohen's or Fleiss' kappa, alongside raw percentage agreement. Fix the adjudication rule and the sampling plan in advance, stratified across the slices the criteria name; a convenience sample of interesting cases measures interestingness.
The reason to measure agreement is a ceiling. If two qualified reviewers agree with each other only seventy percent of the time, no model can be measured above that with this instrument. When agreement is low, sharpen the written instructions and re-run, or accept the ceiling and say so — and those instructions belong in the record, since they define the measurement as surely as the scoring script does. The familiar bands, roughly 0.61 to 0.80 as substantial and 0.81 upward as almost perfect, come from Landis and Koch (Biometrics 1977;33:159–174), who offered them as convenient benchmarks, not a law of nature.
Where a language model does the grading — increasingly common, and legitimate — treat it as a measuring device with known defects. Zheng and colleagues report position bias, verbosity bias, self-enhancement bias and limited reasoning among a model judge's limitations, alongside their finding that a strong judge can exceed 80% agreement with human preferences on their data (Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks Track, arXiv:2306.05685). Name the judge and its version, randomize answer order, disclose when it shares a family with the system under test, and validate against human labels first.
Pin everything, or you measured a moment
"The system" is not the model. Before the first scored run, freeze and record the base model version string and date, the prompts (hashed, not paraphrased in a slide), the retrieval index snapshot, chunking and embedding versions, decoding parameters and seeds, the guardrail configuration, library versions, and — if latency is being claimed — hardware and concurrency. Anything unpinned is a variable in an experiment nobody is controlling.
Hosted endpoints deserve attention, because a stable product name does not imply a stable system underneath; Chen, Zaharia and Zou documented measurable behavior differences between snapshots of the same hosted models a few months apart (How Is ChatGPT's Behavior Changing over Time?, arXiv:2307.09009). Write down which changes force a re-run, and wire the evaluation to fire on each rather than quarterly by memory.
Order of operations — the sequence is the control
The records that make the result defensible later
The audit happens later, and later the people are gone. What survives is what was written down while the work was fresh, which is why the record gets designed at the start. The clearest statement of what a reviewer wants lives in a records rule rather than an AI framework: under 21 CFR 11.10(e), systems handling FDA-regulated electronic records must use secure, computer-generated, time-stamped audit trails that independently record the date and time of operator actions creating, modifying or deleting records; changes must not obscure previously recorded information; and the documentation must be retained at least as long as the records themselves. Part 11 binds only where it binds — but attributable, contemporaneous, non-obscuring and retained is what every regime asks for in its own vocabulary.
The federal framing layers on the same idea. NIST's AI Risk Management Framework 1.0 (NIST AI 100-1, January 2023) organizes the work around Govern, Map, Measure and Manage, with the Generative AI Profile (NIST AI 600-1, July 2024) adding guidance specific to language models — the practical mapping is its own subject. The audit control family in NIST SP 800-53 sets what gets logged and retained on federal systems, and GAO's accountability framework (GAO-21-519SP, June 30, 2021) organizes oversight around governance, data, performance and monitoring.
What to file, and keep
- The dated protocol, accepted before the first scored run
- The dataset manifest — provenance, collection window, sampling method, exclusions, and a hash of the frozen set
- The label definitions, the written annotator instructions, and the measured agreement between annotators
- The scoring script, including the match rule exactly as implemented
- Version strings for model, prompts, index, embeddings, guardrails, and libraries
- The complete run log, including runs that went against the system
- Every deviation from the protocol, with its date and its reason
- The named acceptance decision — who made it, on what evidence, and what they excluded
Monitoring is part of the evaluation, not a sequel
A pass is a statement about a frozen system on a fixed set at one moment. Everything after that drifts: inputs shift as upstream systems change, the corpus grows, the vendor updates the model, and performance degrades unevenly across slices while the aggregate stays flat. So the monitoring plan belongs in the same document as the pass criteria, with alert thresholds tied to a named on-call path and an incident procedure that ends in a new regression case.
- Stratified performance — tracked per slice over time, not as a single line.
- Input drift — distribution shift in what arrives, before output quality visibly moves.
- Abstention rate — a sudden drop often means the system started answering things it should decline.
- Grounding failures — for retrieval systems, unsupported claims and bad citations, sampled continuously.
- Version events — every model, prompt, index or guardrail change logged as an event the metrics can be read against.
- Incident-to-test loop — each real failure becomes a permanent case, dated, in the regression set.
What we build here, and what we will not do
We build the measurement layer: evaluation harnesses combining public benchmarks, safety suites and use-case-specific sets built from your data; red-team prompts aimed at hallucination, leakage and injection; rubric-based scoring with model judges validated against human raters; grounding metrics for retrieval systems; stratified performance and fairness monitoring with drift detection; and the governance artifacts — model cards, datasheets, impact assessments — versioned alongside the model and the data. That is the work described on our responsible AI capability page, and it is the same work whether the regulator is federal or not.
Here is where the line sits, stated plainly so you can rule us out cheaply. We do not sign off for you: the acceptance decision belongs to the accountable owner inside your organization, and we build the evidence that person weighs. We will not hold both sides of an evaluation — if we build the system, we are not also your independent assessor of it, because a finding we would have to remediate is a finding we have an incentive not to write down. We do not give legal or regulatory-affairs opinions about which regime governs your product.
We do not issue authorizations to operate — no contractor does; an authorizing official does. We are not a C3PAO and cannot assess you for CMMC. We hold no facility clearance today and do not perform classified work on classified networks; unclassified design and modeling happen on our side, and any cleared execution path would have to run through a partner holding the clearance. We are not a training vendor, and we do not write proposals for other firms. We will not tell you anything we build arrives pre-accredited, because nothing does — a system inherits controls and evidence, never an authorization. And one specific to this work: we will not report a pass on a slice that never carried enough items to establish one.
How a conversation starts
Send the workflow and the constraint — what the output feeds, who has to accept it, and which rule or reviewer is driving the requirement. The first exchange is scoping and costs nothing: which slices carry the consequence, roughly how many labeled items each needs, whether human review can be blinded given your data, and what the record needs to contain to be readable in two years. If the fit is poor, we would rather reach that early.
Frequently asked questions
Evidence that the criteria were written before the result was known, that the test data resembles production, that the scoring procedure is reproducible by someone else, and that the whole thing is recorded with dates and versions. Almost every follow-up question in a review reduces to one of those four.
It depends on the failure rate the workflow can tolerate. The rule of three from Hanley and Lippman-Hand (JAMA 1983) gives the intuition: with zero observed failures in n items, the 95% upper bound on the true rate is about 3/n, so 40 clean items support a bound near 7.5% and roughly 300 clean items support a bound near 1%.
It can, with disclosure and validation. The MT-Bench authors report position, verbosity and self-enhancement biases in model judges alongside high agreement with human preferences on their data (arXiv:2306.05685). Name the judge model and version, randomize answer order, and measure its agreement with human raters before its scores count.
Not everything, but you do have to re-run something, and the protocol should say which. Hosted endpoints behind a stable name can shift measurably over time, as Chen, Zaharia and Zou documented across snapshots a few months apart (arXiv:2307.09009).
No, and be careful with anyone who says yes. We do not issue authorizations to operate, we are not a C3PAO and cannot assess you for CMMC, and we do not render legal or regulatory opinions about which regime applies to your product. We also will not serve as the independent assessor of a system we built ourselves.