Acceptance is where reproducibility is won or lost
Most organizations accept an AI system the same way they accept a slide deck. Someone schedules a two-hour demo. The vendor screen-shares a notebook, runs a cell, and a chart appears. The accuracy number in the final report matches the accuracy number on the screen. Everyone signs. Eight months later a new engineer is asked to retrain the model on fresher data, opens the repository, and discovers there is no path from the delivered files to the number in the report. The number was real once. It is no longer recoverable.
That gap is not exotic. It is the default outcome of an acceptance process that tests presentation rather than reconstruction. Our engineers have taken handoffs from other teams often enough to know what the failure feels like from the receiving side: a container that builds only on the machine that built it, a training script that reads a CSV nobody can find, a metric computed once in a scratch notebook and typed into a Word file by hand. None of that requires bad faith. It is what happens when nobody was ever asked to prove the pipeline runs twice.
A reproducibility audit closes the gap by moving one question to the front of acceptance: can a person who was not on the delivery team regenerate every reported number from the delivered artifacts, on hardware the vendor has never touched, with no help from the vendor? If the answer is yes, the system is yours. If the answer is no, you own a demo and a maintenance liability.
Reproducibility Audit — Weight We Give Each Check
Editorial weighting from our own delivery practice — illustrative, not a measured statistic.
The five things to require at acceptance
Reproducibility is not a virtue you inspect for after the fact. It is a set of five artifacts you name in the statement of work, price into the deliverable, and refuse to sign without. Written this way, they are testable by a junior engineer with a laptop and a checklist.
- A pinned environment identified by immutable digest, not by a mutable tag.
- A seed and determinism policy that states what is fixed and what is not.
- A data manifest with a cryptographic hash and a fixed split assignment for every file.
- One command that regenerates every number in the final report.
- A results ledger that gives each reported number a name, a value, and a tolerance.
1. The pinned environment
An environment is pinned when a stranger can rebuild it byte-for-byte six months later. In practice that means a container image referenced by its sha256 digest rather than by a tag like latest or v2, because tags are pointers and pointers move. It means a dependency lockfile with hashes for every transitive package, not a loose requirements file with unbounded version ranges. It means the base operating system, the accelerator driver, the CUDA and cuDNN versions, and the hardware class the numbers were produced on, all written down.
It also means a software bill of materials. The SBOM expectation for federal software traces to Executive Order 14028 and the NTIA minimum-elements guidance, and NIST SP 800-218, the Secure Software Development Framework, is the practice reference agencies point suppliers toward. A CycloneDX or SPDX document that lists every component with a version and a supplier is now a normal deliverable, and it doubles as a reproducibility artifact: it tells the auditor exactly what should be present in the rebuilt image.
The test is cheap. Take the delivered digest, pull it onto a host that has never seen the project, and diff the installed package set against the SBOM. Anything present in the running system and missing from the manifest is an undeclared dependency, and undeclared dependencies are where reproducibility quietly dies.
2. Seeds, and the nondeterminism you cannot remove
Fixing a random seed is the part everyone knows about. Setting a global seed for Python, NumPy, and the training framework costs three lines and removes most of the run-to-run drift in data shuffling, weight initialization, dropout masks, and augmentation. Requiring it is not optional; a delivery that cannot state its seeds cannot explain its own numbers.
The harder part is honesty about what seeds do not fix. GPU reduction order is not deterministic by default, several cuDNN kernels select algorithms at runtime based on benchmarking, and TF32 arithmetic on modern accelerators trades mantissa bits for throughput. PyTorch exposes explicit controls for this, including a determinism mode and a required CUBLAS workspace setting, and turning them on usually costs measurable throughput. That trade should be a documented decision, not an accident.
Systems built on a hosted language model carry a third category. Sampling temperature must be pinned at zero for any reported evaluation, the exact model snapshot identifier must be recorded rather than a floating alias, and the raw request and response pairs should be cached to disk so the scoring step can be replayed without re-calling the provider. A hosted model that is silently updated behind a stable name will move the numbers under a delivery that looks unchanged. Cached responses are the only defense the receiving organization actually controls.
"Deterministic where it can be, declared where it cannot"
The acceptance clause that works reads roughly like this: the contractor shall pin all controllable sources of randomness, shall enumerate every remaining source of nondeterminism, and shall state, for each reported metric, the variation observed across a stated number of independent runs. That single sentence converts an unfalsifiable claim into a testable one.
3. The data manifest
The data manifest is the artifact most deliveries skip, and it is the one that decides whether a disputed number can ever be settled. It lists every input file with a SHA-256 hash, a byte count, a row count, the source it came from, the timestamp it was retrieved, the handling caveat that applies to it, and the split it belongs to. Nothing else in the delivery is allowed to decide split membership at runtime.
That last rule matters more than it sounds. If the code draws a random 80/20 split at execution time, then the training set and the test set change identity on every run, and a leakage bug becomes structurally invisible. Splits assigned in the manifest and looked up by hash are checkable by an outsider in minutes. Splits drawn at runtime are checkable by nobody.
Handling caveats belong in the same file. If any record is Controlled Unclassified Information, the marking and category should travel with the manifest entry, which puts the DFARS 252.204-7012 safeguarding and reporting obligations and the 32 CFR Part 2002 marking rules on a concrete footing rather than a general assurance. If any record is protected health information, personally identifiable information, or licensed third-party data with a redistribution limit, the manifest is where a receiving organization discovers that before the data lands in a shared bucket.
4. One command
The single most useful acceptance requirement in the whole list is one line long: make reproduce shall regenerate every number that appears in the final report, writing them to a machine-readable results file. Not a notebook to be executed cell by cell. Not a README with fourteen numbered steps and a warning about the third one. One entry point, checked into the repository, that a person can run without understanding the system first.
The corollary is stricter and matters just as much: every figure, every table, and every metric in the delivered documentation must be generated from that results file by code that ships in the repository. Hand-typed numbers are forbidden. A chart pasted from a spreadsheet is a number with no provenance, and a number with no provenance cannot be defended in a technical review, a debrief, an audit, or a protest.
When we take a handoff, this is the first thing we look for and the fastest signal we get. A repository with a working reproduce target is almost always sound underneath, because the discipline required to build one forces every other artifact into place. A repository without one is usually hiding at least three of the failures in the table below.
How the audit is actually run
The audit is a fixed procedure, and the point of the procedure is to remove the vendor from the loop. Every step below assumes no vendor engineer is available to answer questions, because that is the condition the system will be maintained under for the rest of its life.
Reproducibility Audit — Standard Sequence
Disabling the network at step four is the step people want to skip, and it is the one that catches the most. A run that quietly downloads a package, fetches a model checkpoint, or calls a hosted inference endpoint is not reproducible in a year and is not deployable at all in an air-gapped or classified enclave. Better to learn that during acceptance than during accreditation.
Step five is what turns a single number into evidence. One run gives a point. Four runs give a spread, and the spread is what tells you whether a claimed improvement is real. If the interval around the new method overlaps the interval around the baseline, the delivery has not demonstrated a difference, whatever the headline says. That rule alone has saved more procurement decisions than any model architecture ever will.
What failure looks like
Failures are not mysterious. They fall into a small number of shapes, and each shape has a standard reading and a standard remedy.
| What the auditor sees | What it means | Standard remedy |
|---|---|---|
| Build fails on a clean host | A hidden dependency on the vendor's machine: a private package index, a local cache, a licence file, an environment variable nobody declared. | Correction at the contractor's expense under the inspection clause; no acceptance until the build is clean. |
| Hash mismatch on inputs | The reported numbers were produced against a different dataset than the one delivered. Every downstream metric is unverified. | Deliver the actual inputs or re-run and re-report against the delivered ones. |
| Metric drifts outside tolerance | Uncontrolled randomness, an unpinned dependency, or a data-order effect the delivery never characterized. | Seed policy plus a declared multi-run spread; re-baseline the report to the mean and interval. |
| A number appears only in the report | It was computed by hand, in a notebook that did not ship, and has no traceable provenance. | Regenerate it from the pipeline or strike it from the documentation. |
| The run reaches the network | Not reproducible once the remote artifact moves; not deployable in a disconnected enclave. | Vendor the artifacts into the delivery and re-run offline. |
| Test rows appear in training | Leakage. The headline metric is not an estimate of field performance and cannot be repaired by rerunning. | Rebuild the split from the manifest and re-report; treat the prior number as withdrawn. |
Two of these deserve a note. Leakage is the only failure in the table that cannot be fixed by better packaging, because it invalidates the claim rather than the delivery mechanics. And a hash mismatch is the failure most often waved away as clerical. It is not clerical. If the inputs that produced a number cannot be identified, the number is an assertion.
Tolerances: how close is close enough
Bitwise-identical output is the wrong bar on accelerator hardware, and demanding it produces theater. Floating-point reduction order alone will move the last digits of a loss value across two otherwise identical runs. The right bar is a per-metric tolerance, stated in the results ledger before the audit begins.
Quality metrics. A headline accuracy, F1, or mean average precision should reproduce within a stated absolute band across a stated number of seeds, and the report should carry the mean with an interval rather than the best single run. Half a point of absolute tolerance across five seeds is a common and defensible setting for a mature pipeline.
Latency and throughput. These reproduce only against named hardware. A p50 and p95 latency figure on a specified instance type, within a fifteen percent band, is verifiable. A claim of "real-time" with no hardware named is not a claim at all.
Cost. Unit economics belong in the ledger too: dollars per thousand documents, per million tokens, or per inference hour, at a stated price sheet and a stated date. Cost claims drift with vendor pricing, so the date is part of the number.
Fairness and subgroup performance. Where the system touches benefits, hiring, health, credit, or enforcement, subgroup metrics need tolerances as well, and the subgroup definitions belong in the manifest so they cannot be redrawn after the fact.
Where federal contract language already helps
Federal buyers hold more authority here than they usually exercise. FAR Part 46 governs quality assurance, and FAR 46.202-4 lets a contracting officer impose higher-level quality requirements when the work warrants it. FAR 52.246-4, the inspection clause for fixed-price service contracts, gives the government the right to inspect and to require correction of nonconforming work at the contractor's expense. FAR 46.407 covers what happens when nonconforming supplies are offered. None of this requires a new authority. It requires acceptance criteria specific enough that "nonconforming" has a meaning.
On the defense side, put the reproduction package on a contract data requirements list rather than burying it in an appendix, so it becomes a numbered deliverable with a due date and a review cycle. Sort out rights early: DFARS 252.227-7013 covers noncommercial technical data, 252.227-7014 covers noncommercial computer software, and 252.227-7018 governs SBIR and STTR data, whose protection period runs twenty years from award under the SBIR Policy Directive. A reproduction package you cannot legally run after the contract closes is not much of a package.
Policy has moved the same direction. The NIST AI Risk Management Framework's MEASURE function asks for documented, repeatable evaluation rather than a one-time score. OMB's 2025 memoranda on federal AI use and AI acquisition push agencies toward pre-deployment testing, performance documentation, and contract terms that keep the government able to evaluate what it bought. For grant-funded work, 2 CFR 200.334 sets a three-year record retention floor, and a reproduction package is the cleanest way to satisfy it for computational results.
What this costs, and what skipping it costs
A reproducibility audit on a delivery in the two hundred thousand to two million dollar range takes three to ten business days of engineering time. Most of that is waiting on runs. The engineering judgment concentrates in step two and step six, and the deliverable is a delta report short enough for a program manager to read on one screen.
Compare that with the alternative. A system that cannot be rebuilt has to be replaced when the data schema changes, when the model provider deprecates a snapshot, or when the one engineer who understood it leaves. Replacement is a new procurement, a new lead time, and a repeat of the original cost. The audit is roughly one to three percent of the delivery it protects.
There is a second return that is easier to miss. A delivery that passes a reproducibility audit can be handed to a different firm without a rebuild. That is real optionality: it keeps the recompete honest, it keeps the incumbent's pricing disciplined, and it means a capability outlives the relationship that produced it.
Common objections
Our vendor says full reproducibility is not possible with GPUs.
Partly true, and it is not a reason to skip the requirement. Bitwise reproducibility across accelerators is genuinely hard. Reproducibility within a declared tolerance, from a pinned environment, on named hardware, is routine. The correct answer to the objection is to ask for the tolerance and the number of seeds behind it, then verify both.
We are buying a hosted product, not a custom build. Does this apply?
The environment and seed requirements shift to the vendor's side, but the evaluation requirements do not move at all. Ask for the evaluation set, the scoring script, the model snapshot identifier, and the date every quoted number was measured. If the vendor will not let you re-score their claims on data they have not seen, that is the finding.
The data is sensitive and cannot be handed to an auditor.
Common, and solvable. The manifest travels with hashes rather than records, and the audit runs inside the customer's enclave with the auditor working alongside a cleared or authorized custodian. When even that is closed, a synthetic dataset with the same schema and row counts verifies the mechanics, and the quality metrics are re-run internally against the real inputs by staff who already hold access.
Acceptance is next week. Is it too late?
No, but the bargaining position weakens after signature. The fastest useful move is to hold a defined portion of final payment against a reproduction milestone and give the vendor thirty days to deliver the package. Most teams can assemble it if the pipeline is sound, and the ones who cannot have told you something important.
Bottom line
Reproducibility is not a research nicety that got imported into procurement. It is the property that decides whether an organization owns a system or merely hosts one. Five artifacts, one command, one clean host, and a delta report are enough to tell the difference, and they are cheap enough to require on every AI delivery regardless of size. Write them into the acceptance criteria, test them before signature, and the question of whether the numbers are real stops being a matter of trust.
Frequently asked questions
A fixed procedure in which an independent engineer rebuilds the delivered system on a clean host, restores the data from a hashed manifest, runs a single documented command with the network disabled, and compares every regenerated number against the values in the final report and their stated tolerances.
Five things: an environment pinned by image digest and lockfile with an SBOM, a seed and determinism policy that also declares what remains random, a data manifest with per-file hashes and fixed split assignments, one command that regenerates every reported number, and a results ledger giving each metric a tolerance.
Bitwise equality is the wrong bar on accelerator hardware. Set a per-metric tolerance in advance: an absolute band for quality metrics across a stated number of seeds, a percentage band for latency on named hardware, and a dated price basis for cost figures.
Anyone other than the delivery team. An internal group that had no role in the build works, and so does an outside firm. What matters is that the person running it has no ability to ask the original engineers a clarifying question, because that is the condition under which the system will be maintained.
Most failures are packaging failures and are correctable in days once identified. Federal fixed-price service contracts carry inspection clauses at FAR 52.246-4 that let the government require correction at the contractor's expense. The exception is data leakage, which invalidates the claim itself and requires re-reporting rather than repackaging.