Reproducibility as a procurement requirement: what to write into the SOW

The number in the report, and the number you can regenerate

Every AI, ML, or analytics deliverable ends in a table of numbers. Accuracy on a held-out set. Median latency. False-positive rate at the operating threshold. Eighteen months later a different team picks the work up: a follow-on contractor, a government lab, a program office at another command. They have the report, a repository, and a model file. They run it, and the numbers do not come back. Not approximately. The environment moved, the data snapshot is gone, and nobody recorded which of the eleven training runs produced the table that got briefed.

This is a procurement failure before it is an engineering failure. The government bought a report and a binary. It did not buy the ability to regenerate the report, because nobody put that in the statement of work. A standard software SOW asks for source code, documentation, a test plan, and a demonstration. A contractor can satisfy all four and hand over something no third party can rerun.

The problem is not confined to defense work. A 2016 Nature survey of 1,576 researchers (Baker, Nature 533:452) found more than 70 percent had tried and failed to reproduce another scientist's experiment, and more than half had failed to reproduce their own. Machine learning adds pinned library versions, GPU kernel behavior, and data snapshots that change under you. Academia answered with checklists and artifact tracks. The government can do better, because it has a payment gate.

What "reproducible" has to mean when it is a contract term

Vendors will agree to "reproducible" all day, because in casual use it means "we could probably rebuild it." A contract term has to be tighter:

Given only the delivered package, a competent engineer with no access to the delivering team regenerates every number that appears in the report, within a stated numeric tolerance, on stated hardware, in a stated amount of wall-clock time.

Four phrases carry the weight. No access to the delivering team: if reproduction takes a phone call to the person who built it, you bought a person. Every number in the report closes the gap between a working demo and the specific figures that got briefed. Within a stated tolerance keeps the clause honest, since bit-exact reproduction is often unattainable on GPU hardware. On stated hardware, in a stated time heads off a compliant answer of the form "reproducible on 64 accelerators over three weeks," which satisfies a lawyer and helps no one.

The environment, pinned by digest and delivered offline

Start with the container image, and require it by digest rather than by tag. A tag like python:3.11-slim points at a different set of bytes every few weeks. A digest of the form sha256:9f2c... points at one image forever. The digest costs the vendor nothing and removes the most common cause of a build that worked in June and fails in February.

Then require the image itself, not a reference to it. A digest that only resolves inside a vendor's private registry is a hostage. Ask for an offline archive (a docker save tarball or an OCI layout) alongside the digest, with base layers obtainable from a source the government controls, such as an agency registry or the DoD Iron Bank hardened-container repository.

Under the image, a lock file covering transitive dependencies, not just direct ones. A requirements.txt with six unpinned packages is not a lock file. Include OS packages, CUDA and cuDNN versions, and a minimum driver version. NIST SP 800-218, the Secure Software Development Framework, already asks contractors to collect and share provenance data for every component of each release under practice PS.3.2. A software bill of materials meeting the NTIA minimum elements gets most of the way there. Reproducibility and supply-chain security want the same artifact, which is worth pointing out during negotiation.

Seeds, and the honest limits of determinism

Require seeds for every stochastic component: the language runtime, the array library, the framework, the data-loader workers, and any augmentation pipeline. Recording a single global seed is not enough when four libraries each carry their own generator state.

Then be honest about what seeds buy you on accelerator hardware. Floating-point addition is not associative, and parallel GPU reductions do not guarantee a fixed summation order, so two runs of identical code on identical data can differ in the last few digits and occasionally in the last percentage point once those differences compound through training. PyTorch exposes torch.use_deterministic_algorithms(True), wants CUBLAS_WORKSPACE_CONFIG=:4096:8 for certain CUBLAS paths, and needs cuDNN autotuning disabled. Some operations have no deterministic implementation and raise an error instead of varying silently. That is a feature: it tells you where the nondeterminism lives.

Bit-exact reproduction is a nice property and a bad contract term. Require a tolerance band instead, and require the contractor to state what it is at the moment of delivery. Half a percentage point absolute for accuracy-class metrics and ten percent for timing is defensible for most work, and the right numbers depend on the domain. A contractor who cannot state a tolerance for their own system has told you something about how well they understand it.

The data manifest is the part people skip

Code without data is a recipe with no ingredients. The manifest is the ingredient list, and it should be mechanical rather than narrative. For every input artifact: relative path, SHA-256 digest, byte size, record count, source, and the cutoff date for anything time-sliced. Then a single digest over the manifest itself, so one string identifies the entire input state.

Two details matter more than they look. Train, validation, and test splits belong in the manifest as explicit lists of record identifiers, never regenerated by a runtime shuffle, because a reshuffle under a different library version moves records across the split boundary and quietly changes every reported number. And when the government furnished the data, the manifest is what lets you verify later that the contractor trained on what you provided and only on that. GAO's AI Accountability Framework (GAO-21-519SP, June 2021) puts data provenance and lineage documentation at the center of its data principle for this reason.

The manifest also solves a problem that stops many teams cold: the data cannot travel. If the corpus is CUI, contains PII, or sits under a third-party license, the hashes travel and the bytes stay put. Anyone holding the same data can verify byte-level agreement, and reproduction happens inside the enclave where the data already lives.

What to require	How to phrase it	What it prevents
Pinned environment	Container image identified by immutable digest, plus an offline archive of that image	A build that stops working when a base image tag moves
Dependency lock	Lock file covering direct and transitive dependencies, OS packages, and accelerator libraries	Silent version drift that shifts results with no code change
Seeds and tolerance	All random seeds, plus a stated numeric tolerance for each class of reported metric	Arguments at acceptance about what counts as a match
Data manifest	Per-file SHA-256, size, record count, source, cutoff date; splits recorded by record ID	Training on a data snapshot nobody can reconstruct
One command	A single documented command that regenerates every quantitative result in the report	A twelve-step README that only its author can follow
Standing proof	The command runs in continuous integration at least monthly during the period of performance	A package that rots quietly between kickoff and final delivery

One command, because documentation rots silently

The requirement that carries the most weight per word is the simplest: one command. make reproduce, or ./reproduce.sh. It runs end to end, writes outputs to a results directory, prints a table with the same rows and columns as the one in the report, compares each value against the stated tolerance, and exits nonzero when anything falls outside.

A single command is the only form of documentation that cannot fail quietly. A README drifts out of date for a year and nobody notices until the day someone needs it. A command that no longer runs fails the moment it is executed, which is why the companion requirement is that it run on a schedule in continuous integration during the period of performance. If it breaks in month three, the contractor fixes it in month three, while the person who wrote it is still on the contract.

Ask for a fast path as well. Full reproduction of a training pipeline can take days on hardware the acceptance team does not have standing by, so a reproduce-quick target that runs on a subsample in fifteen minutes lets a contracting officer's representative smoke-test the package the day it arrives.

Why this protects the buyer at transition

Transition is where reproducibility pays. Under 15 U.S.C. 638(r)(4), a federal agency may award a Phase III contract to an SBIR awardee on a sole-source basis, deriving from that firm's earlier work. That authority is valuable when it reflects a choice. It becomes a trap when it reflects the fact that nobody else can rerun the pipeline, because the government is then negotiating into a market of one for as long as the capability matters.

The same logic applies to recompetes. If only the incumbent can regenerate the baseline, every challenger bids against a number they cannot inspect, and the evaluation team has no way to tell whether a claimed improvement is real. A performance figure the government cannot reproduce is a figure the government cannot defend, in a source-selection decision document or anywhere else.

There is an operations argument too. Models depreciate. Input distributions shift, a dependency reaches end of life, a data source changes schema, and the answer is usually to retrain rather than patch. Retraining requires the pipeline, the manifest, and the environment. It does not require the weights, which is the one artifact most contracts remember to ask for.

Lock-in is a build problem before it is a rights problem

Most agencies handle vendor lock-in through data rights, and they should. Under DFARS 252.227-7013 and 252.227-7014, noncommercial technical data and software are delivered with unlimited, government purpose, or restricted rights depending on who paid for development, with restrictions asserted at proposal time under DFARS 252.227-7017. Civilian agencies work from FAR 52.227-14, Rights in Data. SBIR-developed data carries its own protection period under DFARS 252.227-7018, extended to twenty years by the SBA SBIR Policy Directive. OMB memorandum M-25-22 on AI acquisition, issued in April 2025, directs agencies to address IP and data terms with lock-in in view.

All of that answers who may use the work. None of it answers whether anyone can. Government purpose rights in a repository that will not build are a legal asset with no operational value. The reproduction package is the half of the data-rights clause that makes the other half usable, and it costs one paragraph in the SOW.

Where to put it, and where not to

Put the requirement in three places and check that all three agree. It belongs in the SOW as a task, so the work is funded. It belongs in the deliverables list with its own CDRL or line item, so it has a due date and a review cycle rather than living inside a Software Version Description nobody opens. And it belongs in the acceptance criteria, because a requirement that does not gate payment is a preference.

Where not to put it: buried under "documentation." Documentation deliverables get reviewed for completeness, not for execution. A reviewer confirms the document exists and moves on. Reproduction is a test, and it belongs with the other things the government inspects before acceptance under FAR Part 46.

• Named as its own deliverable with a due date, not folded into documentation
• Container image identified by digest and delivered as an offline archive
• Lock file covering transitive dependencies and OS packages
• Data manifest with SHA-256, size, record count, source, and cutoff date per input
• Splits recorded by record identifier, never regenerated by a runtime shuffle
• Seeds recorded, plus a written statement of known nondeterminism
• One documented command that regenerates every number in the report
• Numeric tolerance stated by the contractor and written into acceptance
• The command runs in CI at least monthly during the period of performance

What it costs

Designed in at kickoff, this is a small fraction of engineering effort. The manifest generator is a short script. The container pin is a one-line change. The single command is mostly a consolidation of steps the team already runs by hand, and it tends to pay for itself inside the first month by removing the onboarding cost for every new engineer. That is an estimate from engineering practice rather than a measured figure, and the direction of the estimate is what matters.

Retrofitted, the same work is archaeology. The engineer who knew which run produced the briefed table has rolled off. The data snapshot lived on a laptop that got reimaged. Reconstructing all of it takes weeks and sometimes ends in a defensible shrug. Buying it up front is the cheapest risk reduction available in an AI acquisition, and one of the few a contracting officer can enforce without technical staff on hand, because the test is binary: the command runs and the numbers match, or it does not.

Common questions on scoping the requirement

Frequently asked questions