The question that arrives eighteen months late
Somebody outside your company asks how a number was produced. A large customer's procurement team wanting the derivation behind a risk score. An examiner. An acquirer's diligence team with two weeks and a checklist. The number is eighteen months old, the analyst who produced it has moved on, the pipeline has been rewritten twice, and the source table it read has been backfilled three times. You are now going to spend somewhere between four and twelve engineer-weeks reconstructing a result that took four days to produce originally. That reconstruction is the whole cost of not having built reproducibility, and it arrives at once, on someone else's schedule.
The academic framing has done real damage. Reproducibility got attached to the replication crisis in psychology and to a genre of conference workshop, which taught a generation of engineering leaders to file it under research hygiene. It is the property that determines whether your company can defend its own outputs when defending them becomes commercially necessary.
Three different things get called reproducibility
The word covers three distinct capabilities with different costs, and conflating them is why budget conversations go sideways.
Rerun. You can take the recorded inputs and the recorded code and get the same output again on demand. The cheap one, and the one that answers most external questions.
Rebuild. You can start from the raw source data as it existed at the time and reconstruct the intermediate artifacts, including the trained model. Much more expensive, because the raw data has to still exist in its historical state.
Explain. You can say why the output is what it is. Which inputs drove it, which rules fired, what the model weighted. A different engineering problem, and it does not follow from the other two.
Most commercial obligations need the first, sometimes the second, increasingly the third. A vendor questionnaire asking how a specific customer's score was calculated on a specific date is a rerun question. A model-risk examination under the Federal Reserve's SR 11-7 guidance is a rebuild plus explain question. Deciding which you owe, per system, before you engineer for it is most of the savings available.
What Breaks Reproducibility — Ranked by How Often It Is the Root Cause in Our Reconstruction Work
Relative frequency across our reconstruction engagements, not a survey. The ordering is the useful part: the top three are data and configuration problems, not compute problems.
Teams tend to reach for the bottom row first. Engineers hear "reproducibility" and start reading about deterministic GPU kernels and CUBLAS_WORKSPACE_CONFIG. Real issue, almost never the one that stopped you. What stopped you is that the table you read from does not remember what it contained on the day you read it.
Data mutation is the failure, ninety percent of the time
A warehouse table is a live object. Rows get corrected. Late events land. A vendor resends a file. Someone backfills a transform bug from three months back and silently rewrites two years of history. None of this is misconduct. All of it means a query written today against a date range returns different rows than the identical query returned last March.
So the first structural fix is that any analysis whose result you may need to defend must read from a snapshot with an identity, not from a live table with a date filter. Concretely: a table format that keeps immutable snapshots and lets you address one. Apache Iceberg gives you snapshot IDs and a FOR SYSTEM_VERSION AS OF read. Delta Lake gives you versioned commits and time travel. Both let a run record a snapshot ID rather than a timestamp, which is the difference between "the data as of March" and "exactly these rows."
Two details defeat this if you skip them. Retention: snapshot expiration and vacuum operations delete the history you were relying on, and the defaults are aggressive, often seven days. If your obligation is five years, that is a setting somebody has to own and a storage bill somebody has to approve. Second, upstream systems that overwrite in place. If your warehouse ingests a vendor's daily full extract into the same location every day, you have no history whatever your table format is. The snapshot has to be taken where the data lands.
Where rebuild matters, the honest question is whether the raw source still exists in its historical state at all. Often it does not, and no downstream engineering recovers it. Decide that deliberately, at design time, per source, rather than discovering it during an examination.
The run record, and what has to be in it
The second structural fix is that every consequential run writes an immutable record of everything that determined its output. Not a log line. A structured record with a stable identifier that appears on the output itself.
The set is small, and it is the same set every time.
| What the record pins | How to pin it | How it usually goes wrong |
|---|---|---|
| Input data | Snapshot ID or content hash per source, not a date range | A SQL query with WHERE date < '2026-03-01' and nothing else |
| Code | Commit SHA of a clean tree, with dirty-tree runs refused outright | A branch name, or a SHA recorded from a working copy with uncommitted edits |
| Environment | Container image by digest plus a fully resolved lock file | An image tag like :latest and open version ranges in requirements |
| Configuration | The fully resolved config as it was used, serialized into the record | Config merged from files, flags, and environment variables, never captured |
| Randomness | Every seed, recorded whether set explicitly or generated | A library default seeded from wall-clock time |
| External calls | Endpoint, version, and either cached responses or a recorded digest | A live third-party API whose model was replaced under the same URL |
| Outputs | Content hash of every artifact, so a later rerun can be compared | Outputs written to a mutable path each run overwrites |
The last row is the one teams skip and the one that pays. Without a hash of the original output, a rerun eighteen months later that differs slightly leaves you unable to tell a real problem from a floating-point ordering artifact. With a hash and the original artifact, the diff is a five-minute question.
The external-call row deserves its own warning. A pipeline that calls a hosted language model has a dependency that changes without notice under a stable URL, and provider model versions get deprecated on published schedules. If a scored output depends on that call, reproducing it later requires the stored prompt, model version string, sampling parameters, and the response itself. Log the response. It is small, and it is the only copy that will exist.
What determinism you can actually get
Bit-exact determinism is achievable for most data transformation and classical ML work, and worth having, because it makes the diff test trivial. For deep learning on GPUs it costs throughput. PyTorch's use_deterministic_algorithms plus fixed seeds plus a pinned cuDNN configuration gets you there on fixed hardware and drivers, usually at a five to thirty percent training slowdown depending on the operations. Change the GPU model or the driver and bit-exactness can disappear with everything else identical.
So set the bar per system. Where outputs feed a regulated decision, bit-exact retraining on pinned hardware is worth its cost. For most production models the useful standard is statistical reproducibility with a stated tolerance: the recorded pipeline reruns to metrics within a documented band, every input pinned. Write the tolerance down in advance. A tolerance chosen after seeing the disagreement is not a tolerance.
Inference is a different matter, and stricter. If the same input produces a different score for the same customer on two different days without a version change, you have a defect regardless of your training story. Inference determinism is usually achievable and usually broken by three things: an unpinned preprocessing step, a threshold changed outside the version record, and a feature read from a live store rather than a point-in-time one.
Where the frameworks put this, and what they call it
No governance framework uses the word reproducibility as a heading, which is part of why it goes unbudgeted. Every one of them requires it under another name.
The NIST AI Risk Management Framework puts it in Measure. Measure 2.8 asks for risks to be documented and traceable, and the whole function presumes a measurement can be repeated. Map 2.3 addresses test, evaluation, verification, and validation, which is unperformable against an artifact you cannot rebuild.
ISO/IEC 42001, the AI management system standard, requires documented information for AI system development, data provenance records, and records of resources used. An auditor reading that clause and finding no run records writes a nonconformity, and the finding says traceability, not reproducibility.
SR 11-7, the Federal Reserve and OCC supervisory guidance on model risk management, is the oldest and bluntest of these. It requires developmental evidence sufficient for an independent party to evaluate the model, and independent validation cannot proceed against a model nobody can rebuild. Banks learned this expensively fifteen years ago. Health-tech and ratings firms are learning it now.
For federal-adjacent work the mechanism is more direct. NIST SP 800-53 configuration-management controls, CM-2 baseline configuration and CM-3 change control, plus audit controls in the AU family, are what an assessor reads. A pipeline that cannot state which code and which data produced an artifact fails those controls on the evidence, not on the narrative. The same shape appears in NIST SP 800-171 for CUI environments.
What it costs, both ways
Retrofitting reproducibility onto a mature pipeline runs three to eight engineer-weeks, and the spread is almost entirely about whether the upstream data has usable history. Where snapshots exist and only the run record is missing, closer to three. Where source systems overwrite in place and the fix reaches into ingestion, closer to eight, plus coordination with whoever owns that system.
The cost of not having it is lumpier and larger. A single reconstruction under external deadline runs four to twelve engineer-weeks of senior time, done under pressure, with the outcome uncertain until it is finished. Two consequences beyond the labor are worse. In diligence, an inability to reproduce a headline metric reads as a claim without evidence, and it moves price. And when a customer challenges an output and you cannot produce the derivation, the practical resolution is to concede, because you cannot demonstrate otherwise.
The asymmetry is the argument. Ten to fifteen percent of build cost, spread across the project, against a lump you cannot schedule and cannot bound.
Senior Engineer-Weeks to Answer One External Question, by What Was Recorded at the Time
Ranges from our reconstruction engagements, rounded. Illustrative of the shape, not a measured statistic.
The five-question test for where you stand
Pick a consequential output your company produced about a year ago. A score you sent a customer, a model you deployed, a figure someone relied on. Ask five questions.
Can you name the exact rows it read? Not the table and the date range. The rows, addressed by a snapshot identity that still resolves today.
Can you get the exact code? A commit SHA, on a tree with no uncommitted changes, still reachable.
Can you rebuild the environment? An image digest that still pulls, or a lock file whose packages still exist at those versions.
Can you rerun it without a person? If any step involved someone opening a spreadsheet or clicking a console button, that step is not recorded and probably not remembered.
Can you tell whether the rerun matched? Which requires that you hashed the original output.
Most teams get two or three. The two they miss are almost always the first and the last, which are also the cheapest to fix going forward. Snapshot identity on reads and content hashes on writes, enforced at the pipeline framework rather than left to individual authors, closes most of the gap for new work without touching anything else.
How to sequence the work
Do not start with a platform. Start by naming which outputs you would have to defend. The answer is usually a short list, and the engineering is only worth doing for those. A ratings firm might have four. A health-tech company might have one clinical-facing model and a billing pipeline. Everything else can stay as it is.
Then fix reads before writes. Point every consequential pipeline at snapshot-addressed sources and set retention to match your longest obligation. This one change moves more teams from "cannot reproduce" to "can reproduce" than any other, and it does not require rewriting transformation logic.
Then make the run record automatic. It has to be produced by the framework every run, not by discipline. A record that depends on an engineer remembering to call a logging function will be complete for six weeks. Make the pipeline refuse to run on a dirty tree, and refuse to publish an output that has no record attached. Refusal is the enforcement; documentation is not.
Then attach the record ID to the output where a human will see it. On the report, in the API response, on the dashboard. A small change that does more for external questions than everything upstream, because it turns "how was this produced" from an investigation into a lookup.
Only then consider tooling. MLflow, Weights & Biases, Iceberg or Delta, OpenLineage, dbt with snapshots. All capable, and the choice matters far less than whether the four steps above are enforced. We have worked on teams with an expensive platform that could reproduce nothing, because the platform recorded which artifacts exist while nothing recorded which data produced them.
Bottom line
Reproducibility is a commercial property, and the bill for its absence gets presented by someone outside your company on a schedule you do not control. The engineering is well understood: snapshot-addressed reads, an automatic and immutable run record, content hashes on outputs, and a record ID that travels with the result. Ten to fifteen percent of build cost when done from the start. Three to eight engineer-weeks per pipeline to retrofit. Four to twelve engineer-weeks of senior time, under deadline, each time it is missing and needed. The frameworks you already answer to require it under other names, so the budget argument is available even where the word is not.
Frequently asked questions
No, and the gap is where most failures live. Code versioning pins one of six things that determine an output. Data, environment, configuration, seeds, and external calls are the other five, and mutated source data is the single most common root cause we find.
Rarely for training, usually for inference. For most production models the workable standard is statistical reproducibility within a tolerance stated in advance, with all inputs pinned. For inference, the same input producing a different score without a version change is a defect.
Three to eight engineer-weeks per major pipeline. The spread depends almost entirely on whether upstream data has usable history. If source systems overwrite in place, the fix reaches into ingestion and needs the upstream owner's cooperation.
NIST AI RMF under Measure and Map, ISO/IEC 42001 under documented information and data provenance, SR 11-7 under developmental evidence and independent validation, and NIST SP 800-53 CM and AU control families for federal-adjacent systems. None of them use the word reproducibility.
Snapshot-addressed reads on the handful of pipelines whose outputs you would have to defend, plus content hashes on their outputs. Those two changes close most of the gap for new work and do not require rewriting transformation logic.