The request never arrives at a convenient time. A model has been serving predictions for eight months, someone disputes an output, and the questions come in a row: which model version produced that, what was it trained on, and can you rebuild it. If the honest answer involves a chat search, a notebook on a laptop that has since been reimaged, and an object-store prefix named final_v2_new, you do not have a build. You have an artifact of unknown provenance that happens to work.
Application engineering solved this a decade ago and stopped talking about it. A commit produces a build, the build produces an image, the image carries a digest, and any engineer can reconstruct any release from a tag. Machine learning skipped that step, mostly because training has two inputs instead of one. Code is versioned by habit. Data usually is not, and the environment underneath a training run is a moving target that nobody watches.
Closing that gap is about two weeks of engineering. It pays for itself the first time a model regresses, and it ends with a job that passes or fails rather than a policy nobody has executed.
You are probably here because
- Someone asked what a model in production was trained on, and finding out took three days and a search through old chat threads.
- You reran last quarter’s training script and got a model that scores differently, and nobody can say which input moved.
- The training job reads a live table, so the same query returns a different corpus every week and nothing records what it held on the day of the run.
- The artifact everyone depends on came out of a notebook, on a machine that has since been reimaged.
What actually decides the output of a training run, further down, ranks these in the order we fix them; all four usually come from the same root cause, which is that nothing in the pipeline recorded what the run actually used.
Three different things people mean by reproducible
Most arguments about reproducibility are two people using the same word for different guarantees. Separate them before you promise anything, because the cost gap between the weakest and the strongest is roughly an order of magnitude.
| Level | What it guarantees | What it costs | Require it when |
|---|---|---|---|
| Re-runnable | The job starts and completes on a clean machine from a recorded reference. Output is a model, not necessarily the same model. | Low. Mostly packaging discipline. | Always. This is the floor, and plenty of teams are below it. |
| Equivalent | The rebuilt model scores within a stated tolerance of the original on a frozen evaluation set. | Moderate. Dataset snapshots, locked dependencies, a manifest, a scheduled check. | Any model whose decisions you have to defend, explain, or roll back. |
| Bitwise | The rebuild produces byte-identical weights. | High on GPU. Determinism flags, fixed device count, fixed thread counts, and a throughput penalty. | Rarely. Forensic disputes, and shipping the same artifact from two build farms. |
Almost every team that says "reproducible" needs equivalent and is quietly hoping for bitwise. Equivalence is the right target because it survives hardware refreshes. A model rebuilt on a newer GPU generation will not be bitwise identical to one trained two years earlier, and it should not have to be. What it has to do is land inside a tolerance you set deliberately, on an evaluation set you control.
Bitwise reproducibility is worth having on CPU-bound feature pipelines, where it is nearly free. On multi-GPU training it means freezing device count, reduction order, thread counts and kernel selection together, and it breaks the first time you scale the cluster. Ask for it only when someone can name the dispute it would settle.
What actually decides the output of a training run
Ask an engineer what makes a run reproducible and the first answer is the seed. The seed is real, and it is a long way down the list. Below is how we weight the inputs when we retrofit a pipeline, in the order that determines what gets fixed in week one.
Sources of Run-to-Run Variance — Our Fix-First Weighting
Weights sum to 100. They rank engineering priority, not the size of the numerical effect.
The ordering surprises people, because the bottom two rows are the ones every team already handles. The top three are where builds rot, and they rot silently: nothing throws an exception when a base image tag moves or a source table gets backfilled.
The code is the easy half, and teams still lose it
Record the commit SHA inside the training job, not in the launch script and not in a wiki. The job should read its own version and write it into the run record. Then add the check that gets skipped: assert the working tree was clean at launch. A run started from a tree with uncommitted edits is not reproducible from that SHA, and the difference between "trained at a3f91c" and "trained at a3f91c plus two lines somebody was testing" is invisible six months later. One git status --porcelain call at startup, recorded as a boolean, closes it. In continuous integration make a dirty tree a hard failure for release builds and a warning for experiments.
Notebooks are the other leak. They hold hidden state, execute out of order, and do not diff. Explore in them, then move anything that produces a released artifact into a module with an entry point. The rule we apply: if the output goes to a registry, the code that produced it lives in a file with a test next to it.
Pin the environment by digest, never by tag
A container tag is a mutable pointer. python:3.11-slim resolves to different bytes month to month, and a CUDA base image can change its runtime and driver expectations under the same tag. Pin with the digest instead: FROM python:3.11-slim@sha256:.... The tag tells you what someone intended. The digest tells you what you got.
Dependencies need the same treatment one level up. A requirements file with >= constraints is a set of instructions to resolve fresh at build time, which means the build is a function of the day it ran. Generate a fully resolved lockfile with hashes for every transitive package and install with hash verification enforced: pip-compile --generate-hashes then pip install --require-hashes, or the lockfile from uv, Poetry or conda-lock. Hashes matter more than versions, because the same version number can ship a different wheel for a different platform tag, and a package can be yanked or rebuilt after publication.
System packages are the layer everyone forgets. apt-get install pulls whatever the distribution mirror currently holds. Either pin package versions explicitly, install from a snapshot mirror, or accept that your reproducibility story stops at the operating system boundary and write that down. For the deep-toolchain layer, CUDA and cuDNN versions belong in the image digest and in the manifest, because a minor version change can alter which kernels a framework selects.
One borrowed idea is worth adopting: the Reproducible Builds project's SOURCE_DATE_EPOCH convention for stamping build times deterministically. Timestamps are the most common reason two builds of identical inputs produce different bytes.
Pin the data, which is where it actually breaks
Here is the failure that costs the most and gets the least attention. A training job reads from a warehouse table. The table is live. Rows arrive late, corrections land, records get deleted for a privacy request, labels get revised after a review. The same SQL executed three months apart returns a different corpus, and nothing anywhere records that. The model is not reproducible and the pipeline reports success.
Three fixes work, in descending order of how much infrastructure they assume.
| Mechanism | What you record | Fits when |
|---|---|---|
| Table-format snapshot | The snapshot id or version number the table format already assigns to every commit | Data is in Iceberg, Delta Lake or Hudi. Usually available and unused. |
| Content-addressed storage | The corpus hash a tool such as DVC or lakeFS assigns | File corpora: images, audio, documents, with no warehouse in the picture. |
| A plain file manifest | A list of every input file with size and SHA-256, and one hash over that list | Anywhere. A few dozen lines, no new infrastructure, and often what ships in week one. |
Splits deserve their own paragraph, because the standard approach is broken in a way that looks fine. Shuffling with a fixed seed and slicing gives a deterministic split for a fixed dataset, and reassigns nearly every row the moment new rows arrive. Hash a stable identifier instead: put a record in the holdout when sha256(record_id + salt) % 100 < 20. New data lands in the right bucket, the split survives corpus growth, and no record quietly moves from training to test. Record the salt, since the rule is worthless without it.
The related trap is temporal leakage. Any feature computed from a mutable table has to be computed as of the label timestamp, not as of now. A feature that reads "current account status" at training time is reading the future, the model looks strong offline, and it degrades the day it goes live. Point-in-time correctness is a reproducibility problem and an accuracy problem at once.
Seeds pin less than people assume
Setting random.seed() covers Python's standard library and nothing else. A typical training job draws randomness from at least six places: the Python standard library, NumPy, the framework's CPU generator, the framework's CUDA generator, each dataloader worker process, and the augmentation pipeline. Seed all of them explicitly and record every value.
Dataloader workers are the sharpest edge. Multiple worker processes each carry their own generator state, so the order in which batches are assembled depends on process scheduling unless you pass an explicit generator and a worker_init_fn that derives each worker's seed deterministically from the base seed and the worker id. Without that, the same seed produces a different batch order on a machine with a different core count.
Set PYTHONHASHSEED in the environment rather than in code, because the interpreter reads it at startup and setting it after import does nothing. Its effect is real: string hashing randomization changes set iteration order, and set iteration order leaks into data ordering more often than anyone expects.
The controls that change results on a GPU
In PyTorch, torch.use_deterministic_algorithms(True) switches to deterministic kernels and raises a RuntimeError for any operation that has no deterministic implementation. That loud failure is the point: it surfaces the operation instead of hiding the variance. On CUDA it requires CUBLAS_WORKSPACE_CONFIG=:4096:8 (or :16:8) set before the process starts. Add torch.backends.cudnn.deterministic = True and torch.backends.cudnn.benchmark = False, since autotuning selects convolution algorithms by timing them and timing is not stable. Decide explicitly about TF32 via torch.backends.cuda.matmul.allow_tf32, because the default has moved between framework releases and a silent change in matmul precision looks exactly like a data bug. On CPU, pin OMP_NUM_THREADS and MKL_NUM_THREADS: thread count changes reduction order, and floating-point addition is not associative.
That last point is the physical reason GPU training is not deterministic by default. Parallel reductions and scatter operations accumulate with atomics, so the order in which thousands of threads add their contributions varies between runs. Each difference sits in the last bits of a float, and over an epoch they compound through the optimizer into visibly different weights. Nothing is broken: addition is not associative in floating point, and parallel hardware promises no order.
Distributed training adds one more: change the number of devices and you change per-device batch sizes and the gradient reduction tree, so results move even with every seed fixed. World size belongs in the manifest, and a rebuild has to use the same one to be compared on tight tolerance.
What determinism costs
Determinism is not free and the price varies enormously by model family. Below are the planning figures we budget before measuring, which is exactly how they should be used: as a starting allowance, replaced by your own numbers after one benchmark run.
Throughput Budget for Determinism Controls
Planning allowances only. The spread across model families is wide; measure on your workload and replace these.
The resolution is to run two modes rather than to pick one. Experiments run fast, with autotuning on and determinism off, because during a search you care about relative comparisons and iteration speed. Release builds run with the controls set, once, on the configuration that won. A team that treats determinism as an all-or-nothing switch usually turns it off and never turns it back on.
Send it over and we will tell you what we would change.
Email your Dockerfile or requirements file, plus the part of the training job where it loads its data and sets its seeds, to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comThe build manifest
Everything above becomes usable only when a single machine-written record ties it together. The manifest is a JSON document emitted by the training job, stored beside the weights and pushed into the model registry. Written by hand at release time it is fiction. Emitted by the job it is evidence.
- Repository, commit SHA, and an assertion that the working tree was clean
- Container image referenced by digest, plus the hash of the lockfile that built it
- Dataset snapshot identifier, or the content hash of the input file manifest
- Split rule and its salt, not the resulting list of row identifiers
- Label set version and the revision of the labeling guidance behind it
- The fully resolved configuration after defaults and overrides, as one file
- Every seed, including dataloader worker and augmentation seeds
- GPU model, driver, CUDA and cuDNN versions, device count, and the determinism environment variables
- Start time, duration, and the identity of the runner that executed the job
- Metrics on the frozen evaluation set, with that set's own version and hash
- The hash of the produced weights, taken over tensor contents rather than the container file
That last item has a subtlety worth knowing. Serialization formats can embed ordering and metadata that differ between writes without a single parameter changing, so hashing the file can report a difference that does not exist in the model. Hash the tensors in a defined order, or use a format with a fixed layout such as safetensors, and you get a comparison that means something.
The rebuild job, which is the only real proof
A pipeline can satisfy every item above and still not be reproducible, because none of it has been executed. What converts intent into a property of the system is a scheduled job that rebuilds a released model from its manifest and compares the result.
The mechanics are simple. Monthly, pick one released model, preferably rotating through the portfolio. Read its manifest. Provision a clean runner, pull the image by digest, install from the locked hashes, read the dataset snapshot, set the seeds and flags recorded, and train. Score the result on the frozen evaluation set referenced in the manifest and compare against the metrics recorded there. Pass inside tolerance, fail outside it, and page whoever owns the pipeline on failure.
Budget for the first run to fail. In every retrofit we have done, it does, and the failure is informative: a dependency that was never in the lockfile, a data path that quietly moved, an environment variable that only existed on one engineer's workstation. That first failure is the entire return on the exercise arriving at once, in a controlled setting rather than during an incident.
Large models need a scoped version rather than an exemption. Rebuilding a four-day, eight-GPU run every month is not proportionate. Rebuild on a fixed subsample, verify that the first thousand steps match instead of the full run, and do the complete rebuild quarterly or at release. The first thousand steps catch nearly everything, because environment and data problems show up immediately.
Setting the tolerance honestly
A tolerance picked out of the air produces one of two failure modes: an alert that fires constantly and gets muted, or a threshold so loose that a real regression sails through. Measure it instead.
Train the same configuration five times, changing only the seed, and record the spread of the primary metric on the frozen evaluation set. That spread is the natural variance of your training procedure, and it is the floor for any tolerance you set. If five seeds produce a range of half a point, a rebuild landing within half a point is behaving exactly as expected and a rebuild two points off is a genuine problem. Write both the measured spread and the chosen threshold into the pipeline configuration where the check can read them, and re-measure when the architecture changes.
Compare on the evaluation set, not on the weights. Two models with different parameters can be operationally identical, and two with similar parameters can diverge on the cases you care about. Check the primary metric, the metric on your most sensitive segments, and the prediction distribution. A rebuild that matches on aggregate accuracy but has shifted its score distribution is telling you something.
Seven ways a build quietly stops being reproducible
- A base image referenced by tag rather than digest, so a rebuild months later lands on a different runtime.
- A dependency file with version ranges in it, resolved fresh on every build.
- A training query against a live table with no snapshot reference, so the same SQL returns a different corpus each week.
- A train and test split produced by shuffling, which reassigns rows every time the corpus grows.
- Autotuning left enabled in the release path because it was faster during development.
- A manifest assembled by a person at release time instead of emitted by the job that trained the model.
- An evaluation set that lives in a shared folder and gets edited, so yesterday's metrics are not comparable to today's.
All seven are silent. None throws an exception, fails a test, or appears in a dashboard. They get found by a scheduled rebuild or by an incident, and the rebuild is cheaper.
A two-week retrofit
Retrofit Sequence
The sequence is ordered by dependency, not by importance. The manifest comes first even though it is mostly empty, because an empty field is a visible gap and a missing manifest is not. Step four is the long one, and it needs a data engineer rather than an ML engineer.
Score your own pipeline against the rubric below before you start. It takes twenty minutes and it usually settles the argument about where to begin.
Rebuild Readiness — Weighted Self-Score
Score each line 0 to 100 for your pipeline, multiply by the weight, divide by 100. Below 50 means a rebuild is a research project.
What this buys beyond an audit answer
Provenance is why teams start and the smallest part of the return. The daily payoff is debugging. When every input is pinned you can bisect a regression: old data with new code, then new data with old code, and the cause falls out in two runs. Unpinned, the same investigation is a week of guessing which of a dozen things moved.
Onboarding gets shorter: a new engineer reproduces last quarter's result on day two instead of asking six people what the pipeline used to do. Handoffs stop being lossy, because a manifest transfers the build rather than the folklore. Rollback becomes real, and a team that can restore last quarter's model takes more risk on a new architecture.
There is a governance dividend too. SOC 2 change-management controls and the ISO 27001 configuration expectations both want evidence that production artifacts trace to reviewed, versioned inputs. A manifest and a passing rebuild job answer that in one screenshot, and because they were built for engineering reasons rather than for the audit, they hold up when someone asks a second question. None of it needs a platform purchase: a JSON file emitted by a job, a digest in a Dockerfile, a lockfile with hashes, a snapshot id, a split rule, and a scheduled workflow. The shape is the same on managed training services, on plain virtual machines, and on Kubernetes.
Bottom line
Reproducibility is not a property you declare. It is a job that runs, compares two numbers, and fails when they disagree by more than an amount you measured. Pin the data first, the environment second, and the kernels third. Emit the manifest from the job. Set the tolerance from observed seed variance rather than from an opinion. Then schedule the rebuild and let the first one fail, because that failure is the cheapest version of a problem you would otherwise meet on a Tuesday when a customer is waiting.
Frequently asked questions
Three different things. Re-runnable means the job completes on a clean machine from a recorded reference. Equivalent means the rebuilt model scores within a stated tolerance on a frozen evaluation set. Bitwise means identical weights. Most teams need equivalence, and equivalence is the target that survives a hardware refresh.
On a single device in a fixed environment, yes, using deterministic algorithm selection, disabled autotuning, fixed thread counts and the required cuBLAS workspace setting. Across a changed device count or a different GPU generation, no, because reduction order and kernel selection change. Some operations have no deterministic implementation at all and will raise an error rather than run.
It depends heavily on the model. Our planning allowance is roughly 10 to 15 percent for deterministic kernels and disabled autotuning, more where constrained dataloading becomes the bottleneck, and substantially more if you also disable reduced-precision matmul. Measure it once on your workload. Most teams run experiments fast and apply the controls only to release builds.
Use the snapshot identifier your table format already provides if the data is in Iceberg, Delta Lake or Hudi. For file corpora, use content-addressed tooling such as DVC or lakeFS. If neither applies, write a manifest of every input file with its size and SHA-256, hash the manifest, and store that hash with the run. No data is duplicated and any change is detectable.
By rebuilding it on a schedule. A job reads the manifest, provisions a clean runner, reconstructs the environment from the image digest and the locked dependencies, reads the recorded dataset snapshot, retrains, and compares metrics on the frozen evaluation set against the recorded values. Pass or fail. Anything short of an executed rebuild is a description of intent.