Reproducing a published result

A rebuild is a measurement, not a chore

A team sets out to reproduce a published result for one of two reasons. Either that number is about to become the comparison point for something they are building, or somebody is preparing to spend money on the strength of it. Both make the rebuild an experiment with its own protocol and its own result. Run it that way and it is useful however it comes out. Run it as a box to tick before the real work and it produces a number nobody can interpret.

Three endings are legitimate. A match. A difference you can attribute to a specific decision, which is often worth more than a match, because that decision is now a knob you understand. And a difference you cannot attribute, which is still a finding if it is reported as one. Teams get into trouble treating the first ending as the only acceptable one, because that belief turns a measurement into a search for the settings that make the number appear.

None of this is about suspecting the authors. Edward Raff, who reimplemented 255 papers working from their text alone, closed his study by saying he had rarely had any suspicion that a paper's results were false. Reproduction failures are overwhelmingly failures of written communication under a page limit. The fix is a careful reading and an email, not an allegation.

Two different jobs wear the same word

The National Academies fixed the vocabulary in its 2019 report on reproducibility and replicability in science. Reproducibility means obtaining consistent results using the same input data, computational steps, methods, code and conditions of analysis. Replicability means obtaining consistent results across studies aimed at the same question, each of which collected its own data.

ACM aligned its artifact badging to that language, and badges issued from May 15, 2020 carry the revised version. Under it, Results Reproduced means an independent team obtained the paper's main results using the author-supplied artifacts, within an accepted tolerance for the type of experiment. Results Replicated means an independent team obtained them without those artifacts, by writing their own implementation from the paper's description or through a separate setup.

Which one you did decides what you may claim. Run the authors' container on the authors' data, get the authors' number, and you have shown the artifact is internally consistent. That is not evidence the method will work on your problem. Write your own implementation and match it, and you have shown the paper communicates a method that works, which is the condition under which its number can serve as a baseline.

Most teams doing this for delivery reasons want a third thing, the paper's method running on their own data. That is neither of the above, and it should be labelled as what it is, a transfer test. We have written separately on testing on somebody else's instrument.

What the record says about your odds

Raff's study is the only large empirical measurement of independent reproduction in machine learning we are aware of. He attempted 255 papers published between 1984 and 2017, writing his own code, counting a paper reproduced when the majority of its claims held. He reproduced 162, or 63.5 percent. Publication year had no significant relationship to success, which argues the difficulty is a stable property of technical writing rather than a recent collapse.

Two findings should change how a rebuild is planned. Whether the authors had released code was not statistically significant in his data. Whether an author answered an email was among the strongest dividers he measured. Readability, defined as how many passes through the paper were needed before a working implementation existed, had the strongest relationship of anything he tested. Papers specifying hyperparameters did better, papers with more tables did better, papers with more equations per page did worse. And despite having cluster access, he never reproduced a paper whose results required a cluster.

He also recorded, informally, why things failed: ambiguous notation, a step of the algorithm left out, loss functions given without the resulting gradients, and an unspecified detail everybody considered minor until it moved the result. Each is a communication gap answerable by a specific question to a specific author.

Rows two and three come from the RecSys 2019 best-paper study by Ferrari Dacrema and colleagues, who took 18 neural recommendation papers from top-ranked venues, rebuilt 7 with reasonable effort, and found 6 of those 7 could often be outperformed by simple heuristics such as nearest neighbours. The fourth is Kapoor and Narayanan's rebuild of the civil-war prediction literature, where every paper claiming complex machine learning beat logistic regression failed once leakage was corrected. Read the rows as a planning input: budgeting a rebuild as a two-day task is how it becomes a two-month task nobody scheduled.

Write the claim contract before you write code

Most rebuilds that turn into a mess do so because the target was never written down precisely enough to be hit or missed. Eight decisions, made in an hour and recorded in a file that ships with the work, remove most of that.

• Which number. Table, row, column, and the figure exactly as printed.
• Which data. Named release and version, download date, checksum, and the exact split, by file if one exists.
• Which metric implementation, not which metric name. Macro or micro, tie handling, truncation depth, tokenisation.
• Which arms. The method row and the baseline row it is compared against. Both get rebuilt, baseline first.
• Which budget. Search trials, epochs, and the stopping criterion, applied identically to every arm.
• Which spread. How many seeds, and the interval you will report around your own number.
• Which tolerance. The gap you will accept as reproduced, chosen before the first run.
• Which stop. The hour count at which you report the gap instead of continuing.

The fourth item is the one teams skip and the one that pays. Rebuilding the paper's baseline row before its method row separates a pipeline problem from a method problem in an afternoon. If the baseline does not come out, nothing about the method has been measured, and every hour on the method is spent debugging the wrong thing.

The five things that are almost never on the page

Preprocessing. Tokenisation, resampling, normalisation, outlier removal, deduplication, missing-value handling, lowercasing. These take a paragraph in a paper and a week in a rebuild, and they carry the most leakage risk: a normaliser fitted on the whole dataset before splitting is easy to write and invisible in a description.

The split. Which records went where, and by what unit. A random split on data with repeating subjects puts the same patient, vehicle or document family on both sides of the boundary. A paper that says "we split 80/20" has told you almost nothing, and you should assume your split differs until proven otherwise.

The search budget. How many configurations were tried before the reported one. This is the largest source of unfair comparison in published work and it almost never appears.

The stopping rule. Early stopping on which set, with what patience, on what metric. Early stopping on the test set is a leakage channel that produces a beautiful number and no transferable claim.

The metric implementation. Two libraries computing the same named metric will disagree, sometimes by more than the improvement being claimed. Ranking metrics differ on ties and truncation, text metrics on tokenisation, classification metrics on averaging and empty classes. Before deciding a rebuild missed, check that both sides compute the same function.

What the gap is telling you

A shortfall is not one thing. Treat the size and shape of the gap as a diagnostic and run the test that separates the candidate causes, rather than adjusting settings until the number improves. Six patterns cover most rebuilds.

What you observe	Most likely cause	The test that separates it	What you may say
Both arms far below the paper, its baseline included	Your pipeline. Data version, preprocessing, or metric implementation	Run a third-party implementation of the baseline through your pipeline	Nothing about the method yet. This is a pipeline result
Baseline row matches, method row falls short	A missing implementation detail, an unstated hyperparameter, or a derivation you guessed	Ablate one detail at a time against the paper's intermediate figures, then email the authors	Not reproduced from the text as written, gap named
Small shortfall, inside your own run-to-run spread	Seeds, initialisation, and library nondeterminism	Rerun your arm across seeds and compute the interval before comparing anything	Reproduced within tolerance, with the tolerance stated
Your number is higher than published	A different split, a stronger library default, or leakage you introduced	Check the split by repeating unit, and check every preprocessing step was fitted inside the training fold	Unexplained until the cause is found. Do not bank it
Headline matches, per-slice results diverge	A different data version or population mix	Compare slice composition against the paper's dataset description first	Reproduced in aggregate only, which is not enough to deploy on
Everything matches, including the design flaw	You inherited the original's leakage or its evaluation error	Build a clean split yourself, by repeating unit and by time, and rerun both arms on it	The number is real. The claim it supports may not be

The search budget is part of the claim

Dodge and colleagues gave the field a way to report this in 2019: expected validation performance of the best model found, plotted against the computation budget spent looking for it. They found published comparisons whose conclusions would have gone the other way with more or less compute. Tang and colleagues later showed the specific estimator is biased and derived an unbiased alternative. The disagreement is about the estimator, not the premise, and the premise is what matters here: a number without a budget attached is not comparable to one produced under a different budget.

So a rebuild done on a smaller budget than the original is not yet a baseline. If the authors searched two hundred configurations and you searched twelve, your shortfall is partly a budget measurement. Match the budget or report yours, and never compare a twelve-trial rebuild of their method against a two-hundred-trial tuning of your own.

The field has a long record here. Armstrong, Moffat, Webber and Zobel analysed ad-hoc retrieval results reported at SIGIR from 1998 to 2008 and at CIKM from 2004 to 2008 and found the improvements did not accumulate. A decade later Yang, Lu and Lin examined over a hundred papers, confirmed baselines were often not as strong as they should have been, and found no upward trend on one long-running test collection, with the best reported figures dating from years earlier.

The coda matters as much as the critique. A year after making his own version of that argument, Jimmy Lin published a recantation in SIGIR Forum: heavily pretrained transformer models had substantially improved retrieval effectiveness even without large training sets, and enough independent studies had cleared peer review for the result to be considered solid. A field that documents its weak-baseline problem and then revises in public when the evidence turns is a field whose numbers are worth rebuilding.

Measure the noise before interpreting the gap

Henderson and colleagues argued in 2018 that nondeterminism in standard benchmark environments, combined with variance intrinsic to the methods, can make reported results hard to interpret, and that without significance metrics and tighter reporting standards it is difficult to tell whether an improvement over the prior state of the art is meaningful. Bouthillier and colleagues modelled the whole benchmarking process and showed that variance from data sampling, parameter initialisation and hyperparameter choice moves results markedly.

Underneath that sits a floor you cannot engineer away. PyTorch's own documentation states that completely reproducible results are not guaranteed across releases, across individual commits, or across platforms, and that results need not be reproducible between CPU and GPU executions even with identical seeds. The available controls are worth using: deterministic algorithm selection, turning off the convolution autotuner, and seeding data-loader workers explicitly. They narrow the band. They do not close it.

The order of operations follows. Rerun your own arm across several seeds and compute the spread before looking at the published number. If the difference you are chasing is smaller than your run-to-run variation, you cannot resolve it, and neither could a paper reporting a single run.

A number can reproduce perfectly and still be wrong

Kapoor and Narayanan surveyed research communities that had adopted machine learning and documented leakage in 17 fields, organised into a taxonomy of eight types running from textbook errors to open research problems. Their conflict-prediction rebuild illustrates the failure mode that matters here: the papers reproduced their own arithmetic faithfully. The pipeline was consistent. The split was not, and the claim evaporated once it was corrected.

So a serious rebuild answers two questions. Did I get their number, and does their number mean what the abstract says. The second is answered by checks that have nothing to do with matching: split by the unit that repeats rather than by record, confirm every preprocessing step was fitted inside the training fold, look for near-duplicate records straddling the boundary, and ask whether any feature could only have been computed with knowledge of the outcome. If the original design fails those checks, you are producing the corrected number, which is the more valuable deliverable as long as the report says which one it is.

The same applies to the evaluation set. A benchmark that has circulated for years may have leaked into the pretraining corpus of whatever model you are comparing against, which makes a matched number reassuring for the wrong reason. Our notes on red flags in an AI benchmark cover that from the buying side.

Email the authors, and ask well

Raff's data makes this the highest-yield hour in the exercise. Among the papers where he contacted the authors, whether anyone replied separated the reproduced from the not-reproduced more sharply than almost anything else he measured. Researchers answer specific questions about their own work at a rate that surprises people who have not asked.

Three to five numbered questions, each anchored to a section, equation or table number, each answerable in a sentence. Include your intermediate figures so they can see where the two runs diverge, because an author who spots the fork will usually name it immediately. Ask about the baseline row too, since it is more often the source of a discrepancy and nobody ever asks. Do not open by saying their paper cannot be reproduced. Open with what you built, what you got, and the one decision you are unsure of.

Decide when to stop before you start

Put an hour budget on the rebuild in advance and write down the three endings you are prepared to file. Reproduced within tolerance. Reproduced after the following specific changes. Not reproduced from the text as written, here is the best number we obtained and every difference we know of. All three are defensible in front of a customer, and only the first is a match.

The endings not on that list are the expensive ones. Continuing indefinitely because stopping feels like failure burns a schedule with nothing to show. Quietly adopting a different split, metric variant or longer training run until the number appears produces a match no reviewer can audit. The stopping rule keeps both off the table when the team is tired and the number is close.

What the rebuild leaves behind

The output is not a number on a slide. It is a bundle somebody else can run. Pin the environment to versions rather than ranges. Ship a data manifest with the source, version, download date and checksum for every file. Record the seeds. Provide one command that produces the reported figure from a clean checkout. Keep both arms in the repository. And keep a divergence log: every place your setup and the paper's are known to differ, including the ones you believe are immaterial, because the reader decides that.

Archive what you depended on. Papers with Code, for years the default index linking papers to their implementations, was shut down in July 2025, its historical dataset preserved afterwards in a public repository. A URL printed in a paper is not storage. NeurIPS now points authors at the Software Heritage archive for a permanent citable source-code identifier, and a rebuild should keep its own copy of every artifact it relied on. The discipline that makes a rebuild auditable is the one that makes a delivered system auditable, covered in reproducibility audits for delivered AI systems and evaluation harnesses that survive a review.

Why buyers have started asking for this

The expectation is hardening in public, which helps whoever did the work properly. ACM's badging puts reproduction and replication on the paper itself, with a defined tolerance and an independent team. The Machine Learning Reproducibility Challenge, which began as an experiment at ICLR in 2018, is an official track at NeurIPS in 2026, anchored in a peer-reviewed journal, with organisers stating that negative results and partial failures to reproduce are as valuable as confirmations. NeurIPS requires a paper checklist covering reproducibility and desk-rejects papers that omit it; it stops short of mandating code release but requires a reasonable avenue for reproducibility.

On the funding side, the 2022 White House Office of Science and Technology Policy public access memorandum set a timeline under which agencies were to publish updated public access policies by the end of 2024, have them in effect by the end of 2025, and publish implementation plans covering metadata and persistent identifiers by the end of 2026, covering publications and the data underlying them. Implementation runs agency by agency and the details continue to move, so read the funding agency's own public access page.

The translation for a buyer is short. A supplier citing a published number as evidence should be asked whether they rebuilt it, on whose data, under what budget, and with what gap. One who volunteers the divergence log before being asked has said something about how the rest of the work will be done. That is the same test applied through reproducibility written into the contract and acceptance criteria for a machine learning deliverable. When we rebuild a published result, two things ship: the number, and every place our pipeline and the paper's differ.

Common objections

Bottom line

Rebuilding a published result is ordinary engineering with a real and well-documented failure rate. Decide in advance what number you are chasing, on what data, under what budget, within what tolerance. Rebuild the paper's baseline before its method. Measure your own noise before interpreting anyone's gap. Ask the authors specific questions. Stop when the budget says stop and file the honest ending. Then ship the divergence log with the number, because that is what lets somebody else use your work without repeating it.

Frequently asked questions