Backtesting that does not lie to you

The instrument is biased in one direction

A backtest is a measurement instrument, and this one has a systematic bias that no amount of care fully removes. The reason is organizational rather than statistical: an error that hurts the result gets investigated within a day, because somebody is annoyed and goes looking. An error that helps the result gets celebrated, socialized, and defended, and the people best placed to find it are now invested in it being real. Every process below exists to counteract that asymmetry, and none of them work if the process is run by the person who wants the answer.

Almost all failures land in two families. Leakage means the backtest used information that would not have been available at the moment of the decision. Selection means you looked at many things and reported the best one. Leakage is an engineering defect and it is findable. Selection is a process defect and it is not findable at all after the fact, which is why it has to be handled before the work starts.

Leakage one: restatements

Any field that can be revised — reported figures, corrected vendor values, index membership, risk classifications, ratings — is a lookahead unless your history records when each value became available. A filter written as WHERE period_end <= t passes review because it looks careful; it reads whatever the value was eventually corrected to, and the correction happened after the decision.

The fix is a store that keeps assertions rather than values, and reads that take an as-of date. The check, if you are auditing someone else's work, is one question: for a randomly chosen date in the history, can the system reproduce the exact input the strategy saw, including the values later revised? If the answer needs a caveat, you have found a leak.

Leakage two: the decision clock

Two timestamps get conflated constantly. There is the moment the data existed and the moment you could act on it, and the gap between them is where the profit in a bad backtest lives.

The concrete cases are familiar and still ship: deciding on a closing price and filling at that same close; using a daily bar's high or low, which are only known once the day is over; consuming an overnight file at its file date rather than its actual delivery time; and, in machine learning generally, computing a feature from an aggregate whose window includes the label period. Each is a one-line error with a large effect, and every one of them looks correct in a code review because the wrong timestamp is not visibly wrong.

The structural fix is to make decision time an explicit parameter of the simulation rather than an emergent property of the join. Every input is fetched as-of the decision time, every fill happens strictly after it, and any code path that can see a timestamp greater than the decision time raises rather than returns.

Leakage three: the universe

If the set of things you trade or score is built from what exists today, the backtest is conditioned on survival and everything measured on it is optimistic. This is the best-known bias in the field and it still arrives in production work, usually because the instrument master only holds live entities and nobody framed that as a modelling choice.

Two details separate a serious treatment from a nominal one. Membership must be a dated relation, so you can reconstruct the universe as of any past date rather than filtering today's list. And the delisted must carry their terminal outcome, because dropping an entity from the sample on the day it collapses removes precisely the observation that mattered. A universe that includes dead names but excludes their final period is survivorship wearing a correction.

Leakage four: overlapping labels

Standard cross-validation assumes rows are exchangeable. Time series rows are not. If a label is computed over a forward window of twenty days, then rows one day apart share nineteen days of outcome, and a random split puts nearly identical information on both sides of the fold boundary. The model does not need to generalize; it needs to remember.

The remedy is well established: purge the training set of any observation whose label window overlaps the validation window, and add an embargo of a few periods after the validation block to account for serial correlation that outlives the label. This costs training data and it makes the number smaller, which is the point. If your validation scheme produces a large improvement over a plain time split, be suspicious of the scheme rather than pleased with it.

Leak	How it shows up	The cheapest test
Restatement lookahead	Results concentrated around reporting dates	Reproduce a past day's inputs exactly, revisions included
Decision-clock error	Performance collapses when execution is delayed one period	Shift every signal one extra period back and rerun
Survivorship	Suspiciously smooth history; few large losses	Reconstruct the universe as of five years ago from your own data
Overlapping labels	Random-split validation far better than a time split	Purge and embargo, then compare
Fill optimism	Result scales worse than linearly with size	Trade at the next open instead of the decision price
Selection	Best variant sits far above all the others	Count the trials, then look at the whole distribution

Five tests that catch most of it in a day

None of these require new infrastructure. All of them are worth running before a result is presented rather than after it is questioned.

Delay everything by one extra period. If the result largely survives, the signal has some persistence and the timing is probably not the whole story. If it evaporates, either the edge lives entirely inside one period, which has cost implications you have not modelled, or your timestamps are wrong.

Flip the sign. A strategy whose inverse also makes money is not measuring what you think. It is measuring the cost model, a fill assumption, or a bug that pays either way.

Shuffle the outcomes. Break the link between inputs and results, keeping everything else identical, and rerun the whole pipeline. Anything that still performs is leakage by construction, and this catches feature-engineering leaks that no amount of reading the code will surface.

Run noise through the machinery. Replace the signal with random series that match its autocorrelation and run the full search, including the selection step. The distribution of results you get is the null distribution of your own process, and the honest question about your real result is where it sits in that distribution. Teams are routinely surprised by how good pure noise looks after a search.

Move the fill. Execute at the next available price instead of the decision price. If most of the result disappears, the fills were doing the work, and no live implementation will reproduce them.

Selection: the part that cannot be fixed afterwards

Suppose you evaluate forty variants against the same history. Some will look good. At a conventional significance threshold you would expect a couple of false positives from forty independent tries, and variants of one idea are heavily dependent, which distorts the arithmetic further rather than saving you. The number you present is the maximum over a search, and the maximum over a search is not an estimate of future performance.

There is a literature on adjusting for this. The deflated Sharpe ratio proposed by Bailey and López de Prado discounts a reported result by the number of trials and the shape of their distribution. Harvey, Liu and Zhu argued, from the accumulated body of published factors, that the conventional significance hurdle is far too low once the volume of testing behind it is counted. Both arguments reduce to the same practical instruction: the trial count is part of the result, and a result reported without it cannot be interpreted.

So count. Keep a trial registry — every configuration run against the evaluation data, including the ones abandoned in ten minutes, with a date and an owner. It is unglamorous and it takes a few lines in the harness. Without it, nobody in the organization can distinguish a genuine finding from the best of two hundred coin flips, and the person least able to distinguish them is the one who ran the search.

Costs and capacity decide more results than signal does

A cost model of a fixed number of basis points per trade is fine for a first pass and dangerous as a conclusion. Real costs have three parts and only the first is constant: explicit fees, the spread you cross, and impact, which grows with the size you are pushing relative to available volume. Impact is commonly modelled as scaling with the square root of participation, and the practical consequence is that a strategy is not a thing with a return — it is a curve of return against size, and somewhere on that curve it becomes negative.

Report the curve. Estimate the size at which expected return net of costs reaches zero, and state it beside the headline. A strategy that is excellent at a small size and gone at a meaningful one is a real finding, and it is a completely different business decision from one that scales. Related and equally omitted: the availability and cost of borrow for anything sold short, and whether the turnover implied by the signal is achievable at all given typical daily volume in the names it selects.

Walk forward, and be honest about retraining

Walk-forward evaluation — fit on a window, evaluate on the next block, roll — is the closest simulation of how the thing will actually be run, and it is only honest if the retraining cadence in the test matches the cadence you will operate. A backtest that refits monthly, run by a team that will refit annually because nobody has time, is measuring a system that will not exist.

The same applies to every parameter chosen by looking at the whole history: a lookback length, a threshold, a universe filter, a risk limit. If it was chosen with hindsight, it must be chosen inside the walk-forward loop using only past data, or it is a free parameter fitted to the answer. This is where most of the remaining optimism sits after the obvious leaks are removed, and it is the reason walk-forward results are usually meaningfully worse than the equivalent single split. That is a feature.

This is the same list for any machine learning team

Change the nouns and the failure modes are identical. The offline metric that does not survive deployment is nearly always one of: a feature computed from a window that includes the label; a training population assembled from entities that still exist; a random split across correlated records; an evaluation set that has been looked at fifty times; or a metric that ignores the cost of acting on the prediction. A recommendation model evaluated on logged interactions is measuring a policy that already ran, which is the same shape of error as filling at the price you used to decide.

The productive move is the same too: build the harness so that leaking is hard rather than merely discouraged. Fetch every input as-of the decision time, make the evaluation split a function of time by construction, and let the framework raise when code reaches for a timestamp it should not have. Discipline that depends on everyone remembering does not survive a quarter of deadline pressure.

The mistakes we are called in to fix

• Filtering revised fields by period end, reading corrections that arrived weeks later
• Deciding on a price and filling at it, which is a free option nobody actually has
• A universe built from entities that still exist, so the failures were never sampled
• Random cross-validation over overlapping labels, rewarding memory and calling it accuracy
• An untracked search, so the trial count is unknown and the result uninterpretable
• A holdout with no custodian, quietly consulted several times
• Fixed basis-point costs at every size, hiding the point where the strategy dies
• Parameters chosen over the full history and then evaluated on it

A two-week audit of an existing result

Step one ends more audits than the rest combined. A result that cannot be reproduced from raw inputs by a second person, without a notebook that has been running for three weeks, is not a result yet — and the reproduction attempt itself surfaces most of what the later steps are looking for.

Before you present it

• Reproducible end to end from raw inputs by someone who did not build it
• Every input fetched as-of an explicit decision time
• Revised fields sourced from a store that keeps assertions, not values
• Universe reconstructed as a dated relation, terminal outcomes included
• Purge and embargo applied wherever labels overlap
• All five sanity tests run, with results written down
• Trial count recorded, and the distribution of trials shown
• Holdout held by someone else and opened once
• Costs modelled as a curve; the zero-return size stated
• Retraining cadence in the test equal to the cadence you will run

Bottom line

Backtesting is not hard because the statistics are subtle. It is hard because the incentives point one way and the checks are voluntary. Leakage is an engineering problem and a harness that fetches everything as-of a decision time removes most of it permanently. Selection is a process problem and only a trial registry plus a custodied holdout addresses it, both of which must exist before the search starts. Run the five tests on anything that looks unusually good, publish the trial count next to the result, and report the size at which the strategy stops working. The result you get will be smaller, and it will be the first one that survives contact.

Frequently asked questions