Data that lies: the bugs that look like alpha

A backtest is a claim about what would have happened. The claim is only as good as the data it ran on, and financial data fails in a handful of well-known ways. This post is the catalogue: each bug, what it does to a result, how to catch it, and the fix. Four of them are demonstrated on purpose.

a chosen rule or estimatea gain that looks realwhat it was worth once the bug is removed

The catalogue

Bug	What goes wrong	Fix
Look-ahead	a feature uses a number not yet known at the time	timestamp every input; lag it; one `.shift(1)`
Data snooping	test many ideas, keep the lucky ones	count the trials; a fresh holdout; a higher bar
Survivorship	today’s index members used for the past	a point-in-time universe including the dead names
Label overlap	forward-looking labels near the split peek into the test period	purge rows near the boundary; leave a gap
Revisions	restated figures that were unknown at the time	as-first-released data
Splits and dividends	unadjusted prices show a fake −50%, or miss the cash	adjusted close; total return
Delisting	a dead stock dropped at its last price	include the delisting return
Stale prices	forward-filled gaps become zero returns then a jump	flag the no-trade days

Data snooping: a thousand ideas that know nothing

Make 1,000 trading signals by flipping a coin every day: long SPY on heads, short on tails. Score each on 2019 to 2021. By luck alone, about 2.3% of them should look significant at the usual bar ( $t > 2$ ).

Item	Result
Signals that look significant on 2019 to 2021	18 of 1,000
Their average Sharpe on those years	$\good{1.40}$
Their average Sharpe on 2022 to 2023	$\bad{-0.27}$
The single best of the 1,000: Sharpe before / after	1.68 / −0.49
Correlation between in-sample and out-of-sample Sharpe, all 1,000	−0.04

The eighteen “discoveries” had an average Sharpe of 1.4, a number a fund would advertise, and every one of them was a coin. The first panel of the chart is the whole lesson: a cloud with no slope. Whatever a signal scored where it was chosen says nothing about what it will score next.

The fix is to count. Testing $N$ ideas at a 5% bar expects $0.05 N$ false positives; with 1,000 ideas that is 50. The Bonferroni correction divides the bar by the number of tests: 0.05/1,000 is 0.00005, a $t$ of about 3.9. A published rule of thumb for new factors is $t > 3$ rather than 2, because of how many have been tried. And if test results guided the design, the holdout has become training data, however carefully it was kept apart on disk.

Look-ahead: one missing shift

The backtesting post showed the simplest form: a crossover rule that sets today’s position from today’s close reports +53.3% on SPY; the same rule set from yesterday’s close reports +42.5%. Ten points from a missing .shift(1).

The subtler form hides in preprocessing. Build a “high volatility” flag as a z-score of SPY’s 20-day volatility, centred on the full five years’ mean. The flag fires on 93 days. Build it with only the mean known so far, and it fires on 111 different-looking days; the two first disagree on 13 May 2019, because the full-sample version already knows that 2020 is coming and sets its bar accordingly. The third panel of the chart draws both. Any statistic computed over train and test together, a mean, a standard deviation, a feature-selection step, a scaler, leaks the future into the past. Fit every data-dependent step on the training rows only.

The test for a leak: shift the suspect feature by a day, or remove it, and rerun. If earlier predictions change, something was looking forward. A result that is too good is a result to check before it is a result to report.

Survivorship: built on purpose

A survivor-only universe is what you get by downloading today’s index members and testing on their past. To see what it does, I made 200 stocks from scratch: three years of random daily returns with a small common drift, and a slightly different drift per stock. At the end, any stock that had lost more than half its value was dropped from “the index”. 25 of the 200 were.

Item	Full universe	Survivors only
Average yearly return of a stock	+9.9%	$\good{+13.5\%}$
Buy the 20 cheapest after year one, hold two years	$\key{+12.7\%}$	$\good{+36.0\%}$

The survivors return more, by construction: the ones that fell were removed. The damage is worst for strategies that buy what has fallen, because in a survivor-only universe every stock that fell went on to recover, since the ones that did not are missing. One published check on real data found the same rule returning −42% on a full database and +388% on a survivor-only one. More survivor data does not fix this; it is a systematic bias, not a sampling error.

Adjusted prices, dividends and the dead

A raw price series misleads in three ways:

Case	Raw price says	The truth
$50 to $48 with a $2 dividend paid	−4%	total return 0%: the $2 went to the holder
A 2-for-1 split, $100 to $50	−50%	nothing happened; twice the shares at half the price
Last print $4.00, holders receive $0.50 in liquidation	the stock vanishes	−87.5% that a backtest never charged

The adjusted close folds splits and dividends into earlier prices: a 2-for-1 split halves every earlier price, and a dividend trims them by its yield, so the day-to-day returns are what a holder actually earned. Fill price gaps forward first, then backward only where nothing came before, so a trading halt repeats the last known price instead of inventing one. Keep the dead tickers in the price folder; a bankrupt company’s symbol often ends in Q, and a universe that still has those files is what point-in-time looks like on disk.

The fixes, in order of effort

Lag everything. Every feature uses data up to the previous close. One line.
Fit preprocessing on training rows. Scalers, imputers, feature selection, inside the pipeline, inside each fold.
Count the trials, and raise the bar to match. Keep a research log of what was tried, including what failed.
Hold out time, not rows, and touch the holdout once.
Use adjusted, total-return, point-in-time data, delistings included.
Shift and ablate. Lag the suspect, remove the suspect, rerun. If the result moves more than it should, it was a leak.

Traps

Trap	Truth
18 significant signals out of 1,000 means 18 real ones	About 23 were expected by chance
Revised data is better for backtests	It was unknowable at the time
A split-day −50% is a loss	An adjustment artefact
A bigger survivor-only sample fixes survivorship	The bias is systematic
Shuffling the rows removes leakage	It spreads it
A longer history always helps	Not across a regime change
Regularisation stops leakage	It cannot; leakage is in the inputs

The code

import numpy as np
import pandas as pd

prices = pd.read_csv("prices.csv", index_col="date", parse_dates=True)   # adjusted closes
r = prices.pct_change().dropna()
tr, te = r.loc[:"2021", "SPY"].to_numpy(), r.loc["2022":, "SPY"].to_numpy()

rng = np.random.default_rng(1)
signals = rng.choice([-1.0, 1.0], size=(1000, len(tr)))          # 1,000 coins
pnl = signals * tr
t = pnl.mean(1) / pnl.std(1, ddof=1) * np.sqrt(len(tr))          # a t-statistic per signal
lucky = t > 2                                                     # 18 of them
sharpe_out = (rng.choice([-1.0, 1.0], size=(1000, len(te))) * te)  # the same coins, next period
np.sqrt(252) * sharpe_out[lucky].mean(1).mean() / sharpe_out[lucky].std(1).mean()   # about -0.3

vol = r["SPY"].rolling(20).std()
leaky = (vol - vol.mean()) / vol.std()                            # uses the whole sample's mean
honest = (vol - vol.expanding().mean()) / vol.expanding().std()   # uses only what was known

Summary

Idea	Rule	Remember it as
False positives	tests × α	1,000 × 5% = 50 by luck
Bonferroni	α ÷ number of tests	the bar rises with the trials
Look-ahead	every input lagged to the last close	one missing shift is ten points
Pooled statistics	fit on training rows only	a scaler can leak
Survivorship	point-in-time universe	the dead stay in
Total return	$(P_1 + D - P_0) / P_0$	the dividend counts
Delisting	charge the last loss	−87.5% that a backtest skips

References

Campbell R. Harvey, Yan Liu and Heqing Zhu, ”… and the Cross-Section of Expected Returns”, The Review of Financial Studies 29(1), 5–68, 2016, the case for a t-statistic above 3
scikit-learn: common pitfalls and recommended practices, data leakage through preprocessing
Ernest P. Chan, Quantitative Trading: How to Build Your Own Algorithmic Trading Business (2nd ed., Wiley, 2021), chapter 3: survivorship bias, look-ahead bias and adjusting for dividends
Philip J. Romero and Tucker Balch, What Hedge Funds Really Do: An Introduction to Portfolio Management (Business Expert Press, 2014), chapter 12: adjusted prices, missing data and missing tickers