Backtesting in Python

A trading rule is a hypothesis: “when the 20-day average price crosses above the 100-day average, the market is turning up, so buy.” A backtest is the experiment that checks it. You take years of past prices, apply the rule day by day as if you had been there, keep the books (cash, position, profit and loss), and read off the result at the end. Every systematic strategy, from a hedge fund’s model to a retail investor’s moving-average rule, goes through this step before it touches money, because the alternative is finding out with real money.

The result is a number, and people believe numbers from a backtest too easily. This post explains what a backtest is made of, builds the simplest honest one in a few lines of pandas, runs the same rule through two Python libraries, and uses the differences between the answers to show where backtests go wrong. The last section is a guide to the libraries worth knowing.

What a backtest is made of

Five parts, whatever the library:

• Historical prices. Daily bars with open, high, low, close and volume, or finer. The quality of the data bounds the quality of the test: prices must be adjusted for splits and dividends, and the set of tickers must be the one that existed then, not the survivors.
• The rule. A function from everything known up to today to a position for tomorrow. Moving-average crossovers, momentum, mean reversion, a machine-learning model’s forecast.
• The execution model. The part beginners skip and professionals spend most of their time on. When does the order fill, at what price, and what does it cost in commission, bid-ask spread, slippage (the price moving against you between decision and fill) and market impact (your own order moving the price)?
• Portfolio accounting. Cash, shares held, mark-to-market value, every day.
• Evaluation. Total return, Sharpe ratio, maximum drawdown (the worst peak-to-trough fall), number of trades, win rate, time in the market.

There are two ways to build the machine. An event-driven backtester walks through time one bar at a time, calling your code at each step with only the data up to that point, exactly as a live trading system would; it is realistic, it can model partial fills and queued orders, and it is slow. A vectorised backtester computes the signal for every day at once as array operations and multiplies positions by returns; it is far faster, which makes it the tool for parameter sweeps, and it is easy to leak tomorrow’s data into today’s decision because all of time is in one array. The example below uses both.

One rule by hand: the look-ahead trap

The rule: hold SPY when its 20-day average close is above its 100-day average, hold cash otherwise. One formula says everything a vectorised backtest does:

s, the position (1 = holding SPY, 0 = cash)

The return the rule earns on day $t$ is the market’s return $r_t$ times the position decided the day before. The subscript is the point. The position on day $t$ is computed from day $t$‘s close, so it cannot earn day $t$‘s return; the earliest it can act is day $t+1$. Write $\key{s_t} \, r_t$ instead and the backtest trades on information it did not have yet. That is look-ahead bias, and in a vectorised backtest it is one missing .shift(1).

import pandas as pd

spy = prices["SPY"]                      # adjusted daily closes, 2019-2023
ret = spy.pct_change().fillna(0.0)
fast, slow = spy.rolling(20).mean(), spy.rolling(100).mean()
in_market = (fast > slow).astype(int)    # 1 = hold SPY, 0 = hold cash

cheat = in_market * ret                  # set on the same close: impossible
honest = in_market.shift(1) * ret        # position set on the next day
total = lambda r: (1 + r).prod() - 1

The cheating version is not a wild number. It is $\key{10.8}$ percentage points better than the honest one, which is exactly the kind of gap that makes a mediocre rule look like a good one. The reason is mechanical: a crossover rule buys after a rise has started and sells after a fall has started, so every entry and exit is late by construction. Shifting the position by one day makes it one day later still, and the days just after a crossing are the days the trend is strongest.

Note. A look-ahead bug never announces itself. The code runs, the chart looks plausible, the numbers are a bit too good. The test is to ask, for every input to the rule on day $t$, “was this number printed before the close of day $t$?” If any answer is no, the result is fiction.

Panel 1 shows the rule’s thirteen position changes over five years; the signal is slow by design, which is why it sat out most of the March 2020 crash and also most of the recovery. Panel 2 is the money: the honest rule turned $\usd 10{,}000$ into about $\usd 11{,}600$ with a 0.1% commission, while buy and hold reached $\usd 20{,}600$. On SPY over these five years, the crossover rule lost to doing nothing, and lost less badly in the crash; that is the trade it offers.

The same rule in a library: the fill-timing trap

backtesting.py is the smallest full backtester in Python: event-driven, one instrument, a Strategy class with two methods, a Backtest that runs it, and a statistics table at the end. The crossover rule is eight lines.

from backtesting import Backtest, Strategy
from backtesting.lib import crossover

class SmaCross(Strategy):
    fast, slow = 20, 100

    def init(self):
        sma = lambda s, n: pd.Series(s).rolling(n).mean()
        self.fast_ma = self.I(sma, self.data.Close, self.fast)
        self.slow_ma = self.I(sma, self.data.Close, self.slow)

    def next(self):                       # once per bar, data up to that bar
        if crossover(self.fast_ma, self.slow_ma):
            self.buy()
        elif crossover(self.slow_ma, self.fast_ma):
            self.position.close()

bt = Backtest(ohlc, SmaCross, cash=10_000, commission=0.001,
              finalize_trades=True)
stats = bt.run()

The hand version said $\key{+42.5\%}$. The library says $\key{+16.0\%}$. Same rule, same data, same five years. The gap is three conventions, none of them a bug:

1. The library waits for a real crossing. The hand version is “in the market” from the first day the 100-day average exists, because the fast average already sat above it. The library’s crossover needs the fast line to actually cross the slow one, so its first buy is in September 2019 rather than May. That skipped a summer of gains.
2. The library fills one day later. A signal on day $t$‘s close becomes an order that fills at the next bar’s open. With daily closes only (the example sets open equal to close), that is the close of day $t+1$, and the position earns from day $t+2$. The hand version earned from day $t+1$.
3. Commission. 0.1% of every fill, $\usd 125$ over six round trips. Small here; for a rule that trades daily it is everything.

Put the first two conventions into the hand version and it reports $\key{+17.4\%}$; the rest is the commission. Neither number is the right one. What matters is that one day of fill delay moved the result from 42% to 17%, and a strategy whose backtest depends that much on when the fill is assumed to happen is a strategy you do not understand yet. Every library makes these choices for you; read the defaults before you believe the output.

Searching for parameters: the over-fitting trap

Why 20 and 100? Only because they are round numbers. The natural next step is to try every pair and keep the best, which is the job vectorised backtesting is built for. vectorbt runs every parameter combination as one wide array, using NumPy and Numba under the hood. The 1,225 fast/slow pairs on three years of SPY below took 9 seconds on the first call, almost all of it Numba compiling, and 0.12 seconds on the second.

import numpy as np, vectorbt as vbt

windows = np.arange(5, 255, 5)            # 50 windows, 1,225 fast/slow pairs
fast_ma, slow_ma = vbt.MA.run_combs(train, window=windows, r=2,
                                    short_names=["fast", "slow"])
pf = vbt.Portfolio.from_signals(train,
                                fast_ma.ma_crossed_above(slow_ma),
                                fast_ma.ma_crossed_below(slow_ma),
                                fees=0.001, freq="1D")
total_return = pf.total_return()          # one number per pair
best = total_return.idxmax()

The sweep is run on 2019 to 2021 only. The pairs are then scored on 2022 to 2023, which the search never saw: the same discipline as a train/test split in machine learning, for the same reason.

Panel 3 of the chart plots all 1,225 pairs, in-sample return across, out-of-sample return up. It is a cloud with no slope: the rank correlation between the two is $\key{-0.02}$. The best tenth of pairs in-sample went on to earn a median $\key{+10.1\%}$; the worst tenth earned $\key{+11.0\%}$. The search found nothing real. It found the pairs that happened to fit 2019–2021, and 2022–2023 was a different market (a bear year, then a recovery) in which a different set of pairs happened to fit.

Note. Picking the best of 1,225 backtests and reporting its return is not a backtest of a strategy. It is a backtest of your search, and the expected result of searching is to find luck. The honest number is the one from data the search never touched, and even that gets used up: look at the test set twice and it has become training data.

There is a human side to this. A backtest reads like a record of what happened, and we treat records as facts. The search that produced it is not in the chart. So “it backtested at 70%” persuades people who would never believe “I tried 1,225 things and this one worked”, even though it is the same claim.

Habits that keep a backtest honest

• Lag the signal. The position on day $t$ comes from data up to day $t-1$. In pandas that is .shift(1); in an event-driven library it is the default, which is a reason to use one.
• Charge for every fill. Commission, spread and slippage, with a pessimistic guess if you have no better one. A rule that only works without costs does not work.
• Hold out time. Choose parameters on one period, report on a later one. For a strategy you will keep re-tuning, use walk-forward testing: fit on a window, test on the next, slide, repeat, and report only the stitched-together test periods.
• Count the trials. Keep a note of how many variants you tried, and treat the best one’s result with the suspicion that number deserves.
• Use the data that existed. Tickers that were later delisted belong in the test; a universe built from today’s survivors backtests beautifully.

The libraries

Six names come up. The short list is two.

backtesting.pyOne instrument, a rule in a few lines, a statistics table and an interactive chart

Whythe smallest complete event-driven backtester: Strategy with init and next, a Backtest that runs it, built-in parameter optimisation with heatmaps; built on pandas, NumPy and Bokeh; actively released

Move on whenyou need a portfolio of many instruments, or you want to sweep thousands of parameters

vectorbtThousands of parameter combinations, many assets, in seconds

Whyvectorised on NumPy and Numba (with an optional Rust engine), so a whole grid of strategies is one array operation; rich portfolio statistics; the open-source community edition of a commercial product, under Apache 2.0 with a Commons Clause (you may not sell it as a product)

Move on whenthe rule needs path-dependent logic that does not vectorise, or you need realistic order handling

BacktraderA full event-driven platform: multiple data feeds, timeframes, brokers, sizers, analysers

Whythe most complete pure-Python engine, written by Daniel Rodriguez from 2015, with live-trading adapters for Interactive Brokers and Oanda and a large archive of answered questions

Move on whenyou need maintenance: the last release on PyPI is from April 2023 and the repository last changed in 2024, so it runs on today's Python by goodwill, not by a release cycle

Zipline (reloaded)Research over a universe of stocks with factor models and a point-in-time database

Whythe event-driven engine Quantopian built and ran its crowd-sourced fund on, until it closed in late 2020; now the zipline-reloaded fork maintained by Stefan Jansen for the readers of his book; initialize and handle_data callbacks, data bundles you ingest once, and the Pipeline API for computing factors across every asset on every day

Move on whenyou want one ticker and a quick answer; the data ingestion alone is an afternoon

LEANInstitutional-grade: many asset classes, C# or Python, the same code for backtest and live

WhyQuantConnect's open-source engine (Apache 2.0), event-driven, with brokerage integrations; the lean CLI runs backtests, research notebooks and live trading locally in Docker

Move on whenyou are not planning to trade live through it: the engine is heavy, and the data model assumes QuantConnect's formats

PyAlgoTradeHistorical interest only

Whyan early, readable event-driven library; its README now says it is deprecated and no longer maintained, the last release was in 2018, and its author's successor project is Basana, an async framework aimed at crypto exchanges

Move on whennow; read it to learn how an event loop, a broker and a strategy fit together, then use something maintained

The practical pattern: prototype in pandas, where you can see every line and there is nothing to misunderstand; sweep in vectorbt when there are parameters to search; confirm in an event-driven engine (backtesting.py for one instrument, LEAN or Backtrader for a portfolio) so the fill timing and costs are modelled the way a live system would. If the three disagree, as they did above, that disagreement is what you have learned.

Where this shows up at work

A backtest is the same object as a replay test in a settlement or payments system: historical inputs, the new logic, a diff against what actually happened. The traps are the same too. Using a field that was back-filled later is look-ahead. Tuning a fraud threshold on last month’s cases and reporting its catch rate on last month’s cases is in-sample. Replaying only the transactions that completed is survivorship. If you have built a replay harness you already know how to read a backtest, and when to distrust one.

Summary

Next

Portfolio optimisation covers what to do with several assets once you trust the numbers; the Sharpe, Sortino and information ratio post explains the ratios a backtest reports. The scripts behind this post run on the same five tickers as the rest of the site.

References

• Philip J. Romero & Tucker Balch, What Hedge Funds Really Do (Business Expert Press, 2014), ch. 2 on returns and risk measures.
• Marcos López de Prado, Advances in Financial Machine Learning (Wiley, 2018), ch. 11 “The Dangers of Backtesting”, on backtest over-fitting and counting trials (outside the usual reading list on this site, cited because it covers the sweep in this post).
• Library documentation, read for the cards above: backtesting.py, vectorbt, Backtrader, Zipline reloaded, LEAN and PyAlgoTrade.