Bias and variance: where a model's errors come from

This is the idea behind overfitting, and it’s explained in section 2.2.2 of An Introduction to Statistical Learning. It’s the reason a model that looks perfect in testing can fail in production, and the reason the next three posts (bagging, random forests, boosting) exist at all.

In the decision trees post, a fully grown tree scored 1.000 on its training months and 0.615 on the test months. This post explains why, with a formula you can actually measure.

The idea, with a person

Imagine two new risk analysts forecasting next month’s volatility.

Analyst A has one rule: “a jumpy month is followed by a jumpy month.” Simple, consistent, and often wrong in the same way. That’s bias: a model too simple for the real pattern.
Analyst B remembers every single month they’ve ever seen and predicts from the most similar one. Give B a different few years of history, and B’s forecasts change completely. That’s variance: a model that reacts too much to the particular data it learned from.

A good analyst is in between: they learn the real pattern and ignore the one-off quirks.

The formula

For a model $\key{\hat{f}}$ trained on some data, the expected squared error on a new point $x_0$ splits into three parts. This is equation 2.7 in An Introduction to Statistical Learning:

f̂, the model's predictionf and y, the truthbiasvariance

\begin{aligned} E\left(\good{y_0} - \key{\hat{f}}(x_0)\right)^2 &= \risk{\text{Var}}\left(\key{\hat{f}}(x_0)\right) \\ &\quad + \left[\bad{\text{Bias}}\left(\key{\hat{f}}(x_0)\right)\right]^2 \\ &\quad + \text{Var}(\varepsilon) \end{aligned}

Written out term by term (same thing, with each part spelled out):

\begin{aligned} E\left[\left(\good{y} - \key{\hat{f}}(x)\right)^2\right] &= \underbrace{\left(E[\key{\hat{f}}(x)] - \good{f(x)}\right)^2}_{\bad{\text{bias}}^2} \\ &\quad + \underbrace{E\left[\left(\key{\hat{f}}(x) - E[\key{\hat{f}}(x)]\right)^2\right]}_{\risk{\text{variance}}} \\ &\quad + \underbrace{\sigma^2}_{\text{noise}} \end{aligned}

Symbol	Meaning
$\good{f(x)}$	the true average answer at $x$ , the real pattern
$\key{\hat{f}}(x)$	what our model predicts at $x$
$E[\key{\hat{f}}(x)]$	the model’s average prediction, if we could train it again and again on fresh samples
$\sigma^2$	the variance of the noise: randomness in $\good{y}$ that no model can predict

In words:

Bias²: how far the model’s average guess is from the truth. Too simple → large.
Variance: how much the model’s guess jumps around depending on the sample it trained on. Too complex → large.
Noise: the part of the error that stays even with a perfect model.

The expectation $E[\cdot]$ here is over many different training samples. That is the thing to hold on to about variance. It is not a property of one trained model. It measures how much the model would change if you had collected different data.

Measuring it

Usually you can’t measure bias and variance, because you don’t know the true pattern $\good{f(x)}$ . With made-up data, you can. I invented a true curve for next month’s volatility as a function of this month’s, shaped like the real SPY data in the linear regression post: a gentle slope, plus a step up once the market is already above about 25% (a stress regime feeds on itself). Then I added random noise with a standard deviation of 3 points (so $\sigma^2 = 9$ ).

Then, for each tree depth, I repeated this 300 times: draw 80 fresh months, train a regression tree, record its predictions. That gives me 300 versions of each model, enough to measure its average and its spread.

Tree depth	Bias²	Variance	Noise	Bias² + variance + noise	Measured error on new months
1	8.00	6.25	9.00	23.24	23.29
2	0.57	4.46	9.00	14.02	14.00
3	0.01	2.82	9.00	11.83	11.82
5	0.02	6.09	9.00	15.11	15.06
8	0.03	8.49	9.00	17.52	17.44
15	0.04	9.11	9.00	18.14	18.21

Two things to see. First, the formula works: the three parts add up to the error I measured directly on new months, to within simulation noise. Second, the trade-off:

Depth 1 (blue on the right): a single question. Every version gives nearly the same flat steps, and all of them miss the shape of the curve. High bias.
Depth 15 (red): every version zig-zags through its own noise, and no two agree. Bias is almost zero, because on average they follow the curve, but each one alone is far off. High variance.
Depth 3 gives the lowest total error here. Not the most flexible model, not the simplest. Three questions are enough to place the step and the slope on either side of it.

A detail I did not expect: depths 1 and 2 have more variance than depth 3. With so few splits, where the cut-offs land jumps around a lot between samples. The general trend (more depth, more variance) holds after that.

Overfitting and underfitting

These are the everyday names:

Item	Underfitting	Overfitting
Cause	Model too simple	Model too complex
Main error	Bias	Variance
Training error	High	Very low
Error on new data	High	High
Fix	More flexible model, better features	Simpler model, more data, or averaging

The tell-tale sign of overfitting is a big gap between training error and error on new data, like the decision tree with a perfect 1.000 on training months and 0.615 on test months.

How to find the sweet spot in practice

You never know the true curve in real life, so you can’t compute bias and variance directly. Instead:

Hold out data. Train on part of the history, test on the rest. Only the test score counts.
Cross-validation. Split the data into, say, 5 parts. Train on 4, test on the 5th, rotate, and average. More reliable than one split.
For time series, test forward in time. In finance, never test on dates before your training data. Train on January to June, test on July. Otherwise the model quietly learns from the future.
Tune complexity on the test score: tree depth, number of features, and so on.

And there’s a third option, which the next posts build on: keep the complex, low-bias model, but reduce its variance by averaging many of them. If each deep tree zig-zags differently, the average of many will follow the real curve. That’s bagging.

The code

import numpy as np
from sklearn.tree import DecisionTreeRegressor
import market_data as md                 # my helper with the made-up curve (not shown)

grid = np.linspace(5, 45, 200)           # this month's volatility, where we test
truth = md.vol_curve(grid)               # known only because the curve is made up

preds = []
for _ in range(300):                     # 300 fresh training samples of 80 months
    x, y = md.vol_samples(80)
    tree = DecisionTreeRegressor(max_depth=15).fit(x.reshape(-1, 1), y)
    preds.append(tree.predict(grid.reshape(-1, 1)))
preds = np.array(preds)

bias2 = np.mean((preds.mean(axis=0) - truth) ** 2)   # average guess vs truth
variance = np.mean(preds.var(axis=0))                 # how much guesses jump around

Summary

Idea	Meaning	Remember it as
Bias	Average guess is off from the truth	Too simple
Variance	Guess changes a lot with the training sample	Too sensitive
Noise	Randomness in the data itself	Nobody can fix it
Decomposition	error = bias² + variance + noise	You can trade one for another
Overfitting	training great, new data poor	Too much variance
Fixes	simpler model, more data, or average many models	Bagging attacks variance

Bagging: train many models on slightly different data and average them, cutting variance without adding much bias.

References

James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), section 2.2.2: the bias-variance trade-off (equation 2.7)
scikit-learn: cross-validation