Bagging: many copies of a model, averaged

Bagging is covered in section 8.2 of An Introduction to Statistical Learning. The method comes from Leo Breiman’s 1996 paper “Bagging Predictors”. The name is short for bootstrap aggregating.

The bias and variance post ended with a problem: a deep tree has almost no bias but huge variance. Its answer depends heavily on which data it happened to see. Bagging fixes that by averaging many trees.

The idea: ask a committee

If you ask one risk analyst how volatile next month will be, you get their answer, quirks and all. Ask twenty analysts who each learned from a slightly different few years of history, and take the average: their individual quirks partly cancel out, and what’s left is closer to the real pattern.

Bagging does this with models:

Make many bootstrap samples of the training data.
Train one model (usually a deep decision tree) on each.
Average their predictions. For classification, average their probabilities, or take a vote.

Step 1: the bootstrap sample

A bootstrap sample is drawn from the training data with replacement, and it has the same size as the original. Some rows get picked more than once, some not at all.

Eight months, numbered 1 to 8, and three bootstrap samples of them:

Sample	Rows drawn	Left out
1	1, 3, 7, 7, 7, 7, 8, 8	2, 4, 5, 6
2	1, 1, 2, 5, 5, 7, 7, 8	3, 4, 6
3	2, 3, 4, 7, 7, 7, 8, 8	1, 5, 6

Each tree sees a different version of the history, so each tree comes out different.

How many rows does a bootstrap sample leave out? The chance that a given row is not picked in one draw is $1 - 1/n$ . Over $n$ draws:

\left(1 - \frac{1}{n}\right)^n \;\longrightarrow\; \frac{1}{e} \approx 0.368 \quad \text{as } n \text{ grows}

So each tree never sees about 37% of the data. With 10,000 rows, I measured 36.7%. Those left-out rows are called out-of-bag rows, and they’re useful: each tree can be tested on the rows it never saw, which gives a free estimate of performance without a separate test set.

Step 3: why averaging reduces variance

An Introduction to Statistical Learning (section 8.2.1) starts from a simple fact: averaging $n$ independent numbers, each with variance $\risk{\sigma^2}$ , gives an average with variance $\risk{\sigma^2} / n$ . Bagged trees aren’t independent, though: they learn from overlapping data. If you average $B$ predictions that each have variance $\risk{\sigma^2}$ , and every pair has correlation $\key{\rho}$ , the average has variance (The Elements of Statistical Learning, equation 15.1):

ρ, how alike the trees' errors areσ², one tree's variance

\risk{\text{Var}}(\text{average}) = \key{\rho}\,\risk{\sigma^2} + \frac{1 - \key{\rho}}{B}\,\risk{\sigma^2}

Symbol	Meaning
$B$	number of models averaged
$\risk{\sigma^2}$	variance of one model’s prediction
$\key{\rho}$	correlation between two models’ predictions

Plain meaning: the second term shrinks as you add models. The first term doesn’t. If the models were completely independent ( $\key{\rho} = 0$ ), averaging 100 of them would cut variance 100-fold. But trees trained on overlapping data make similar mistakes ( $\key{\rho} > 0$ ), so variance falls towards a floor of $\key{\rho}\,\risk{\sigma^2}$ and stops. (It’s the same maths as the diversification floor in a portfolio of correlated assets.)

Measuring it

Same made-up problem as the bias and variance post: predict next month’s volatility from this month’s, on a curve I invented so the truth is known. Same method: 200 fresh training samples of 80 months, measuring how much the predictions move between samples.

Trees averaged	Bias²	Variance	Expected error on new months
1 (one deep tree)	0.04	8.88	17.93
2	0.03	6.69	15.72
5	0.02	5.23	14.26
10	0.03	4.71	13.74
25	0.03	4.44	13.47
100	0.02	4.20	13.22

Bias stays near zero. Averaging deep trees doesn’t make them too simple.
Variance halves, from 8.9 to 4.2, most of it by the first 10 trees.
Then it levels off. That’s the $\key{\rho}\,\risk{\sigma^2}$ floor. From these numbers, the trees’ predictions are correlated at roughly $\key{\rho} \approx 0.47$ (my estimate: $4.2 / 8.9$ , ignoring the small leftover term).

On this small, simple problem, the best single tree from the bias and variance post (depth 3, error 11.8) still beats 100 bagged deep trees (13.2). Bagging removes variance. It doesn’t remove the need to choose a sensible model.

On the stress-month data

Now the classification problem from the decision trees post: will next month be a stress month? Scores are AUC on the 2,405 test ticker-months of 2022 and 2023 (0.5 is a coin flip, 1.0 is perfect):

Base tree	One tree	Bag of 200	Change
Fully grown	0.615	0.788	big gain
Depth 8	0.758	0.817	good gain
Depth 5	0.797	0.823	small gain
Depth 3	0.804	0.828	small gain

This table says most of what there is to say about bagging:

It helps a lot when the base model is unstable (high variance), like a fully grown tree.
It helps less when the model is already stable. A depth-3 tree changes little between bootstrap samples, so there’s less noise to average away; it still gains two points here, because real market data is noisier than my toy curve.

And a warning about the out-of-bag estimate. From the training data alone, the bag of fully grown trees reported an AUC of $\key{0.907}$ . Its real test score was 0.788. The estimate is far too rosy, and the reason is the data, not the method: consecutive days share most of the same month, so a row left out of a bootstrap sample has near-twins inside it, and “never seen” isn’t really true. On a time series, the only honest test is forward in time. The out-of-bag score is free and useful when rows are independent. Here it would have fooled me.

The code

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

bag = BaggingClassifier(
    DecisionTreeClassifier(max_depth=8),   # the model to copy
    n_estimators=200,                       # how many copies
    oob_score=True,                         # test each tree on the rows it never saw
    random_state=0,
)
bag.fit(X_train, y_train)
bag.predict_proba(X_test)[:, 1]             # averaged probability of a stress month
bag.oob_score_                              # out-of-bag accuracy: optimistic on a time series

Summary

Idea	Formula / rule	Remember it as
Bootstrap	sample $n$ rows with replacement	each tree sees a different history
Left out	$(1 - 1/n)^n \to 1/e \approx 37\%$	free test data: out-of-bag (if rows are independent)
Aggregate	average (regression), average or vote (classification)	the committee answer
Variance of an average	$\key{\rho}\risk{\sigma^2} + (1-\key{\rho})\risk{\sigma^2}/B$	more models help, until correlation stops them
When it helps	unstable, high-variance models	deep trees: yes; shallow trees: barely

Random forests: bagging, plus one extra trick that makes the trees less correlated, so the $\key{\rho}\,\risk{\sigma^2}$ floor drops and averaging works even better.

References

James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), section 8.2.1
Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022), chapter 18: bagging
Hastie, Tibshirani and Friedman, The Elements of Statistical Learning (2nd ed., Springer, 2009), chapter 15: the variance of an average of correlated trees (equation 15.1)
Leo Breiman, “Bagging Predictors”, Machine Learning 24(2), 123–140, 1996
scikit-learn: bagging meta-estimator