Machine learning for quantitative finance in ten ideas

This is the map of the series. Each idea gets one paragraph, its key number, and a link to the post that works it through with formulas, worked numbers and scikit-learn code. The numbers come from the blog’s shared problem: predict next month’s volatility of a ticker from seven features computed from its own past prices, five tickers, 3,685 training rows from 2019 to 2021 and 2,405 test rows from 2022 to 2023. Errors are RMSE in volatility points unless said otherwise. At the end is the formula sheet for the whole series.

Every number a model sees must have been knowable at the time. Every score needs a split, a metric and a baseline. Everything else is a choice of model.

1. The target decides the problem

Every target is a learnable rule plus noise nobody can predict, $Y = f(X) + \varepsilon$ . Choose a number and it is regression; a label and it is classification; the same features serve both. The noise sets a floor no model gets under, and on next-month volatility that floor is high: predicting the training mean scores 9.84, and the best model only reaches about 7.5. Full post: What is machine learning?.

2. The data is part of the model

Which rows you keep, fill, order or reuse changes results as much as the algorithm does. Random folds on a time series score far better than forward folds (the model peeks at the future); a missing-value flag carries information of its own; a handful of outliers moves a fitted slope from 0.56 to 0.83. Full post: The data is part of the model.

3. Every feature has a timestamp

For every feature ask: when did this value become knowable, and could I compute it live? Leakage is a feature that secretly contains the answer, through time (it uses the future) or through pooling (a statistic computed over train and test together). A leaked feature scored 6.81 against an honest 7.50, and it would never have worked in production. Full post: Every feature has a timestamp.

4. A linear model is a weighted sum

A linear model predicts a weighted sum of the features, with weights chosen to minimise squared error. Read each weight as “one unit more of this, holding the rest fixed”; read its t-statistic as whether it is distinguishable from zero. Ridge and lasso shrink the weights, and a larger penalty is a simpler model. Full post: Linear regression.

5. Nearest neighbours store everything

Keep all the data; at query time find the $k$ closest rows and average them. Small $k$ is a complex model, large $k$ a simple one, the reverse of tree depth. It needs scaled features (9.01 raw against 8.44 scaled) and cannot extrapolate: the best $k$ here was 100. Full post: Nearest neighbours.

6. A tree asks one question at a time

A tree is a chain of yes/no questions, one feature and one threshold each, ending in a leaf that predicts a constant. It grows greedily, choosing each split by information gain (entropy or Gini) or by variance reduction, and never looks back. Fully grown it memorises: a perfect 1.000 on training months and 0.615 on test. Full post: Decision trees.

7. Averaging helps only when the errors differ

Bagging trains many copies of a model on bootstrap samples and averages them. Each copy sees about 63% of distinct rows; the rest are out-of-bag. Variance falls to a floor of ρσ², so decorrelating the copies (random forests, random splits) helps more than adding copies, and bagging a stable model like linear regression changes nothing. Full post: Bagging.

8. A score needs a split, a metric and a baseline

Training data fits the parameters, validation chooses the settings, the test set is touched once. A number means nothing until you know which split it came from, which metric it is, and what the baseline scores. For a rare class, accuracy lies and the threshold matters: the same model’s recall tripled when the cut moved from 0.5 to 0.2. Full post: Scoring a model.

9. The gap between seen and unseen is the whole game

Error on new data is bias² + variance + noise. Training error always falls as the model gets more flexible; validation error falls, bottoms out, then rises. Read any model from the two numbers: high and high is underfit, very low and much higher is overfit. More data fixes variance only. Full post: Bias and variance.

10. Model choice is a systems decision

Linear, nearest neighbours, trees and forests differ in training cost, cost per prediction, whether they keep the data, need scaling, extrapolate, and can be read. Only the linear model extrapolates: asked about a month twice as volatile as any seen, the trees answered about 62 and the line answered 151. Choose for the system the model lives in, not for the leaderboard. Full post: Choosing a model.

The formula sheet

Topic	Formula	Remember
Signal plus noise	$Y = f(X) + \varepsilon$	noise is the floor
Bias-variance	$\text{Bias}^2 + \text{Var} + \text{Noise}$	more data fixes variance only
MSE, RMSE, MAE	mean of $e^2$ ; its root; mean of $\lvert e \rvert$	RMSE ≥ MAE
$R^2$	$1 - \text{RSS}/\text{TSS}$	against predicting the mean
t-statistic	$\hat\beta / \text{SE}$	about 2 is the bar
Odds, logistic	$p/(1 - p)$ ; log-odds linear in $x$	one unit of $x$ multiplies odds by $e^\beta$
Gini	$1 - \sum p^2$	0.5 at worst for two classes
Entropy	$-\sum p \ln p$ (or $\log_2$ for bits)	ln 2 = 0.693, 1 bit
Information gain	parent entropy − weighted child entropy	split on the largest
KNN capacity	about $N/k$	bigger $k$ , simpler model
Curse of dimensionality	side $= q^{1/p}$	neighbourhoods explode
Bootstrap	$(1 - 1/N)^N \to 0.37$	63% in the bag
Averaged variance	$\rho\sigma^2 + (1 - \rho)\sigma^2/B$	decorrelate
Sensitivity, specificity	TP / actual positives; TN / actual negatives	divide by the true counts
Precision	TP / predicted positives	divide by the predicted count
VIF	$1/(1 - R_j^2)$	above 5 to 10, collinear
Bayes	$P(A \mid B) = P(B \mid A)P(A)/P(B)$	use counts of 10,000; a rare prior drags it down

The arithmetic to be fluent in

MAE, MSE and RMSE from a list of misses.
Gini and entropy from class fractions; information gain in bits.
The bootstrap 63/37 split and the $\rho\sigma^2$ floor.
The reversed dials: small $k$ is complex, big λ is simple.
Bayes with counts: 1% base rate, 90% sensitivity, 3% false alarms gives 90/(90 + 297) = 23%.
Sensitivity, specificity and precision from a confusion matrix.
Read a train-test pair: 0.5 and 2.0 is overfit; 1.9 and 2.0 is underfit.