Choosing a model is a systems decision

The series has now built a linear model, nearest neighbours, a tree, a forest and a boosted model on the same problem. This post puts them in one table and adds the columns that accuracy leaves out, because those columns decide which model gets deployed.

the chosen model or its predictionerror, or a cost

Five models, one table

Next month’s volatility from this month’s prices, fitted on 2019 to 2021, scored on 2022 to 2023, timed on one laptop core:

Read it column by column.

Accuracy. The spread is one volatility point, and the two simplest models win. On this problem the signal is weak and nearly linear, so the models with the least room to fit noise did best. That is not a general law; on the stress-month classification in the forests post the ordering ran the other way. It is a reminder that the accuracy column has to be measured, not assumed from the model’s reputation.

Training time. The forest took 1.5 seconds, 750 times the linear model. That only matters against a budget: a 40-minute fit inside a four-hour overnight window is free; the same fit every time a feature changes is not.

Prediction time, and its tail. The linear model answers a question in 20 microseconds; the forest in 5.7 milliseconds, because every prediction walks 300 trees. Nearest neighbours is cheap to train and the only model whose prediction cost grows with the data, because it scans every stored row. The last column is the one a trading deadline cares about: not the average latency but the worst one in a hundred. A mean of 2 ms inside a 10 ms deadline with a 99th percentile of 40 ms means one order in a hundred misses, and the misses cluster in the busy moments.

What it keeps. Nearest neighbours keeps every training row forever, which is a memory cost and sometimes a privacy or licensing one. The others keep parameters: seven numbers for the linear model, 15 leaves for the tree, 27,193 leaves for the forest.

Interpretability. The linear model’s seven coefficients can be read aloud: more 20-day volatility means more next-month volatility, at 573 points per unit. The tree with 15 leaves can be drawn on a page. The forest cannot, and a model that cannot be explained cannot be audited, which in some settings makes it unusable whatever its accuracy. Partial-dependence curves and importance scores recover some of it, at a cost.

Extrapolation: the question a test set never asks

The test months all lie within the range of the training months. Ask each model about a month twice as volatile as any it has seen, with the recent-volatility features doubled from the wildest training row:

Every tree-based model and the neighbour model answer with roughly the most extreme value they have ever seen, because a tree’s leaf is a constant and a neighbour’s answer is an average of stored rows. The linear model follows its slope off the edge of the map. Neither is right; the point is that they fail differently, and that a model’s behaviour outside its training range is a design choice, not something a test score reveals. Finance meets the outside of the range on the worst day of the decade.

Stability

A linear model refitted on slightly different data gives nearly the same line. A single tree can flip its root split and everything below it; that instability is why the bagging post exists. A forest is stable again by averaging. Stability across random seeds is cheap to measure and worth reporting next to the score; it is not the same as robustness to a change of regime, which only forward testing shows.

The comparison table

Worked costs: a table of one million rows and twenty features. One nearest-neighbour prediction touches twenty million numbers; one linear prediction touches twenty; a depth-12 tree makes twelve comparisons; a 500-tree forest of them makes six thousand.

How to decide

1. Write down the constraints before the accuracy: latency budget and its tail, how often the model refits, whether a human must be able to explain a prediction, how much data there is per feature.
2. Measure the baseline, then the simplest model that fits the constraints. Linear first.
3. Move up only when the gain is bigger than the noise between seeds and worth its extra cost.
4. Check the behaviour outside the training range on purpose.
5. Prefer the model whose failures you understand.

There is no best model for every dataset. There is a best model for this dataset under these constraints, and the constraints are most of the decision.

Traps

The code

import time
import numpy as np

for name, model in models.items():
    t0 = time.perf_counter(); model.fit(X_train, y_train); fit = time.perf_counter() - t0
    singles = []
    for row in X_test[:200]:
        t0 = time.perf_counter(); model.predict(row[None, :]); singles.append(time.perf_counter() - t0)
    p50, p99 = np.percentile(singles, [50, 99])                   # the median and the tail

extreme = X_train[np.argmax(X_train[:, 4])].copy(); extreme[3:5] *= 2   # off the edge of the map
model.predict(extreme[None, :])

Summary

References

• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), section 2.1.3: the trade-off between flexibility and interpretability
• Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022), section 1.2.4: no free lunch
• scikit-learn: computational performance, prediction latency and throughput