The data is part of the model: splits, gaps and outliers

Before any model is fitted, four decisions have been made about the data, usually without noticing. This post makes each one on purpose and measures what it does, on the series’ problem of predicting next month’s volatility from this month’s prices.

a chosen way of scoring or fillingerrorthe outlier

The split decides what the score means

The same random forest, scored three ways:

How it was scored	RMSE	What the number means
Fit on 2019 to 2021, score on 2022 to 2023	9.80	one honest number for one period
Five shuffled folds, averaged	9.93	each fold tests on months whose neighbours were in training
Five forward folds, averaged	$\risk{13.53}$	each fold trains on the past and tests on what came next
Predict the training mean	9.84	the baseline

The forward folds look worst, and they are the ones telling the truth. Their five scores were 29.66, 11.25, 9.30, 9.98 and 7.45. The first fold trained on 2019 alone and was then asked about 2020, the crash year, and missed by thirty points. The single time split never saw that, because its training set already contained 2020. The shuffled folds never saw it either, because every fold had crash months on both sides. One averaged number hides a regime the model had never met.

For a time series, the test must be later than the training. Consecutive days share most of a month, so a random split puts near-twins of every test month into the training set and the model gets credit for recognising them. Walk-forward folds (train on everything before a point, test on the block after it, slide the point) are the version of cross-validation that respects time; the second panel of the chart draws the five used here. The scikit-learn post showed the same effect on a classifier: shuffled folds gave an AUC of 0.912 and forward folds 0.734.

A second rule follows. Every step that learns from the data is part of training: the scaler, the imputer, the feature selection, the choice of settings. Fit them inside the training part of each fold, never on the whole table.

Gaps: why a value is missing matters more than that it is

Three kinds of missing, with three different right answers:

Kind	Plain meaning	Market example	Delete the rows?
Missing completely at random	pure accident	an API glitch	yes; only the sample shrinks
Missing at random	explained by other columns	small companies report less often	with care, conditioning on those columns
Missing because of the value	the gap is caused by what the value would have been	no volume because nothing traded; no earnings because bankrupt	no; the gap is the signal

Real run. I removed the 20-day volatility from 11.5% of the training rows, with the removal more likely when the volatility was high, the way a halted or illiquid stock goes missing in real data. Then three repairs:

Repair	Test RMSE
Drop the rows with a gap	10.67
Fill the gap with the column’s mean	10.31
Fill with the mean and add a “was missing” column	$\key{10.25}$
Nothing missing, for reference	9.80

Dropping was worst, because it threw away the most volatile months, exactly the ones the model needed to learn from. Mean-filling put a calm number into a wild month. The flag let the model learn that “missing” itself means “probably volatile”. For prices the standing rule is different and simpler: fill forward, so a halt repeats the last known price, and fill backward only where nothing came before.

Outliers: the loss function decides how far they pull

A fit minimises a loss, and the loss decides how much one far-out point matters. Three residuals of 1, 1 and 10:

Loss	Total	The outlier’s share
Squared, used by ordinary least squares	1 + 1 + 100 = 102	98%
Absolute	1 + 1 + 10 = 12	83%

Squaring is why one big miss dominates. Real run. Forty points on a line of slope 0.5, then one point moved 25 units off the line:

Fit	Slope, clean	Slope, with the outlier
Squared loss (ordinary least squares)	0.50	$\bad{0.83}$
Absolute loss (median regression)	0.51	0.56
Huber loss (squared when small, absolute when large)	0.51	0.53

One point in forty moved the least-squares slope by two thirds. The third panel of the chart draws it. A crash day in a return series does the same to any model fitted by squared error, which is most of them. The choices are to use a robust loss, to use a model where only the side of a split matters (a tree), or to keep the squared loss and know what it is doing. Deleting the point is not on the list: a crash is real data, and a model that has never seen one will meet one live.

The rules

Split by time; test later than training; walk forward for cross-validation.
Fit scalers, imputers and feature selection inside the training part of each fold.
Ask why each value is missing before deciding what to do about it; keep a missingness flag.
Know your loss; squared error chases outliers.
Never delete a crash.

Traps

Trap	Truth
A bigger sample fixes a biased sample	Bias is systematic, not random
Mean-filling is neutral	It shrinks the spread and hides the signal
Trees suffer most from outliers	Least squares does; a tree only sees which side
Normalising with the full table’s mean is harmless	It leaks the test set into training
Walk-forward is unfair because it reuses the past	The test is always later; that is the point
Shuffled folds are fine if the model is simple	They leak regardless of the model

The code

from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import HuberRegressor, LinearRegression, QuantileRegressor
from sklearn.model_selection import KFold, TimeSeriesSplit, cross_val_score

rf = RandomForestRegressor(300, min_samples_leaf=5, random_state=0)
cross_val_score(rf, X, y, cv=KFold(5, shuffle=True), scoring="neg_root_mean_squared_error")   # leaks
cross_val_score(rf, X, y, cv=TimeSeriesSplit(5), scoring="neg_root_mean_squared_error")      # honest

import numpy as np
missing = np.isnan(X[:, 4])
X[missing, 4] = np.nanmean(X[:, 4])                      # fill with the training mean
X = np.hstack([X, missing[:, None].astype(float)])       # and keep the fact that it was missing

LinearRegression().fit(x, y)                             # squared loss
QuantileRegressor(quantile=0.5, alpha=0).fit(x, y)       # absolute loss: the median line
HuberRegressor().fit(x, y)                               # squared when close, absolute when far

Summary

Idea	Rule	Remember it as
Time split	test later than training	no near-twins across the line
Walk-forward	train before, test after, slide	five numbers, not one
Preprocessing	fit inside the training fold	the scaler is a model too
Missing values	ask why; keep a flag	the gap can be the signal
Loss and outliers	squared error chases them	98% of the loss from one point

Regression and classification: the two questions most models answer, and how the question decides the model and the metric.

References

James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), sections 3.3.3 and 5.1: outliers and high-leverage points, cross-validation
scikit-learn: cross-validation of time series data, and common pitfalls
Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022), section 1.5.5: missing data