Machine learning for trading, part two, in ten ideas

The first series was about prediction: a table of examples, each with the right answer, and models from linear regression to boosting that learn to predict the answer for a new row. This second series is about decisions. A trading system does not only predict tomorrow’s return; it holds a position, and today’s choice changes what tomorrow’s choice costs. The first eight ideas build the learner that can handle that, reinforcement learning, and ask where it breaks in a market. The last two widen the toolkit to neural networks, models built for sequences, and language models, and to what it takes to trust any of them with money. The examples again use daily prices from 2019 to 2023 for SPY, AAPL, JPM, XOM and GLD. Read this page for the shape, then go to the post for the idea you need.

Two habits do most of the work this time:

The learner optimises the reward you wrote down, not the goal you had in mind. And a result that looks good in the training years, before costs, has not yet shown anything.

1. Reinforcement learning: learning from reward, not from labels

A supervised learner is handed the right answer for every example. A reinforcement learner is handed a situation (the state), makes a choice (the action), and afterwards receives one number (the reward). Nobody tells it what the right choice was. Its goal is the sum of the rewards over the long run, and what it learns is a policy: a rule from state to action. The same ten days of SPY can be either kind of problem. As rows of features with next-day returns as labels, they ask “what will tomorrow bring?”. As one episode of buy-then-hold, they ask “what should I do, given that today’s choice is tomorrow’s starting point?”. The second question is harder for three reasons: the agent’s own actions choose its training data, the reward for a choice arrives spread over the days that follow, and to find a better action it must sometimes try one it believes is worse. Full post: Reinforcement learning: learning from reward, not from labels.

2. MDPs: a state, an action, a reward and the value of what follows

A Markov decision process is the formal shape of the loop. Its one strong assumption is on the state: it must hold everything about the past that matters for the future, so that knowing the state, you need not know how you got there. The learner is judged by the return, the sum of the rewards still to come, with each later reward shrunk by a discount factor so that a reward far away counts for less than one tomorrow. The value of a state is the return you can expect from it under a policy; the value of an action is the same, for taking that action first. Bellman’s idea is that these values satisfy a recipe: the value of now equals the reward now plus the discounted value of next. The best policy is the one that, in every state, picks the action of highest value. Full post: MDPs: a state, an action, a reward and the value of what follows.

3. Q-learning: one update rule, repeated

A Q-table is a grid with one row per state and one column per action, holding the learner’s current estimate of each action’s value. Q-learning fills it from experience with one rule. After each step the learner looks at what it got (the reward) and what the best it can now hope for is (the largest entry in the next state’s row), combines them into a target, and moves the old estimate a fraction of the way towards it. That fraction is the learning rate: how much one new experience changes an estimate. The discount factor plays its separate role inside the target: how much the future counts. Because the target uses the best next action rather than the one actually taken, the table learns the values of acting greedily even while the learner is still exploring; this is what off-policy means. Entries that are never visited never improve, so convergence is a matter of coverage, not of running longer. Full post: Q-learning: one update rule, repeated.

4. Exploration: when to try something worse on purpose

A learner that always takes the action it currently rates best may never find out that another is better: its first few tries of the right action were disappointing and it never went back. Sutton and Barto’s ten-armed test shows the greedy method finding the best action in only about a third of the tasks. The simple fix is epsilon-greedy: with a small probability, pick an action at random; otherwise take the best known. Exploring more finds good actions sooner and then wastes a share of every step on them; exploring less learns slowly and then exploits better, so the usual compromise is to shrink the random share as experience grows. In a market, exploring costs real money, which is why the learning happens in a simulation and why the behaviour used in training is not the behaviour used live. Variability under a stable process (a stochastic environment) and a process that changes over time (a nonstationary one) are different problems, and the second is the one markets pose. Full post: Exploration: when to try something worse on purpose.

5. Dyna: learn a model of the world, then practise in it

Real experience is slow and, in trading, expensive. Dyna uses each real step twice: once to update the Q-table directly, and once to update a model of the environment, a record of which next state and reward followed each state and action. Between real steps the learner then draws remembered state–action pairs from the model and runs the same Q-update on them, many times, as if it were replaying experience. The gain is sample efficiency: more learning per real step. The cost is that the model can be wrong, and planning on a wrong model makes the learner more confident of a wrong answer, not less. In a deterministic world remembering the one outcome is enough; in a stochastic one the model has to hold the spread of outcomes or sample from it. The lesson reaches past Dyna to every backtest and every simulator: practice data is only as good as the process that made it. Full post: Dyna: learn a model of the world, then practise in it.

6. States and rewards: what you tell the learner decides what it learns

A Q-table needs a finite list of states, and market variables are continuous, so each indicator is cut into bins and the bin numbers are combined into one state label. Every choice there is a choice about the problem. Too few bins merge situations that need different actions; too many leave most states visited once or never. If two situations that call for different trades share a state, the learner cannot tell them apart, whatever it does next. Holdings belong in the state for that reason: the same signal means “buy” to an agent in cash and “do nothing” to one already long. Reward is the other dial. Today’s price change, today’s change in account value, realised gain and gain after costs all teach different habits, and a bonus or penalty added to speed learning up can teach the learner to collect the bonus instead of making money. Full post: States and rewards: what you tell the learner decides what it learns.

7. RL for trading: the market as an environment, and where that breaks

Turning the learner into a trading system means answering, in order: what the state is built from (only what was known that morning), what an action means (a target position, not a repeated order), how a held position carries from day to day, and what the reward charges for. Commissions and the price you move against yourself can turn a strategy that is profitable before costs into one that loses after them, and high turnover is usually a sign the reward or the action mapping is wrong rather than a sign of skill. A policy trained on one period is tested on a later one that it never saw, once, without going back to tune on the result. Even then a market changes in ways a maze does not: the rules that produced the training years need not hold in the next. Ritter’s paper gives a worked example of a reward that charges for risk as well as for costs, and shows a Q-learner finding a known edge in a simulated market with trading costs. Full post: RL for trading: the market as an environment, and where that breaks.

8. Comparing learners: a rule, a tree and a Q-table on the same data

A hand-written rule, a supervised learner that predicts the next return and trades on the prediction, and a reinforcement learner that outputs the position directly are solving three different problems with the same prices. The supervised one needs labels and a separate rule to turn a forecast into a trade; the reinforcement one needs a reward and a simulator, and learns the trade directly. Replaying a fixed price history does not make an environment interactive if the agent’s trades cannot change what it sees next. A fair comparison gives all three the same features at the same times, the same test period, the same starting capital, the same costs and the same benchmark, and reports the same measures for each. When they disagree, the reason is usually in the objective each was given, not in which algorithm is cleverer. Full post: Comparing learners: a rule, a tree and a Q-table on the same data.

9. Neural networks: linear regression, stacked and bent

A single-layer network takes the features, forms several weighted sums of them (one per hidden unit), passes each sum through a fixed nonlinear function such as ReLU, and fits a linear regression on those outputs. Without the bend, stacking linear layers would still be one linear model; the activation function is what lets the network represent relationships a straight line cannot. Training means choosing the weights that make a loss small: forward propagation computes the prediction, backpropagation computes how the loss changes with each weight, and gradient descent steps the weights downhill, usually on a small batch of examples at a time, with one pass through all the data called an epoch. More layers and more units mean more capacity and more ways to fit noise, so regularisation, a penalty on the weights, dropout, early stopping, is part of the recipe rather than an extra. The loss the network minimises is still prediction error, which is not the same as trading profit. Full post: Neural networks: linear regression, stacked and bent.

10. Sequence models and responsible AI: what changes when the data is a sequence, and who it affects

Market data has an order, so three families of model are built to respect it. A convolutional model slides the same small filter along the series and picks up short local patterns. A recurrent model carries a hidden state from one step to the next, so earlier inputs can influence later outputs. A transformer uses attention, which lets each position weigh every other position with weights that depend on the data, so it can relate distant points and compute in parallel, but it has to be told the order explicitly. Zeng and colleagues combine the first and third to model short- and long-range dependence in intraday prices. Language models extend the toolkit to text, and bring a failure mode numbers never had: fluent output with nothing behind it. A fabricated source or figure in a research note is a financial risk, so checking sources, keeping a person in the loop, being able to explain a decision and monitoring the system in use are part of the design, not a review added at the end. Full post: Sequence models and responsible AI: what changes when the data is a sequence, and who it affects.

References

• Sutton and Barto, Reinforcement Learning: An Introduction, 2nd edition (MIT Press, 2018). Chapter 1 for what reinforcement learning is and how it differs from supervised learning; Chapter 2 for the ten-armed testbed and epsilon-greedy action selection; Chapter 3 for Markov decision processes, the return, discounting, policies and value functions; section 6.5 for the Q-learning update; Chapter 8 for models, planning and Dyna-Q; the introduction to Part II for why large state spaces force generalisation.
• Tom M. Mitchell, Machine Learning (McGraw-Hill, 1997). Chapter 13 for the learning task, delayed reward and credit assignment, and Q-learning.
• Kevin P. Murphy, Probabilistic Machine Learning: Advanced Topics (MIT Press, 2023). Section 35.1 for the map of value-based, policy-search and model-based methods.
• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning with Applications in Python (Springer, 2023). Chapter 10 for single-layer networks, activation functions, fitting by gradient descent, epochs and minibatches, and ridge and dropout regularisation.
• Bishop and Bishop, Deep Learning: Foundations and Concepts (Springer, 2024). Chapter 12 for attention and transformers.
• Gordon Ritter, Machine Learning for Trading (2017). A Q-learner with a reward that charges for risk, in a simulated market with trading costs.
• Zeng, Kaur, Siddagangappa, Rahimi, Balch and Veloso, Financial Time Series Forecasting using CNN and Transformer (2023). Convolutions for short-range and attention for long-range dependence in intraday prices.
• CFA Institute Research Foundation, Handbook of Artificial Intelligence and Big Data Applications in Investments (2023). Chapter 1 for explainability and the black-box problem in investment models; Chapter 11 for human oversight in an automated research system.