Probability: how often something happens

Probability sits under almost everything in machine learning and finance: judging whether a strategy’s result is luck, how a learner decides, how risk adds up. Kevin Murphy’s Probabilistic Machine Learning starts its maths with it in chapter 2. This post covers the five rules, each with a plain meaning, a market example, the formula, and the numbers worked out.

The examples come from the daily moves of SPY, the S&P 500 fund, over 2019 to 2023: 1,257 trading days of adjusted closes. A day is up if the close is above the day before, otherwise down. Every count below is from that one series, so you can check them with the code at the end.

What a probability is

A probability is a number from 0 (never happens) to 1 (always happens). The everyday way to read it: out of many tries, what share ends in this outcome?

Of the 1,257 days, 690 were up:

a probability we worked outan up daya down day, a big dropthe signal we condition on

$$\begin{aligned} P(\good{\text{up}}) &= \frac{\text{up days}}{\text{all days}} \\[10pt] &= \frac{690}{1{,}257} \\[10pt] &= 0.549 \\ &= 54.9\% \end{aligned}$$

Read $P(\good{\text{up}})$ as “the probability of an up day”. Slightly more than half: the market drifts up, but only just, one day at a time.

Rule 1: “not” (the complement)

Plain meaning: the chance something does not happen is everything else.

$$P(\text{not } A) = 1 - P(A)$$

This rule is most useful for “at least one” questions, which are hard to count directly.

Example: a trading week has five days. What is the chance that at least one of them is down?

Counting “at least one” means one down day, two, three… That’s messy. The opposite is simple: all five up. If days were independent, each up with probability 0.549:

$$P(\good{\text{five up days}}) = 0.549^{5} = 0.0498$$ $$P(\bad{\text{at least one down day}}) = 1 - 0.0498 = 0.9502$$

A $\key{95\%}$ chance that a week contains a down day. The record agrees: of 251 five-day blocks, 96.4% had one. Small risks add up fast when you repeat them, and so does a market that is “usually up”: that only means up on more days than not.

Rule 2: “or” (the addition rule)

Plain meaning: add the two chances, then take away the overlap, because you counted it twice.

$$P(A \text{ or } B) = P(A) + P(B) - P(A \text{ and } B)$$

Example: 45.1% of days are down. 8.4% of days are big: the market moved more than 2% either way. 4.7% are both, a big down day. What share of days would make a nervous investor check the news, because the day was down or big?

$$\begin{aligned} &P(\bad{\text{down}} \text{ or } \text{big}) \\ &\quad = 0.451 + 0.084 - 0.047 = \key{0.488} \end{aligned}$$

That’s 613 of the 1,257 days. Without subtracting the overlap, you would say 53.5% and count the 59 big down days twice.

If two things can’t happen together (like “up” and “down”), the overlap is zero and you simply add.

Rule 3: “given” (conditional probability)

Plain meaning: once you know something, you only look at the cases where it’s true.

$$P(A \mid B) = \frac{P(A \text{ and } B)}{P(B)}$$

Read the bar $\mid$ as “given”. $P(\good{\text{up}} \mid \risk{\text{signal}})$ means “the chance of an up day, among days when the signal was on”.

Example: a simple signal: at yesterday’s close, was the price above its own 20-day average? (Technical traders call this being above the moving average.) Here are the days split two ways, leaving out the first 19 days that have no average yet:

$$P(\good{\text{up}} \mid \risk{\text{signal on}}) = \frac{472}{835} = 56.5\%$$ $$P(\good{\text{up}} \mid \risk{\text{signal off}}) = \frac{204}{403} = 50.6\%$$

Overall, 54.9% of days are up. With the signal on, $\key{56.5\%}$; with it off, $\key{50.6\%}$. The signal carries a little information, about six points’ worth. Most technical signals look like this once you count.

Watch the direction. These are not the same question:

$$\begin{aligned} P(\good{\text{up}} \mid \risk{\text{signal}}) &= \frac{472}{835} \\[10pt] &= 56.5\% \end{aligned}$$ $$\begin{aligned} P(\risk{\text{signal}} \mid \good{\text{up}}) &= \frac{472}{676} \\[10pt] &= 69.8\% \end{aligned}$$

“The signal is on for 70% of up days” sounds impressive and means almost nothing: the signal is on for 67% of all days. “Days with the signal on are up 56.5% of the time” is the number that matters. People mix the two up all the time, in data and in strategy pitches.

Rule 4: “and” (the multiplication rule)

Plain meaning: the chance of both is the chance of the first, times the chance of the second once the first has happened. Bishop and Bishop’s Deep Learning: Foundations and Concepts calls this the product rule (equation 2.9).

$$P(A \text{ and } B) = P(B) \times P(A \mid B)$$

Check it on the table: $P(\risk{\text{signal}}) = 835/1{,}238 = 67.4\%$, and $P(\good{\text{up}} \mid \risk{\text{signal}}) = 56.5\%$, so

$$\begin{aligned} &P(\good{\text{up}} \text{ and } \risk{\text{signal}}) \\ &\quad = 0.674 \times 0.565 = 0.381 = 38.1\% \end{aligned}$$

which matches the table: 472 out of 1,238.

Independence is the special case where knowing one thing tells you nothing about the other. Then $P(A \mid B) = P(A)$, and the rule becomes a plain multiplication, but only for independent events:

$$P(A \text{ and } B) = P(A) \times P(B)$$

I used this in the “at least one down day” example. Is it fair? For the direction of the day, nearly: $P(\good{\text{up}} \mid \text{yesterday up}) = 54.1\%$ and $P(\good{\text{up}} \mid \text{yesterday down}) = 56.0\%$, both within a couple of points of the 54.9% base rate. Yesterday’s direction tells you almost nothing about today’s.

The size of moves is another story. A drop of more than 2% happens on 4.7% of days. Given that yesterday was such a drop, it happens on $\bad{8.5\%}$ of days, nearly twice as often. Big moves cluster (the technical name is volatility clustering), so multiplying “independent” probabilities for a run of bad days makes a crash look much less likely than it is.

Rule 5: Bayes’ rule (turning “given” around)

Plain meaning: Bayes’ rule turns “how likely is the evidence if it’s true” into the question you actually care about: “how likely is it true, now that I’ve seen the evidence”. It’s equation 2.10 in Bishop and Bishop:

$$P(A \mid B) = \frac{P(B \mid A) \times P(A)}{P(B)}$$

Example: a backtest that looks good. You test a trading idea on past prices and it comes out with a Sharpe ratio above 1. How likely is it that the idea has a real edge? Three assumptions, stated so you can argue with them:

• Of the ideas people try, $\bad{1\%}$ have a real edge: $P(\text{edge}) = 0.01$
• A real edge passes the backtest $\risk{80\%}$ of the time: $P(\risk{\text{pass}} \mid \text{edge}) = 0.80$
• An idea with no edge still passes $\risk{5\%}$ of the time, by luck or by being tuned to the past: $P(\risk{\text{pass}} \mid \text{no edge}) = 0.05$

Most people reason “80% versus 5%, so it’s probably real”.

First, how often does a backtest pass at all? Passes come from real ideas or from lucky ones (this step is called the law of total probability; Bishop and Bishop derive it from the sum rule, equations 2.8 and 2.11):

$$\begin{aligned} P(\risk{\text{pass}}) &= 0.80 \times 0.01 + 0.05 \times 0.99 \\ &= 0.0080 + 0.0495 \\ &= 0.0575 \end{aligned}$$

Then Bayes:

$$\begin{aligned} P(\text{edge} \mid \risk{\text{pass}}) &= \frac{0.80 \times 0.01}{0.0575} \\[10pt] &= \frac{0.0080}{0.0575} \approx 0.14 \end{aligned}$$

Only about $\key{14\%}$ of good-looking backtests are real. It’s easier to see with counts. Out of 10,000 ideas:

• 100 have an edge, and 80 of them pass.
• 9,900 don’t, and 5% of them pass anyway: 495.

The test is decent. The passes are still mostly false, because ideas without an edge outnumber real ones a hundred to one, and 5% of a huge number is still a big number. The cure is in the assumptions: lower the 5% (test on data the idea never saw) and raise the 1% (only test ideas with a reason to work).

Bishop and Bishop work through the same trap with a medical screening test (equations 2.20 to 2.22): a cancer that affects 1% of people, a test that catches 90% of cases and wrongly flags 3% of healthy people. A positive result means only about a $\key{23\%}$ chance of cancer.

This mistake has a name: base rate neglect. People look at how accurate the test is and forget how rare the thing is. It’s why one good backtest is never enough, and why the Sharpe ratio post keeps saying “never trust it on a short history”.

The code

The counts from the price file, the Bayes calculation, and a simulation of 5 million ideas that checks the answer by brute force:

import numpy as np
import pandas as pd

px = pd.read_csv("prices.csv", index_col="date", parse_dates=True)["SPY"]
r = px.pct_change().dropna()
up = r > 0
print(up.mean())                                   # 0.549  P(up)
print(1 - up.mean() ** 5)                          # 0.950  P(at least one down day in five)

signal = (px > px.rolling(20).mean()).shift(1)     # above its 20-day average at yesterday's close
both = pd.DataFrame({"up": up, "signal": signal}).dropna()
print(both.groupby("signal")["up"].mean())         # 0.506 off, 0.565 on: P(up | signal)

prior, hit_rate, false_alarm = 0.01, 0.80, 0.05
p_pass = hit_rate * prior + false_alarm * (1 - prior)
print(hit_rate * prior / p_pass)                   # 0.139  P(edge | pass)

rng = np.random.default_rng(0)
edge = rng.random(5_000_000) < prior
passed = np.where(edge, rng.random(5_000_000) < hit_rate, rng.random(5_000_000) < false_alarm)
print(edge[passed].mean())                         # 0.139, the same by counting

Summary

Next

In the next post in this series: expected value, the average result you get when you repeat a risky decision many times, from a casino bet to a day in the market.

References

• Christopher M. Bishop and Hugh Bishop, Deep Learning: Foundations and Concepts (Springer, 2024), chapter 2, section 2.1: sum and product rules (equations 2.8, 2.9), Bayes’ theorem (2.10, 2.11), the screening-test example (2.20 to 2.22)
• Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022), chapter 2