Why prices are hard to beat: efficiency, random walks and a roulette wheel

Three claims hide inside “markets are efficient”, and they are not the same claim. This post takes them apart, tests the simplest one on five years of SPY, and then uses a roulette wheel to show the second way a strategy can look like it works when it does not.

a measured pattern or a chosen rulea gaina loss

Three forms of efficiency

Efficiency is about what information the price already contains. The three forms, from the weakest claim to the strongest:

Efficient does not mean prices are always right, or that nobody ever profits. It means no reliable edge after costs. Measure it by how fast prices react to news: big, liquid, heavily watched markets react within seconds and are the most efficient; small and obscure ones react slowly. The evidence for the semi-strong form is that very few fund managers beat their index persistently, fewer than chance alone would produce. The people who try hardest to beat the market are the ones who make it efficient.

The random walk, tested on SPY

If past prices are useless, tomorrow’s price is today’s plus a surprise nobody can predict, and the best forecast of tomorrow’s return is zero. That is a random walk. Two things follow: yesterday’s return should tell you nothing about today’s, and streaks of up or down days should be no longer than a coin would produce.

Two of the checks pass: the streaks are exactly what shuffling the same days produces, and the up-after-up rate is close to the up-after-down rate. The first does not. A correlation of −0.167 at one day is a real pattern: a down day leans towards an up day next. Most of it comes from the violent weeks of March 2020, when the market reversed day after day, but it is there.

So build the rule. Hold SPY the day after a down day, sit in cash the day after an up day. With no trading costs it returned $\good{+84.4\%}$ over the five years, against +106.2% for buy and hold and +14.5% for the opposite rule. Now charge for the 633 position changes it makes, as in the post on trading costs:

That is what “efficient” means in practice. A pattern can be statistically real and still worth nothing, because the edge is smaller than the cost of trading it. The pattern survives in the data precisely because it is too small to be worth removing.

Two patterns are tradeable in principle: momentum (what went up keeps going, from news that spreads slowly and from big orders worked in pieces) and mean reversion (what went too far comes back). Competition shortens both. A stop-loss suits momentum and hurts mean reversion, because it sells right before the bounce.

A pair trade and its z-score

The usual way to trade mean reversion is a pair: long one stock, short a related one, betting that the gap between them closes. Measure the gap as a z-score, how many standard deviations the price ratio sits from its recent average:

For JPM against XOM over the five years, the 60-day z-score ran from −4.6 to +3.4 and sat beyond ±2 on 178 days. On 94% of those days the ratio was closer to its mean twenty days later. That sounds like a machine, and two things undercut it. First, “closer to the mean” is nearly guaranteed by the way the z-score is built: the mean and standard deviation move too. Second, the relationship itself drifts: the 60-day correlation between the two stocks ran from −0.21 to +0.84. A z-score measures how unusual a gap is; it is not a promise that the gap closes, and the pair can simply stop being a pair.

Arbitrage in the strict sense means identical payoffs bought and sold at the same moment: the same stock at $50.00 on one exchange and $50.40 on another, costs $0.10, is a riskless $0.30. A pair with correlation 0.8 is not that. Four limits keep real gaps from closing on schedule: costs and carry, capital and margin calls while the gap widens first, crowding (everyone exits together), and borrow cost or recall on the short leg.

The roulette wheel: a high hit rate on a losing game

The second way a strategy fools you has nothing to do with costs. American roulette has 38 pockets, 18 red. A bet on red pays even money, so each spin is a bet with $p = 18/38 = 0.4737$ and expected value $\bad{-5.26}$ cents per dollar. The doubling rule: bet $1, double after every loss, back to $1 after a win, stop at +$40.

Each win recovers all previous losses plus $1, so the only way to lose is a run of losses the bankroll cannot cover. With no limit on the bankroll every one of 2,000 simulated plays reached +$40, median 84 spins. The expected value per spin is still negative; it hides in the size of the bet needed to climb out of a run.

Give the player a bankroll and sweep it:

Check one row by hand: $0.791 \times 40 - 0.208 \times 256 = 31.6 - 53.2 = -21.6$. A bigger bankroll raises the hit rate from 49% to 97% and makes the median play a winner, and the mean stays negative the whole way, because each doubling of the bankroll makes the rare loss twice as big while only halving its probability. The expected value of the game never changed. A high hit rate with a negative mean is a strategy that wins small and loses big; it is the shape of selling insurance, of selling options, and of the pair trade above when the pair breaks. The median hides the tail; the mean shows it; and only a measure of the tail, such as drawdown or the Sortino ratio, tells you the two apart.

Traps

The code

import numpy as np
import pandas as pd

prices = pd.read_csv("prices.csv", index_col="date", parse_dates=True)
r = prices.pct_change().dropna()
spy = r["SPY"]

spy.autocorr(1)                                  # -0.167: yesterday leans against today
up = spy > 0
up[up.shift(1) == False].mean()                  # 0.560: P(up | yesterday down)

rule = (~up).astype(float).shift(1).fillna(0)    # hold the day after a down day
(1 + rule * spy).prod() - 1                      # +84.4% before costs
trades = rule.diff().abs().fillna(0)             # 633 position changes
(1 + rule * spy - trades * 0.005).prod() - 1     # about -95% at 0.5% a change

ratio = np.log(prices["JPM"] / prices["XOM"])
z = (ratio - ratio.rolling(60).mean()) / ratio.rolling(60).std()   # the pair's z-score

def play(rng, bankroll=None, p=18/38, target=40):   # the doubling rule, one play
    bet, total = 1, 0
    for _ in range(2000):
        if total >= target or (bankroll is not None and bankroll + total <= 0):
            break
        if bankroll is not None:
            bet = min(bet, bankroll + total)
        if rng.random() < p: total += bet; bet = 1
        else:                total -= bet; bet *= 2
    return total

Summary

References

• Eugene F. Fama, “Efficient Capital Markets: A Review of Theory and Empirical Work”, The Journal of Finance 25(2), 383–417, 1970, the three forms
• Philip J. Romero and Tucker Balch, What Hedge Funds Really Do: An Introduction to Portfolio Management (Business Expert Press, 2014), chapters 3 and 8: arbitrage in the broad sense, and efficiency as reaction speed
• Ernest P. Chan, Quantitative Trading: How to Build Your Own Algorithmic Trading Business (2nd ed., Wiley, 2021), chapter 7: mean reversion, momentum, and cointegration against correlation