Nobody taught me how to backtest a trading strategy.
So I read the books (and wrote two), the documentation, the research, and the blogs.
Most of them end the same way: an equity curve going up and to the right.
You copy the code. It loses money. So you change the moving average from 50 days to 43. Now it looks great.
A folder full of backtests.
And you don't trust a single one.
That's not a coding problem. Nobody showed you how to check if a result is luck.
The research on backtests is brutal:
- Try 45 or more variations of a strategy on five years of daily data, and the best one will likely post a Sharpe ratio of 1.0 or better. (Bailey et al., 2014)
- Across 888 Quantopian algorithms, the backtest Sharpe ratio barely predicted out-of-sample results (R² below 0.025), and more backtesting made the gap bigger. (Wiecki et al., 2016)
And on and on and on...
The solution: The 3-check backtest:
The easiest way to tell if your backtest is real is by shuffling price data. Generate a bunch of random results and see if the real result is statistically better. Most the time it's not.
Here's how to do it.
Step 1: Run the backtest and record the Sharpe ratio
Fund managers report holdings at month end, so they dump losers and buy bonds.
The rule: long TLT for the last week of the month, short it for the first week, flat otherwise.
Simple.
Write the rule down first. Run it once on daily TLT returns since 2015 and record the Sharpe ratio.
Step 2: Try to beat the coin flip
I call this the shuffle score.
Literally shuffle the price data in random order, run the backtest, and record the Sharpe ratio.
Then do it again 1,000 times.
The idea is to determine the share of 1,000 shuffled copies of your strategy that the real one beats. Exact same strategy on random price data.
If you have any sort of edge, your real Sharpe beats at least 95% of them. If your real Sharpe lands somewhere in the middle, then your strategy is doing no better than randomly shuffled data.
Here ret holds daily TLT returns:
On ten years of pure random numbers, the rule posted a 0.35 Sharpe and beat 87% of its shuffles.
Noise looks like a strategy. That’s why the bar is 95%.