So far: Cost, Depth and Impact are three numbers you can read off any market's book, and the same three compare venues, coins and stock perpetuals (lesson 9).
Every real number in this guide was worked out from records of trades and quotes. A backtest does the same for a strategy: it replays the records and adds up what the strategy would have made. If the records hold something you couldn't have known at the time, or miss something that happened, the backtest is wrong. Some of those faults make a strategy look better than it would have done.
A backtest on bad data
Here is a toy strategy on six toy coins over four months. Every midnight it buys the two coins that rose most the day before, and sells them at the next midnight. If one falls 8% below what it paid, it sells that one at once, with a stop.
The grey line is its money on point-in-time data. Put a fault into the data and run it again: the orange line is what the backtest reports.
Tick one fault at a time and run the backtest. Compare the orange line, what the faulty data reports, with the grey one.
- coin 1
- coin 2
- coin 3
- coin 4
- coin 5
- coin 6delisted
Not run yet. Tick the faults to put into the data, then run the backtest. With none ticked, both runs see the same data.
On point-in-time data the strategy loses 11% over the four months. Each fault on its own turns that loss into a gain, and with all three the backtest reports a gain of 53%. The strategy is the same in every run. Only what the backtest knew has changed.
Three ways data goes wrong
A close that is an hour late gives the signal the first hour of the day it trades. The coins that rose most “yesterday” are partly the coins rising right now, and the backtest buys them at the midnight price, from before that hour. It even sees coin 6 open 45% down after its delisting news, in time to skip it. This is look-ahead bias, and any input stamped earlier than you could have had it does the same.
A recorder that falls behind when trading is busiest drops trades, and the busiest minutes are often the sharpest moves. In the toy, five of them set off the stop on a coin the strategy held, and each time the coin ended the day above its stop. Without those minutes in the data, the backtest never sells, and books the recovery. In real life the stop sold, 8% down.
A list of coins made today holds only the coins that survived. Coin 6 jumped, then collapsed when its delisting was announced in the third month, and it is not on today's list. So the backtest never bought it the night before the crash, and never took the loss. This is survivorship bias.
This lesson is a short version of a course of its own. Building market data, which comes later, is about how market data is built and kept right.
What you learned
- Point-in-time
- Every record holds only what could be known at its timestamp, and nothing is added, dropped or changed later.
- Look-ahead bias
- An input stamped earlier than you could have had it lets a backtest trade on the future.
- Missing data
- Gaps fall where trading is busiest, and they can hide the very moves that hurt.
- Survivorship bias
- A list of markets made today leaves out the ones that failed, and the losses they caused.
Check yourself
1. Your signal reads each day's close, but the close was taken an hour after the opening price you buy at. What is wrong with the backtest?
2. You backtest on the coins an exchange lists today. Which coins are missing?
3. In the figure, which fault took away the days the stop sold?