Why most bots lose

The trap

Here is how almost every home-made trading bot is born. Someone has an idea, say a moving-average crossover. They test it on a few years of prices. The result is mediocre, so they try different settings: a 10-day average instead of 20, a 50-day instead of 200. Somewhere around the thirtieth try, a combination turns a profit that beats the market. It looks like a discovery.

It usually is not. Prices are noisy, and any fixed stretch of noisy history contains patterns that will never happen again: a dip that happened to recover, a rally that happened to stall exactly where a 15-day average crossed a 40-day one. Try enough rules and one of them will line up with that particular stretch of noise. The more you search, the better the best result looks and the less it means.

The name for this is overfitting: a strategy so closely fitted to the data it was built on that it describes the noise in that data rather than anything that repeats. The way to catch it is simple and humbling. Test the winner on data it has never seen.

Read the strip of gold dots. On the market it was chosen from, the winner ranked first of 82 by definition. On markets it had never seen, it scatters across the whole range and averages somewhere near the middle, which is exactly where a combination picked at random would land. The search did not find a good strategy. It found the luckiest one.

Now tick Sluggish markets and search again. These markets have a genuine pattern: news soaks in over several days, so short trends tend to continue. The winner now tends to rank somewhat better than the middle on new data, because part of what the search found was real. Only part, though: the in-sample result still overstates it. That is the honest best case for a backtest. Even when there is something there, the number you found is too high.

Multiple testing: the monkey problem

Overfitting has a cousin that needs no parameters at all. If you give 200 people a coin each and ask them to flip it ten times, someone will flip ten heads, or close to it. Nobody would hire that person as a professional coin flipper. But replace the coins with trading strategies, or fund managers, or newsletter writers, and the same arithmetic quietly produces stars.

Some monkeys beat buying and holding. The best one usually beats it handsomely, with a chart that would look great in an advertisement. Then, given a new market, it lands wherever the coin takes it. There was never anything behind its record but luck.

This is why a single impressive track record proves so little when you do not know how many others were tried and quietly dropped. Trading courses showcase their successful students. Bot sellers show the backtest that worked. Funds that had a bad decade close and disappear from the comparison tables. You are being shown the best monkey.

Other ways backtests lie

• Look-ahead bias. Using information the strategy could not have had at the time. The classic version: deciding to buy based on today's closing price and assuming you bought at that same close. Subtler versions use a company's earnings on the day of its fiscal quarter's end, when they were actually published weeks later. The bot lab avoids this by only trading after each day's close, at whatever price the book offers next.
• Survivorship bias. Testing on today's list of companies. A test of "buy stocks in the S&P 500" that uses today's members leaves out every company that was in the index, collapsed and got removed. Using today's members bakes in the knowledge of who survived.
• Ignoring costs. Spreads, slippage, commissions and taxes. A strategy that trades daily needs a large edge just to break even, as the bot lab showed.
• Regime change. Markets change. A pattern that held through a decade of falling interest rates may vanish when rates rise. Once a pattern is published, other traders pile in and arbitrage it away; academic studies of published stock-market anomalies have found their returns tend to shrink after publication.
• Too little data. Two years of daily prices sounds like a lot, but a moving-average strategy might trade only a dozen times in that span. A dozen trades is an anecdote.

How professionals try not to fool themselves

None of these problems can be removed completely, but serious quantitative researchers fight them with habits like these.

• Hold data back. Decide everything using one period, then test once on a later period you have not looked at. If you peek at the holdout and adjust, it stops being a holdout.
• Walk forward. Repeat the hold-out test many times as the window rolls through history: fit on a stretch, trade the next stretch, slide forward, repeat. The strategy is only judged on the gold periods below, all of which it saw for the first time.

Gold is where the strategy is judged; the green window before it is where its settings were chosen.

• Fewer knobs. A rule with two settings has less room to fit noise than one with twelve. If a strategy only works at exactly a 17-day average and falls apart at 15 or 19, it is fitted to noise. Look for broad plateaus in the heatmap, not sharp peaks.
• A reason. Prefer patterns with an economic story about who is on the other side and why they keep losing, such as investors who react slowly to news, or hedgers willing to pay for insurance. Patterns found by blind search rarely have one.
• Count the tries. Adjust expectations for how many variations were tested. Statistical tools such as the deflated Sharpe ratio shrink a result based on the number of trials behind it.
• Start small. Trade a new strategy with a little money, or on paper, and compare live results with the backtest. Live results are almost always worse; the question is by how much.

What happens to real traders

Studies that follow real people trading on their own tell a consistent story. Researchers with access to every trade on the Taiwan Stock Exchange found that fewer than 1% of day traders were able to earn predictable profits after fees, year after year. A study of people in Brazil who day traded index futures on more than 300 days found that 97% of them lost money. And the scorecards S&P publishes on professional fund managers, people with research teams and decades of data, find that most actively managed funds trail a simple index of their market over long periods.

None of this says patterns never exist or that nobody beats the market; some firms clearly do, with enormous investment in data, technology and people. It says that the default outcome of trying, for an individual with a laptop and a backtest, is losing money slowly to costs and occasionally quickly to leverage. The next chapter is about the alternative, which is far less exciting and works far better.