Backtesting to Build Confidence

To make the trade or not to make the trade? That feeling of indecision before enacting a trading strategy can paralyze some traders, but fortunately, backtesting offers an excellent way to build confidence about a trade’s statistical chances. Knowing how the same strategy would have performed in historical market conditions can help you choose today whether to pull the trigger or step away.

Backtesting is exactly what it sounds like. Rather than trying to predict the future and guess how a trading strategy might perform in a given market, the idea behind backtesting is to replicate how running that strategy in a past timeframe would have resulted. It involves using real data of how a market previously performed, to determine whether the strategy would have been profitable in the past. For instance, a trader might look back through the past 30 years of market returns and determine whether or not it would have been profitable to buy silver each time its price pulled back 20%. If the strategy would have worked in the past, that might motivate the trader to confidently start running the strategy in hopes of future profits.

Technical traders are the most likely to backtest their strategies, for the obvious reason that they’re the most likely to use charts to predict future price movements. Value investors or fundamental traders are more likely to compare current market prices to a theoretical true value rather than worry about chart behavior. However, backtesting can be a valuable tool for anyone who might be entering an unfamiliar market and trying to gain some understanding and intuition about how that market tends to perform.

Getting down to the nuts and bolts, the actual method of backtesting can be as simple or as sophisticated as you choose to make it. Let’s say you’re thinking about building a trading program to go long in crude oil each time the continuous weekly futures chart posts a double bottom (a charting pattern used in technical analysis). Even if you just look at a historical chart and make a mental note of each double bottom and whether your program would trade profitably in the ensuing market conditions, that little act alone could probably be considered “backtesting.”

More likely, however, before putting your money on the line, you would like to have a more definitive result. Simple backtesting will more typically involve spreadsheets that list a market’s historical returns in various periods (monthly, weekly, daily, 10-minute, etc.) and as far back through time as you feel is relevant to your strategy. You will be able to able to run statistical calculations on the past data to determine the historical probability of a certain outcome under any circumstance you choose to study. But it will ultimately be up to each individual trader to determine what level of statistical confidence is necessary to ‘green light’ a strategy.

If you have the time and the computing power, a more rigorous, sophisticated level of backtesting can be done using a stochastic model with random variables to test how a proposed strategy might have performed in the past. A Monte Carlo simulation, for instance, could run that same crude oil return stream through multiple trial runs (as many as your computing power allows you to do efficiently) based on the assumed probabilities of outside market influences. The more trial runs your backtesting simulation goes through, and the more random factors with known influence you include in the simulation, the more confidence a trader would feel about knowing the likelihood of a proposed strategy’s success in the future.

Keep in mind that the results of your backtesting studies may not turn out to be a simple ‘yes’ or ‘no’ indication for the strategy. They may simply lead you to fine tune the strategy further, identifying the situations where it works profitably and flagging the conditions that put the strategy at risk. It may even just serve to give you more realistic expectations of what profit level should be targeted, or what risk level should be deemed acceptable.

The very, very important disclaimer about backtesting should be evident – it’s obviously a method that relies on the assumption that what happened in the past will happen again in the future. More sophisticated backtesting, like Monte Carlo simulations, may express what might happen in the future as a probabilistic range of outcomes, but they still calculate those future probabilities based on past statistical observations. Crucially, they often calculate future likelihoods by using the assumption of a normal distribution for each variable (an average expectation, with probabilities assigned to standard deviations from that expectation).

Experience has taught everyone in these markets that assuming anything about market behavior can be a dangerous game. If you assumed Credit Default Swaps insulated the market from a housing bubble burst, or if you assumed that holding theoretically (historically) non-correlated, diversified assets would shield your portfolio from overall loss in 2008, you learned not to fully trust historically-calculated statistical probabilities as the ultimate predictor of the future.

So bear that in mind when backtesting a strategy you’d like to build in to your trading program. No amount of history can accurately predict what will happen in the future. Confidence about how a strategy behaves in some previous market conditions doesn’t necessarily help you determine how that strategy will behave in untested future conditions. It certainly behooves you to select an appropriate time period for your backtesting, preferably one that is similar to how you expect market conditions to be when you begin the program.

You can probably get access to as many years of historical data as you like, but ask yourself if the crude oil market’s behavior in the 1980’s is a relevant stand-in for how crude oil futures are likely to behave in the months ahead of you. Are the grain markets’ return streams from the 1970’s appropriate data to use for backtesting a grain trading strategy today? Particularly in the case of stock markets and ETF’s, the trading world has changed dramatically in just the past few years. Because of the changing nature of markets, you can see how backtesting may be better applied to short-term, high-frequency trading programs than to long-term, buy-and-hold strategies.

Nevertheless, as a method of developing your analysis of an unfamiliar market, or just building confidence for a new trading strategy in a well-known market, pausing and running a rigorous backtest of a strategy is one of the most helpful things a trader can do for himself and his portfolio.

By Elaine Kub