Henry Carstens
Introduction
If someone says, “I’m making money buying the SP futures on Friday’s close and selling on Monday’s close, you’ve got to get in on this!” How would you know whether that idea was a good or bad?
What about a new trading idea you have – is it good or bad and how do you tell?
The way you tell is by testing. Testing is nothing more than learning the life history of a trading idea and it’s done by turning trading ideas into trading rules and letting the computer test the rules and give you back the ideas life history. Here, for example, is a snippet of the life history of the Buy Friday Sell Monday idea –
Are you more comfortable with “You’ve got to get in on this,”after seeing this test than you were before seeing it?
Who This Introduction Is For
This introduction is for traders who have never tested their ideas against a large amount of historical data, it is about looking back over years to learn the life history of a trading idea.
Its scope is a gentle introduction to testing trading ideas, not an all-encompassing opus.
Where We Are Going
I am going to teach you how to test trading ideas. I’m going to teach you how to learn the life history of any trading idea, in any market and any time frame. When we’re done, you’ll know how to test your ideas against historical data (or partner with someone who does) and how to begin evaluating those tests.
Remember, just because you test a system with a computer, doesn’t mean you have to trade the system with a computer. Testing doesn’t affect how you trade, it affects what you trade.
Why Testing is Important
Testing is important because it helps a trader separate good trading ideas from bad ones in a scientific manner.
Testing is also important because the market changes. Trading ideas ebb and flow in effectiveness – it’s an ever changing cycle that some ideas handle better than others. Only by testing can we identify how well a trading idea has handled and is likely to continue handling the test of time.
Discretionary Traders, Mechanical Traders and Mechanical Trading Systems
Discretionary traders trade based on ‘feel’ for the market. Mechanical traders put rules, like our buy Friday sell Monday rule, into a computer and let the computer trade the rule.
In between these two extremes, there are discretionary traders with rules and mechanical traders who turn their computers on and off using discretion. The middle, I suspect, encompasses just about everyone.
A mechanical trading system isn’t any different than a discretionary one; it’s just that the trading rules have been written for a computer instead of a person.
A Simple Trading System
A mechanical trading/testing system is a set of rules that tell a computer when to buy and when to sell. Once the computer has the rules for a trading idea, it can learn the idea’s life history by simulating how it would have performed over time.
Here’s another example of some of the simple statistics mechanical testing can tell us about a trading idea – things like how much it wins, how often it wins and how much it has made over time (from our Friday-Monday rule again) –
What’s Next
We’re going to learn the process for testing a trading idea, the tools you’ll need, see the test results, and learn how to interpret them. To finish things up there are some ideas for future directions, links and references that will help you on your way and I’ll give you some sample computer code you can use to run your own tests.
The Process
The Goal
The goal of testing is to validate a trading idea. We are going to do that in one simple allencompassing test – a consistency test. If a trading idea passes the Consistency Test, we will finish the system by testing additional entry rules and stops. If the Consistency Test fails, we will put the failed idea aside and move on to testing the next trading idea.
One test, Pass or Fail, proceed or discard, that’s all there is to it. First, however, we’re going to need a testing platform and/or a testing partner.
Select a testing platform or testing partner
Selecting A Testing Platform
There are lots of testing platforms to choose from. Some are solely for testing and others are for both testing and trading. For the small trader or trading team without a lot of support resources, the key in choosing a testing platform is to choose a platform that pushes all the historical data to you so you never have to touch, much less think about, the data. It’s one less thing to think about, one less problem to solve when you could be inventing and testing the new ideas that will take you to the next level.
The code used to test the ideas presented here and the testing template given later were written for TradeStation for the above reason – I found that TradeStation gave me the ability to focus on testing trading ideas instead of maintaining the data I would be using to do the testing. [Neither I nor Vertical Solutions is associated with or receives compensation from TradeStation]
Selecting A Testing Partner
If the idea of programming rules into a computer isn’t your cup of tea, choose a testing partner instead. Dr. Brett Steenbarger, author of Enhancing Trader Performance and the TraderFeed weblog (see Links) has added a new section to his blog that connects discretionary traders with computer testers. This is a great way to get started testing, both sides learn a lot from each other (Testing your trading ideas)
Identify the Key Idea
Once a testing platform/partner has been identified, the first real step in the testing process occurs: identifying the Key Idea to be tested. Every trading system has a single Key Idea. For example, in our Friday-Monday system, the Key Idea is to buy on Friday’s close and exit on Monday’s close.
The Key Idea does not contain any stops or filters (such as only trade from January to June or exit with a stop ten points below Friday’s close). The Key Idea is the one rule that all the other rules depend on; it is the engine that powers performance.
Here is the way the code for our Key Idea might look:
if today is Friday then
buy on the close
if today is Monday then
sell on the close
Remember, there is one and only one Key Idea per trading system.
Run the Test and Interpret the Results
Once the Key Idea has been turned into a rule for the computer, it is time to run the test and learn the life history of the Key Idea.
The most important thing to look for in the test results is consistency over time. This is the Consistency Test we mentioned earlier. A system that has been consistent over time is far more likely to continue being successful than one that has a history of ups and downs.
Pass/Fail
We are going to say a Key Idea passes the Consistency Test if it has been consistent over time and we’re going to fail our Key Idea if it has been inconsistent.
The section ‘Where to Go From Here’ provides some ‘next steps’ for further evaluating the life history of a Key Idea.
Example: Consistency Test for the Friday Monday Key Idea
These are the test results for the Key Idea of Buy Friday’s close and exit on Monday’s close. Does the Key Idea pass the Consistency Test?
The Key Idea is pretty consistently profitable and it didn’t do too badly in the decline from 2000 to 2002. This is a pretty consistent winner and gets a passing grade which means we will continue to work with this system by testing stops and other entry rules.
If the Key Idea had failed the Consistency Test we would put it aside and move on to a new Key Idea.
Now we’re going to take a more in depth at look at testing that include testing the effect of stops on the Friday Monday Key Idea.
A More In-Depth Look at Testing
KISS
The key to obtaining valid test results is simplicity: Always work from the simple to the complex and never in the other direction. Identify and test the Key Idea first and build on Key Ideas whose history stands up to the Consistency Test.
Do not try to make a failed Key Idea work by adding stops and additional entry rules – if the Key Idea’s motor is broken, no amount of chrome is going to help it in the live
trading world.
You Need Lots of ‘n’
The more times your Key Idea has occurred in the past, the n (for number of times), the more reliable your forecast for the future will be, e.g., a history of 100 trades is more
predictive than a history of 10 trades, all things being equal.
Test the most general version of your Key Idea in order to see the longest and most complete history available.
How Far Back Should I Test?
As a general rule, there should be at least 100 occurrences of the Key Idea.
It is also good to test over several market regimes. Using the SP Index as an example, there have been 3 recent multi-day regimes, 1997-1999, 2000-2002, and 2003 to present. It is a good idea to test over all of them.
Testing stops and additional entry rules
Once the Key Idea has passed its initial Consistency Test, it is time to test stops and possibly additional entry rules. The secret to testing these adornments to the Key Idea is to test them one at a time and independently. Test each stop rule independently of the others and only keep the stop rule if it significantly affects the results. Test each entry rule independently of other entry rules and stops (In the testing template in the Appendix, there are separate sections and subsections for each of these). Only keep an entry rule if it significantly affects the results.
What is significant? A reduction in the number of trades but a smaller reduction in the overall profit (which tells you that you’ve removed more bad trades than good ones), a higher average trade or bigger win loss ratio. The more you test, the more you will gain a feel for what is significant.
One additional rule that might be used with our Friday-Monday Key Idea is a catastrophe stop loss. Let’s add one and see how it does; here are the results for the Friday Monday Key Idea plus a 1% stop loss –
Good, it still passes the Consistency Test and things do look a bit better.
Let’s look at some more of the history:
Bingo! Look how much better the Key Idea performs with the addition of a 1% stop loss. The average trade is now $255 vs $200 before the stop loss, a 25% improvement with a corresponding improvement in net profit. The win loss ratio is higher and the largest losing trade has been cut by about 40%. The catastrophe stop was a very good addition to the Key Idea.
This is another reason why testing is so important – not only can it help you tell good ideas from bad ones, it can help you make a good idea better. (See, however, The Pitfalls of Testing, below.)
Rule of Thumb
A good rule of thumb is that the number of entry rules and number of exit rules should be roughly equal.
If you find yourself with two entry rules and eight stop losses, it’s probably a good time to go back and look at the Key Idea again and test it for consistency.
Finishing up
It All Gets Easier With Time
Like any repetitive activity, testing gets easier over time (see Enhancing Trader Performance in the Books section, below). You now have the basic tools to test trading ideas and to make those ideas better. The more you test, the easier testing becomes and the more natural not just testing but developing trading ideas becomes. First, however, you need to know about pitfalls of testing.
The Four Pitfalls of Testing
The four pitfalls of testing are slippage, commissions, fills and curve-fitting. Each of these reduce the profits of your system in ways that aren’t always obvious. Testing gives an optimal view of the trading idea’s life history, we need to account for slippage, commissions, bad fills and curve-fitting to bring testing results into alignment with the real world.
Most testing platforms have an easy way to deduct slippage and commissions and account for bad fills. None, however can account for curve-fitting, but by using the Consistency Test on a single Key Idea as the basis for your idea development will go a long way to keeping the urge to apply a special rule to eliminate that one big losing trade.
Let’s take a look at each of the four pitfalls.
Slippage
Slippage occurs when you are filled at the ask instead of the bid. If this happens on entry and exit, the total slippage per trade is two ticks. Two ticks can easily wipe out 50% of the profits from a trading system.
Commissions
Commissions need to be deducted from the test results, too, again subtracting from the tested system results.
Fills
If you specify limit orders in your trading system, tests will always show your system as filled, even if only one contract traded at that price in the real world. Limit trading in
testing is always equal to or better than the results the mechanical system will get in the real world.
Curve-fitting
Curve-fitting occurs when rules are added to the Key Idea for the specific purpose of eliminating bad trades. Systems that have been ‘curve-fit’ never hold up in live trading.
Conclusion
What We’ve Covered
We’ve seen how learning the life history of a trading idea can give us a better idea about the soundness of that idea. We’ve learned how to identify the Key Idea for testing and seen how to generate and evaluate its life history. We used the Consistency Test as a simple Pass/Fail test for a Key idea: We kept and continued to work on Key Ideas that were consistently profitable while putting inconsistent Key Ideas aside. Once validated by the Consistency Test, we learned to independently test and add stops and entry filters to the Key Idea and we saw how all this worked by following the Friday Monday Key Idea from Consistency Test to testing it with a 1% stop loss. Finally we learned about the biggest pitfalls of testing.
Next Steps
The next section, ‘Where to go from here’ has some ideas on further tests you can run to delve even deeper into the life history of a Key Idea and the References and Links section provides some helpful links.
Where To Go From Here
Two new tests for looking at the life history of a Key Idea are presented here followed by Links, References and some sample code.
Additional Testing: t-Test and Optimal f
While a simple check for consistency across time goes a long way to validating a system, there are two more validating tests you can run, the t-Test and optimal f. These tests are both based on looking at another page of testing results like these –
These results show how many times the Key Idea traded, its percentage of wins, its win/loss ratio and its average trade, among other things.
The t-Test is a simple statistical test that tells you how likely these test results are to have occurred by chance alone. A t-Test of less than 1.6 favors chance, above 1.6 and you’re more likely to have found something real – a tradable Key Idea. The higher the score, given n over at least 20, the more likely you’ve found a tradable history.
The t-Test is calculated as
t = square root (n) * (average trade / standard deviation of trades)
In our Friday Monday example above (which has a standard deviation of 3368) the t-Test result is:
t = Square root(494) * 199.95 / 3368
t = 1.3
A t of 1.3 is almost significant, and coupled with the Consistency Test we ran earlier means there’s enough promise in the history of the Friday Monday Key Idea that we
should continue to work with the system by testing stops and/or additional entry rules.
The second test to run is called optimal f. Optimal f is the market’s line in the sand for your Key Idea. Optimal f is the maximum number of contracts you can trade given your account size. Trade any more contracts and your account becomes more and more likely to break under risk of ruin. You don’t have to trade the optimal f number of contracts, but you should never trade more than the optimal f number.
Optimal f is calculated as
f$ = (largest losing trade)/((((1+win/loss ratio)*percent wins) – 1)/win/loss ratio)
In our Friday Monday example the optimal f is:
f$ = $19,100/((((1+1.01) * .54) – 1)/1.01)
f$ = $238,750/contract
Meaning we should have $238,750(!) in our account for each big SP futures contract we trade with the unadorned Friday Monday system. Trading any more than that such as
trading a single contract in a $20,000 account or trading 5 contracts in a $250,000 account, puts the account at the risk of ruin.
t-Test and Optimal f for the Friday Monday Key Idea with a 1% Stop Loss
Let’s take a look at how the t-Test and optimal f change when we add the 1% stop loss rule to the Friday Monday Key Idea –
The standard deviation of the average trade in these results is 2950 (not shown above):
t = square root (494) * 255 / 2950
t = 1.9
The t-Test is much better than the earlier t-Test of 1.3. You will get a much better feel for these numbers after you work with them a bit. Here a 50% increase in the t-Test putting it safely into ‘These results not likely to have occurred by chance alone” territory means the catastrophe stop is a very good rule.
Now for optimal f$
f$ = 8425 / ((((1+1.2) * .52) – 1)/1.2
f$ = $70,208
Another good result! When f$ falls it means the system is getting stronger. (Note that ‘strong in trading is a relative term. There are no guarantees here and you should not trade at optimal f levels unless you are really understand both yourself and your trading system. For instance a losing trade of $8,000 on a $70,000 dollar account is over a 10% loss in one trade. Note too, that this trading system has lost 9 times in a row sometime in its history as shown above under Max Consecutive Losing Trades.)
The t-Test and optimal f are easy to calculate from test results and give you valuable information about the strength and tradability of your system.
You are now well on your way to moving from a simple tester of trading ideas to actively developing trading ideas and making them stronger and more robust.
References and Links
These links and books will help you explore the world of testing.
Testing Platforms
Links
Henry Carstens and Vertical Solutions
I am a professional trader and trading systems developer. My company, Vertical Solutions, tests and develops mechanical trading systems for hedge funds and
professional traders. My website contains a number of papers on trading system development and testing:
You can e-mail me at carstens@verticalsolutions.com
Questions and comments about Introduction To Testing Trading Ideas are requested and welcome:
Dr. Brett Steenbarger
Dr. Steenbarger is a professional trader and psychologist who writes about mixing discretion, time frames, testing and much more, in his trading blogs. His TraderFeed blog
contains a section where traders and testers can link up:
Daily Speculations
Dr. Niederhoffer is the most profound generator of trading ideas of our time – he is not to be missed. His blog is here,
www.dailyspeculations.com/wordpress/, his books are listed below.
Books
Appendix: Sample Code
The following code can be cut and pasted into TradeStation and used as a back-testing template. It is for intermediate to advanced users, comfortable with computer programming and well along in their testing proficiencies. It is not for everyone, but it is something anyone can grow into.
To use the template, look for the section called Key Idea and enter your Key Idea there.
Test additional entry rules by adding, one at a time and independently, rules to the Entry Rules section and stops by adding rules, again one at a time and independently, to the Exit Rules section.
Apply this system to a TradeStation chart where Data1 is intraday bars of the timeframe you want to test and Data2 is daily bars of the market you want to test.
Share and enjoy.
{ h Intro to Testing
Henry Carstens
Vertical Solutions
021007 hac v1.0 Initial
Testing template for Introduction to Testing
Share and enjoy but unsupported beyond this line : )
Data1: the intraday data, such as 10 minute SP. Use the intraday data
to place entries, exits and stops. Also use to test entering and exiting at
different times.
Data2: the daily data, such as SP. Refer to the daily data when you need
information like yesterday’s high, the lowest low of the last n days, etc.
}
Inputs:
eEntryTime(1310), {time to enter a trade, 1310 is end of day west coast time}
eExitTime(1310), {time to exit a trade, 1310 is end of day west coast time}
hold(1), {duration of the trade in days, 0 for intraday, 1 for one day, etc}
length(0), {generic testing parameter}
trigger(1); {0: for long and short, 1 for long only, -1 for short only }
Variables:
moy(0), {month}
todaysdow(0), {day of week: 1:Mon, 2: Tues, etc}
y(0); {year}
{Do any preprocessing for the system here}
if (BarStatus(1) = 2 and time = Sess1FirstBarTime) then
begin
{Some simple housekeeping, I like to keep these handy even if I’m not using them}
moy = Month(date);
y = Year(date);
todaysdow= DayOfWeek(date);
{Add other one-time processing here}
end;
{This section gets called every bar and is where the entries and exit rules go}
if (BarStatus(1) = 2) then
begin {Put any processing that needs to occur every bar here}
{Entry rules go here}
if (MarketPosition = 0 {not already in a trade}
{Key Idea goes here}
{Other Entry Rules – be sure to add and test one at a time and independently}
) then
begin
{Place the actual orders – use ‘trigger’ as an easy way to test long only and short only rules}
if (trigger >= 0) then
begin
Buy(“L Ent”) next bar on open;
end
else if (trigger <= 0) then
begin
Sell Short(“S Ent”) next bar on open;
end;
end;
{Exit rules go here}
if (CurrentContracts > 0) then
begin
{exit rules for longs}
if (MarketPosition = 1) then
begin
{unconditional stop rules go here}
{stop loss rules go here}
if (PositionProfit < 0) then
begin
end
{break even stop rules here}
else
begin
end;
end
{exit rules for shorts}
else if (MarketPosition = -1) then
begin
{unconditional stop rules go here}
{stop loss rules go here}
if (PositionProfit < 0) then
begin
end
{break even stop rules here}
else
begin
end;
end;
end;
end;
{This section gets called once at the end of the day}
if (BarStatus(1) = 2 and time = Sess1EndTime) then
begin
end;
For TradeStation users, this code is available in Easy Language format at
www.verticalsolutions.com/download/intro_to_testing/h_testing_templatev1.0.ELD