Backtesting Question

The following was first sent as a private message:

[QUOTE]I have a question that I hope you can give me some guidance on. If I'm backtesting on 1, 30, and 60min price bars how far back should I go in time if I only have 40 occurences (i.e., data points) to test on 60min bars over the span of an entire year, 86 data points on 30 min price bars and 400 on 1 min bars?[/QUOTE]

I don't worry as much about how big "n" is (the number of occurrences), so much as capturing many different market environments. I didn't really know how to describe this or quantify it until Billy posted Steenbarger's article on stationarity [URL="http://www.effectivevolume.com/showthread.php?4208-The-Robot-and-Stationarity-The-Best-Kept-Secret-in-Trading-Success"]here [/URL].

The short answer on how far back to go is: as long as possible. There are, of course, constraints that may make it unfeasible or inadvisable to test all available historical data. For instance, the presence of the uptick rule distorts the 20 day MF, so any backtesting Pascal does in this epoch is not applicable to the current one where we don't have the rule.

Your computer might be able to backtest 10 years of 60 minute SPY data comfortably, but not 10 years of 1 minute data. In that case, you might survey a longer term chart and choose a sampling of three to four month windows at various times of the year over several market cycles. This is possible to do in TradeStation, but may not be on other platforms, and even at that, it is a bit cumbersome. Unfortunately, most platforms are not geared toward proper backtesting procedures.

Another thing to consider, since you're using intraday data, is that stationarity increases substantially the shorter the holding time, as factors such as proximity to the open/close/news factor heavily, among many other things. It's difficult to create a robust system with a meaningful edge on a 1 minute chart without taking these complex factors into consideration.

Anyone else, please feel free to chime in.

[QUOTE=EB;19944]In that case, you might survey a longer term chart and choose a sampling of three to four month windows at various times of the year over several market cycles.[/QUOTE]

Let me preface my comments by saying that I am a sophmore compared to the others here but I believe that this is one aspect too often overlooked.

Ideally, when backtesting, one should split their data sets into a training set and a test set. The training set is used to calculate and set the parameters of your model and the test set is used to verify. You can split the data set in half or even thirds, where you rotate the training set between the partitions.

Obviously, you want to achieve simlar returns between the training and test sets or your model is biased to one set - hence why you rotate. Finally, the above tests are only applicable when the data sets are stationary as EB eludes above. Good luck.

Good point Harry. TradeStation now incorporates a Walk-Forward Optimizer (formerly, The Grail), which does in/out sampling. Some preliminary tests I've done with it have revealed that systems I thought were fairly robust actually are not.

[QUOTE=EB;19959]Good point Harry. TradeStation now incorporates a Walk-Forward Optimizer (formerly, The Grail), which does in/out sampling. Some preliminary tests I've done with it have revealed that systems I thought were fairly robust actually are not.[/QUOTE]

EB, can you expand on it a little more? Why systems that you thought were good, appeared not to?

Thanks.

Bob,

Many people say the market displays fractal qualities, i.e., self-similarity across different timeframes. Would you agree and as a corollary, how much confidence can one have in the proposition that if a trading system works on one timeframe, e.g., 30 or 60 minute, it will work on daily, weekly, etc?

[QUOTE=Andrei;19963]EB, can you expand on it a little more? Why systems that you thought were good, appeared not to?

Thanks.[/QUOTE]

It might be misleading to give specific information. Just some intraday trend following systems with time filters that I had played with years ago.

[QUOTE=adam ali;19969]Bob,

Many people say the market displays fractal qualities, i.e., self-similarity across different timeframes. Would you agree and as a corollary, how much confidence can one have in the proposition that if a trading system works on one timeframe, e.g., 30 or 60 minute, it will work on daily, weekly, etc?[/QUOTE]

Interesting question, which gave me pause to reconsider my belief in the fractal nature of markets. I would have to say I retain the belief, but the realities of specific time factors present intraday (open, close, news, etc.) easily disrupts the evolution of the fractal such that trading systems on intraday timeframes need to be adjusted and will not necessarily work even though they may work on higher time frames.

My personal belief is that ofcourse higher timeframes action may repeat shorter timeframes, but that repetition is not cyclycal i.e. doesn't repeat itself automatically over certain timespans or timespan multiply factor. Anyways you would need some technology to catch those similarities :)