I'm not suggesting anything is sleazy, I'm sure it's a great tool. I'm just trying to understand how it works and how it is applied. The sum total of my knowledge on this product is a brief fast paced webinar. I also understand the purpose of the webinar was to demo features, not be a training session on back testing methodology. So, I would like to understand how it is applied to real trading. I also acknowledge I have more background in other areas of investment and as a newbie I'm open to learning what does and does not apply to the world of options. So, a few options newbie questions:
1. A key principle of back-testing/fitting is the concept of out of sample testing: Pick a point in time, say 2010 and develop your rules/system using only the data that was available a that time (the sample data). Then, walk it forward from 2011 to present to see how it would have performed (the testing data). Not splitting the data in some fashion is almost a guarantee of fantastic results. So, does the software (or how it is generally applied) test in-sample or out-of-sample? In the webinar, I only saw in-sample which prompted the original post. (Systematic Trading by Robert Carver provides a great discussion of over fitting).
2. For a strategy to be considered robust, it generally should work across a markets and time frames. If I am developing a trend following system for futures and it only works for soybeans over the past nine months, I probably don't have much of a strategy. Unless, of course, there is a thesis to go with it: e.g. identification of something that changed in only the soybean market nine months ago. So, I would like to understand how looking at the behavior of an individual stock over a relatively short sample period translates to a robust enduring strategy. How do we know we are not being fooled be randomness?
3. In another post in this thread you stated a win rate as evidence the tool works. Great, but I would like to know more to really know the effectiveness of the tool. What is the win rate for the base strategies without application of the back tester? More importantly, how much did the back tester increase profit for winners and/or decrease the losses on the losers?
4. You also stated in another response that as a practical matter you can't look at "...even 20 occurrences". Why not? How can there be confidence in any back test with a tiny sample size. I would just like to understand this one.
Thanks