When Can You “Trust” a Backtest?

Although they may not be selling a strategy or investing concept to investors, they do have incentives to get their research published in academic journals that their peers may read and respect. There is also the financial incentive of job security by earning tenure at their university, for it’s“Publish or Perish” as the phrase goes. This means few, if any, are immune from the incentives to create an attractive looking backtest.

Does this mean we should dismiss all backtests? Certainly not, it just means we need a process, or series of scientific tests, that we run all our backtests through in order to keep ourselves out of trouble and from falling prey to good stories told by good salesmen. I must give credit where it’s due here. Both Dimensional Fund Advisors and the books and writings of Larry Swedroe have influenced my thinking when it comes to this topic.  For those who want to go deeper, I recommend reading Larry Swedroe’s book, “Your Complete Guide to Factor-Based Investing.”

The 5 characteristics to look and test for when considering investments are:

1. Persistent across time. The strategy or factor can be tested on long periods of historical data to increase statistical confidence. Larry Swedroe often points out that the average investor thinks three years is a long time, five years is a really long time, and 10 years is an eternity...yet if you ask academics, they will tell you that 10 years is nothing more than random noise that likely should be ignored. For example, the S&P 500 returned -1% per year from 2000-2009. Would that have been a good indication of long term expected returns for buying large cap US stocks? Since then, the S&P 500 has compounded at more than 12% per year, which is better than its long-term average. This is similar to how in August of 1979 BusinessWeek wrote a cover story called “The Death of Equities” after the S&P 500 had experienced a similarly long period of poor performance. The S&P 500 went on to compound at more than 17.5% annualized for the next two decades, turning $100,000 into more than $2.6 million.
    
2. Pervasive across markets and geographies. The strategy or factor holds up when tested on other markets and countries. For example, the momentum effect has been found to exist in stocks, bonds, commodities and currencies. It’s also pervasive across sectors and in the historical data of nearly every country.
    
3. Robust to various definitions. An effect should still show up when constructed with similar parameters. For example, the value effect is both persistent and pervasive as well as robust to alternative specifications. Whether it's price to book, price to sales, price to earnings, price to dividends, price to just about anything...you find in the historical data that value stocks (low price relative to fundamental measurements) outperform growth stocks (high price relative to fundamental measurements) over the long term all across the world.
    
4. Investable. The strategy exists not just on paper but survives real world issues such as manager fees and realistic assumptions for transaction costs. Many anomalies discovered in the historical data persist simply because they are difficult to implement at size in the real world. Academics refer to this as "limits to arbitrage,” where an anomaly persists in the data because it's difficult or impossible to actually implement at scale and therefore it’s really only a paper illusion.
    
5. Intuitive. Does the strategy make intuitive sense with (preferably) a simple risk-based explanation or,at minimum, a logical behavioral based explanation? For example, the market factor (stocks producing higher returns than T-bills) has persisted for decades even though it's as "well known" as any factor there is. Since everybody knows about it, why doesn't it get arbitraged away? One rational answer is that you simply cannot arbitrage away risk. Not only do stocks occasionally underperform cash over 3, 5, and 10-year periods, they can do so by A LOT (S&P 500 example in #1). The same is true of all other academically accepted factors like size, value and momentum.  For this reason, investors who have a sufficient time horizon and temperament can consider tilting their portfolio towards these "open secrets" that have a long history of producing higher than market returns.

Now, which factors pass all five of these tests and therefore are eligible for inclusion in our client portfolios?

• The market factor (stocks have higher expected returns than cash and bonds). Risk based explanation.
• The term factor (bonds with longer maturities have higher expected returns than bonds with shorter maturities). Risk based explanation.
• The size factor (small caps have higher expected returns than large caps). Risk based explanation.
• The value factor (value stocks have higher expected returns than growth stocks). Both risk and behavioral based explanations.
• The trend/momentum factor (stocks and asset classes that have outperformed over the last 6-12 months have near term higher expected returns than stocks and asset classes that have underperformed). Behavioral based explanation.
• The volatility risk premium factor (also known as the insurance risk premium…selling financial insurance in the form of puts and calls has positive expected returns). Risk based explanation.

Conclusion

When investors understand the concepts discussed in this article, your investing life will never be the same. Using this checklist dramatically improves the odds of success and can keep investors from falling prey to a good sales pitch or chasing a good looking backtest that is unlikely to persist going forward in time on a net of all costs basis (transaction costs, manager fees, and taxes).
 

The real opportunity now becomes the portfolio construction process. We know that the pursuit of traditional active management is largely a waste of time and money, because any manager or backtest performance can be largely explained by exposure to these well-known factors that can now be accessed at low or at least fair costs. These factors discussed have low correlations to one another, so diversifying broadly across them dramatically reduces the risk of your total portfolio. So much so that modest amounts of leverage may be appropriate for those with the patience and perspective to seek higher expected returns.

The biggest risk of a portfolio diversified by factors becomes more behavioral, where your portfolio will at times perform far differently than conventional market benchmarks that are only exposed to the single factor of market beta. Since the only purpose of investing should be to achieve your long-term goals with the least risk, this should be an acceptable trade-off for a well-educated and well-behaved investor.

Jesse Blom is a licensed investment advisor and Vice President of Lorintine Capital, LP. He provides investment advice to clients all over the United States and around the world. Jesse has been in financial services since 2008 and is a CERTIFIED FINANCIAL PLANNER™ professional. Working with a CFP® professional represents the highest standard of financial planning advice. Jesse has a Bachelor of Science in Finance from Oral Roberts University. Jesse manages the Steady Momentum service, and regularly incorporates options into client portfolios.