Alternative Beta can be Great: But Beware of Data-Mining!

Alternative Beta can be Great: But Beware of Data-Mining!

May 9, 2016 Behavioral Finance, Research Insights

Last updated on January 18th, 2017 at 03:32 pm

Print Friendly

Quantifying Backtest Overfitting in Alternative Beta Strategies

  • Antti Suhonen, Matthias Lennkh, and Fabrice Perez
  • A version of the paper can be found here.
  • Want a summary of academic papers with alpha? Check out our Academic Research Recap Category.

Abstract:

We investigate the biases in the backtested performance of “alternative beta” strategies using a sample of 215 commercially promoted trading strategies across five asset classes. Our results lend support to the cautions in recent literature regarding backtest overfitting and lack of robustness in trading strategy performance during the ”live” period (out of sample). We report a median 73% deterioration in Sharpe ratios between backtested and live performance periods for the strategies in our sample. We establish a link between performance deterioration and strategy complexity, with the realized reduction in live vs. backtested Sharpe ratios of the most complex strategies exceeding those of the simplest ones by over 30 percentage points. The robustness of strategy exposure to risk factors varies between asset classes and strategies, and appears reasonable in equity volatility and FX carry strategies, but quite weak in the equity value strategy in particular.

Alpha Highlight:

This paper investigates the backtested and subsequent live performance of alternative beta funds. To be clear, the researchers don’t study “smart beta” products, which in general are long-only investment products, often in equities, that use different weighting schemes relative to a passive index. By contrast, “alternative beta” strategies focus on risk premiums across multiple asset classes, might go long and short, and have low correlations with traditional investments.

In theory, alternative beta sounds great, but the proof is in the pudding: Do the theoretical factor exposures really provide the hoped-for returns once they go live? Are these robust, sustainable strategies? Or are we just “factor fishing?” The authors examine 215 alternative beta strategies across 5 asset classes from 15 investment banks.

The authors describe their sample selection process below:

  1. Investment bank-sponsored alternative beta strategies are formulaic and non-discretionary strategies that follow set investment rules (asset selection, rebalancing, execution, risk management etc.) that typically cannot be changed by the index sponsor after the live date.

  2. Upon the release of a new alternative beta strategy, the sponsoring investment bank will provide investors with a backtested time series illustrating the hypothetical past performance of the proposed strategy. Given the immutability of the strategy rules, there is some justification in arguing that this past performance would have some value in assessing the future behavior of the strategy.

  3. Our sample contains the daily backtested (pre-live date) performance of each strategy, the daily live (post-live date) performance, as well as the exact live date as reported by the strategy sponsor. The average length of the backtest periods is 10.7 years and that of the live periods is 4.6 years, allowing reasonable statistical analysis.

This is an interesting sample to analyze. Providers create a strategy, show the backtest (which often look good), and then launch a product to armies of advisors. Rinse and repeat. We have written about this activity within the context of ETFs and here is a paper on mutual funds. This paper adds to the literature by investigating alternative beta strategies.

So what happens after the fund launches?

Similar to the other research in ETFs and Mutual Funds, the data shows a significant dropoff in performance:

We find a median Sharpe ratio (SR) of 1.20 across the 215 alternative beta strategies during their respective backtest periods, compared to 0.31 during live performance.

Here is a visualization of Figure 1 from the paper that makes this even more clear:

Figure 1 below illustrates our key findings. The figure shows the backtested and live performance (excess return index) of one alternative beta strategy in the sample over a ten-year period, with the vertical line in October 2010 marking the live date of the strategy. The strategy is representative of the median in our sample, with a backtested SR of 1.42 and live SR of 0.36, or a realized SR haircut of 75%.

alternative beta overfitting
The results are hypothetical results and are NOT an indicator of future results and do NOT represent returns that any investor actually attained. Indexes are unmanaged, do not reflect management or trading fees, and one cannot invest directly in an index. Additional information regarding the construction of these results is available upon request.

Additionally, the authors make some recommendations for business practices around alternative beta. For instance, they point out that when examining strategy returns on Bloomberg, it is often difficult to identify when a backtest ends and live trading begins. Also, they suggest that banks supply hypothetical returns under different combinations of strategy parameters. This can provide insight into a strategy’s robustness.

Conclusion:

Overall, the paper highlights an important lesson for all investors — before investing in the new alternative product with the great backtest (because they always do), ask whether or not the strategy has sustainable edge/alpha.

 


Note: This site provides no information on our value investing ETFs or our momentum investing ETFs. Please refer to this site.


Join thousands of other readers and subscribe to our blog.


Please remember that past performance is not an indicator of future results. Please read our full disclaimer. The views and opinions expressed herein are those of the author and do not necessarily reflect the views of Alpha Architect, its affiliates or its employees. This material has been provided to you solely for information and educational purposes and does not constitute an offer or solicitation of an offer or any advice or recommendation to purchase any securities or other financial instruments and may not be construed as such. The factual information set forth herein has been obtained or derived from sources believed by the author and Alpha Architect to be reliable but it is not necessarily all-inclusive and is not guaranteed as to its accuracy and is not to be regarded as a representation or warranty, express or implied, as to the information’s accuracy or completeness, nor should the attached information serve as the basis of any investment decision. No part of this material may be reproduced in any form, or referred to in any other publication, without express written permission from Alpha Architect.




About the Author

Jack Vogel, Ph.D.

Jack Vogel, Ph.D., conducts research in empirical asset pricing and behavioral finance, and is a co-author of DIY FINANCIAL ADVISOR: A Simple Solution to Build and Protect Your Wealth. His dissertation investigates how behavioral biases affect the value anomaly. His academic background includes experience as an instructor and research assistant at Drexel University in both the Finance and Mathematics departments, as well as a Finance instructor at Villanova University. Dr. Vogel is currently a Managing Member of Alpha Architect, LLC, an SEC-Registered Investment Advisor, where he heads the research department and serves as the Chief Financial Officer. He has a PhD in Finance and a MS in Mathematics from Drexel University, and graduated summa cum laude with a BS in Mathematics and Education from The University of Scranton.