Algorithm Aversion — Why people don’t follow the model!

Algorithm Aversion — Why people don’t follow the model!

July 29, 2015 Continuous Content, Daily Academic Alpha
Print Friendly

There are many studies showing that models beat experts, including the meta-study “Clinical versus mechanical prediction: A meta-analysis” by Grove et al. (2000).

However, given this knowledge that models beat experts, forecasters still prefer to use the human (expert) prediction as opposed to using the model.

Why is this?

A recent paper by Dietvorst et al. (2014), titled “Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err” examines this phenomenon. Here is the abstract of the paper.

Research shows that evidence-based algorithms more accurately predict the future than do human forecasters. Yet when forecasters are deciding whether to use a human forecaster or a statistical algorithm, they often choose the human forecaster. This phenomenon, which we call algorithm aversion, is costly, and it is important to understand its causes. We show that people are especially averse to algorithmic forecasters after seeing them perform, even when they see them outperform a human forecaster. This is because people more quickly lose confidence in algorithmic than human forecasters after seeing them make the same mistake. In 5 studies, participants either saw an algorithm make forecasts, a human make forecasts, both, or neither. They then decided whether to tie their incentives to the future predictions of the algorithm or the human. Participants who saw the algorithm perform were less confident in it, and less likely to choose it over an inferior human forecaster. This was true even among those who saw the algorithm outperform the human.

Here is an interesting example (from the paper) to describe why this may occur:

Imagine that you are driving to work via your normal route. You run into traffic and you predict that a different route will be faster. You get to work 20 minutes later than usual, and you learn from a coworker that your decision to abandon your route was costly; the traffic was not as bad as it seemed. Many of us have made mistakes like this one, and most would shrug it off. Very few people would decide to never again trust their own judgment in such situations.

Now imagine the same scenario, but instead of you having wrongly decided to abandon your route, your traffic-sensitive GPS made the error. Upon learning that the GPS made a mistake, many of us would lose confidence in the machine, becoming reluctant to use it again in a similar situation. It seems that the errors that we tolerate in humans become less tolerable when machines make them.

We believe that this example highlights a general tendency for people to more quickly lose confidence in algorithmic than human forecasters after seeing them make the same mistake. We propose that this tendency plays an important role in algorithm aversion. If this is true, then algorithm aversion should (partially) hinge on people’s experience with the algorithm. Although people may be willing to trust an algorithm in the absence of experience with it, seeing it perform—and almost inevitably err—will cause them to abandon it in favor of a human judge. This may occur even when people see the algorithm outperform the human.

The paper goes on to show that as human confidence in the model increases, humans are more likely to use the model (even if they have viewed the model and have seen it fail). This is described in the figure below.

algorithm aversion
Source: “Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err” by Dietvorst et al. (2014)

However, note that even when people have “much more” confidence in the models, around 18% of people still use the human prediction!

Our Thoughts:

We are a firm that believes in evidence-based investing and understand that (in general) models beat experts. However, most people prefer the human option after seeing a model (inevitably) fail at some point in time.

How does this tie to investing?

If we are trying to beat the market through security selection, an investor has three options: use a model to pick stocks, use a human , or combine the two. Inevitably, the model will underperform at some point, since no strategy wins all the time (if a strategy never failed, everyone would invest and the edge would cease to exist). When a model underperforms for a certain time period, it does not mean that the model is inherently broken. In fact, the model could have simply failed over some time period, but the long-term statistical “strength” of the model remains intact. Steve, the human stock-picker, will also under-perform at some point; however, Steve can probably tell a better story over a beer as to why he missed the mark on last quarter’s earnings, that pesky SEC investigation, etc. And since drinking a beer with stock-picker Steve is a lot more fun than drinking a beer with an HP desktop, we will probably give Steve the benefit of the doubt.

Successful investors understand that models will fail at times; however, being able to stick with the model through thick and thin is a good strategy for long-term wealth creation. For the rest of us, there’s always stock-picker Steve. Cheers.

***

Please remember that past performance is not an indicator of future results. Please read our full disclaimer. The views and opinions expressed herein are those of the author and do not necessarily reflect the views of Alpha Architect, its affiliates or its employees. This material has been provided to you solely for information and educational purposes and does not constitute an offer or solicitation of an offer or any advice or recommendation to purchase any securities or other financial instruments and may not be construed as such. The factual information set forth herein has been obtained or derived from sources believed by the author and Alpha Architect to be reliable but it is not necessarily all-inclusive and is not guaranteed as to its accuracy and is not to be regarded as a representation or warranty, express or implied, as to the information’s accuracy or completeness, nor should the attached information serve as the basis of any investment decision. No part of this material may be reproduced in any form, or referred to in any other publication, without express written permission from Alpha Architect.

***

If you liked this post, don't forget to subscribe to our blog.




About the Author

Jack Vogel, Ph.D.

Jack Vogel, Ph.D., conducts research in empirical asset pricing and behavioral finance, and is a co-author of DIY FINANCIAL ADVISOR: A Simple Solution to Build and Protect Your Wealth. His dissertation investigates how behavioral biases affect the value anomaly. His academic background includes experience as an instructor and research assistant at Drexel University in both the Finance and Mathematics departments, as well as a Finance instructor at Villanova University. Dr. Vogel is currently a Managing Member of Alpha Architect, LLC, an SEC-Registered Investment Advisor, where he heads the research department and serves as the Chief Financial Officer. He has a PhD in Finance and a MS in Mathematics from Drexel University, and graduated summa cum laude with a BS in Mathematics and Education from The University of Scranton.


  • umair usman

    how do we know that the efficacy of the model has permanently changed?? ie how often must we update it??

  • Jack Vogel, PhD

    Great question, unfortunately the answer depends on the model. For example, a model of the growth rate of flip-phone sales would have needed to be adjusted after Apple launched the iPhone; whereas (I imagine this to be true) the sales growth of the local beer store is pretty consistent year to year. Depending on what one is attempting to model will determine how often the model needs to be assessed (and possibly changed).

  • Steve

    Hey, I’m not a stock picker! 😀

    I like the suggestion by Greenblatt in “the little book…” for those who cannot resist stock picking…to at least keep within the best stocks of the model. After all, if you’re only after (say) 10 stocks from a decile of 100 stocks (perhaps doing this a few times a year)…then it shouldn’t matter TOO much whether you pick them via darts or try a bit of stock picking.

    At least doing it that way, you might discover that you actually have alpha of your own! And if (more likely) not…no harm done (so the theory goes)

  • umair usman

    makes good sense .
    Fundamentals , fundamentals, fundamentals…the search never stops.
    Appreciate the reply

  • Jack Vogel, PhD

    Here are the results from Greenblatt’s test of model vs self-managed:

    http://socialize.morningstar.com/NewSocialize/blogs/joelgreenblatt/archive/2012/01/18/adding-your-two-cents-may-cost-a-lot-over-the-long-term.aspx

    May 1, 2009 – April 30, 2011:

    Self-managed = 59.4%
    S&P 500 = 62.7%
    Model = 84.1%

    Small sample size, but interesting results!

  • Steve

    Very interesting, and I think it shows that people often can’t, “follow the model”. But it doesn’t change what I said above about sticking with the model’s stocks and either randomly throwing darts or doing some stock picking – *as long as* the model rules are still stuck to. Re-reading the morning star article highlights to me that the problems were mainly to do with, “self-managed investors sold stocks without replacing them, held more cash, and/or stopped updating the strategy on a periodic basis after the markets and their portfolio declined for a period of time” – which is not, “following the model”.

    An investor with a stock picking inclination (perhaps they fancy themselves as capable of finding insights into the fundamentals or perhaps they fancy themselves as being able to spot trends well on price charts etc)….as long as they still stuck to the rules (e.g. hold a minimum 20 stocks, hold for one year and replace etc)….I maintain that the results should be no different than choosing 20 stocks at random from the top decile (which reminds me of the test you guys did – a monte carlo of 30 stocks picked from the value decile). And *if* that investor happens to have some hidden, innate Buffett like skills…who knows?

  • Jack Vogel, PhD

    I agree, picking 20 stocks from the model is not bad, but at some level, you still need to “follow the model”

  • Kumar V

    well, most models are based on past data and theories about why they work. unless the model attempts to ‘learn’ there will always be a good reason to question the model when the human suspects paradigmatic shifts (eg touchscreen iPhone vs flip phones)

  • Kumar V

    the other point is that models are built by humans. as much as we like to package models as ‘evidence’ based, there is a lot of art in massaging and interpreting the data, not to mention the creative aspect of coming up with theories for why something has been true in the past. therefore, as i see it, models serve to systematize human beliefs or biases! that is why IMHO it makes sense to always view models with some suspicion and override them when the risks of going with the model don’t seem to be worth it. (i agree with you in general that it depends on the model, but i haven’t seen much in terms of learning models out of the traditional finance academia)

  • Jack Vogel, PhD

    I agree, one should always question the model.

  • Michael Milburn

    One of my issues with “following the model” is that as I watch trades come and go, I am always questioning it – and coming up with reasons why the model should’ve been able to avoid that bad trade. (Ex: Currently Mylan is one of my bad mean reversion trades, and I’m wondering about a rule about not trading anything w/ hostile takeover bid underway… SUNE also a mean reversion trade that has quickly turned ugly, and it was a company I had a bad opinion of to start with, but I decided to follow the model anyhow. It would’ve been better if I didn’t already have views of the Solar Energy space.)

    So I guess I need a better rule about locking down a model and just trading it as is – vs. making adjustments on the fly. It’s difficult to resist watching trades unfold and wondering – if I’d just have filtered for X, Y, or Z maybe I could’ve avoided the bad trade. Most of my testing tells me “sorry, you just gotta eat the losses because trying to avoid them hurts the model in other ways”, but resisting the tweaking is difficult when your gut tells you “I don’t care what the model says, this stock sucks, don’t buy it.”

  • Michael Milburn

    …not to mention that I know how simple things are underneath the hood of the model, and that there’s really no knowledge at work. It’s just looking for patterns that in the past are associated with higher probability trades.

  • Fundumentals are an indicator but on top of that one needs to consider risk.

    Is a model just a reflection of the risk-reward ratio sought?

    Value, is a vague term when the risk-reward ratio is not clearly defined.

  • umair usman

    this is a good point as well.
    i dont think anyone ever says that Value is the end all, but people assume so thanks to the ‘buffet effect;

  • True.

    In the end, any signal–value, momentum, volatility, etc.–needs to serve as a proxy for expectation errors in the market. One “beats the market” by front-running expectations of the market, not necessarily by simply buying “value” stocks. Fundamentals, technicals, sentiment, and so forth, only work to the extent they aren’t already discounted into the current price. They don’t work just because they sound good or they are an interesting story.

    When it comes to value, specifically, we would argue that well-designed value signals work because they capture a systematic “overreaction to bad news” expectation error–investors, on average, throw the baby out with the bath water when it comes to the cheapest stocks in the market. These signals don’t work just because they are based on price/fundamental ratios, it just happens to be the case that price/fundamentals serve as a proxy for bad market expecations and we can use these signals to frontrun the market, on average.

    An recent example of this type of research is from Piotroski and So 2012 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1757025 or Jack’s dissertation.

  • Thank you for bringing up this topic. I have found that building a model that consistently beats the index (and most money managers) is the easy part. It’s convincing potential investors that the model actually works and is likely to keep working in the future is the tough part. I believe we must be doing something right if the Index Trouncing Portfolio is showing a gain of 29.72% versus 3.34% for the S&P500 index in the first 7 months of 2015. See http://us.timing.net/performance-itp/ for prior year results.

  • Steve

    Another one for the collection, highlighting my current thoughts about allowing for a *little* tweaking, but only *within* the model.

    http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2616787