Predicting Mental Defects: Does the Model or the Expert Win?

Predicting Mental Defects: Does the Model or the Expert Win?

June 14, 2014 Behavioral Finance
Print Friendly
(Last Updated On: January 18, 2017)

Clinical intuition and test scores as a basis for diagnosis

  • Klehr, R.
  • Journal of Consulting Psychology, 13, 34-38
  • An online version of the paper can be found here
  • Want a summary of academic papers with alpha? Check out our free Academic Alpha Database!

Abstract:

The present study has compared the ability of a group of 15 experienced clinicians to diagnose mental defectives and schizophrenics on the basis of purely qualitative interpretation of test responses, with the diagnostic results of applying an objective scoring technique to the same material. The material evaluated consisted of the CVS test responses of 36 subjects: 12 schizophrenics, 12 mental defectives, and 12 normals. The results indicate that the performance of the clinicians under these circumstances is comparable to the use of an objective scoring technique, both performances being significantly better than chance. A control group of graduate students did not differ significantly from chance.

Prediction:

In this experiment 36 subjects were chosen. The subjects were broken into 3 categories:

  1. Schizophrenics
  2. Mental defectives
  3. Normal

15 advanced level psychologists who had long experience in clinical testing were selected to evaluate these subjects’ “normality.” As a control, 15 inexperienced graduate students were also asked to evaluate the 36 subjects. There are two data sources:

  1. Results from a standardized intelligence test shown to predict mental defects (CVS)
  2. Experience and intuition.

The results for the experiment are in Table I. On the left axis we have the actual diagnosis and then we have the performance of the different groups (clinical experts, grad students, and the systematic approach (test)).

How does this work?

Take line 1. As a benchmark, any group should get 4/12 correct identifications because the 3 categories are spread evenly among each group of 12. Of the 12 Schizophrenics, experts (labeled Clin.) called 6.6/12, the system got 6/12. Experts said .7/12 Schiz. were defectives; 4.7/12 were normals. The system said 1/12 were defectives; 5/12 were normal. The students were no better than chance. For calling mental defectives, the experts got 8.8/12 and the machine got 10/12. For normals, experts hit 8.8/12, the machine called 7/12. Klehr_1949.pdf - Adobe Reader_2014-06-09_09-50-01 This study highlights a few aspects of models versus experts:

  1. Experienced experts and the model have similar performance.
  2. Inexperienced experts have poor performance.

Thoughts?


Note: This site provides NO information on our value investing ETFs or our momentum investing ETFs. Please refer to this site.


Join thousands of other readers and subscribe to our blog.


Please remember that past performance is not an indicator of future results. Please read our full disclosures. The views and opinions expressed herein are those of the author and do not necessarily reflect the views of Alpha Architect, its affiliates or its employees. This material has been provided to you solely for information and educational purposes and does not constitute an offer or solicitation of an offer or any advice or recommendation to purchase any securities or other financial instruments and may not be construed as such. The factual information set forth herein has been obtained or derived from sources believed by the author and Alpha Architect to be reliable but it is not necessarily all-inclusive and is not guaranteed as to its accuracy and is not to be regarded as a representation or warranty, express or implied, as to the information’s accuracy or completeness, nor should the attached information serve as the basis of any investment decision. No part of this material may be reproduced in any form, or referred to in any other publication, without express written permission from Alpha Architect.


Definitions of common statistics used in our analysis are available here (towards the bottom)




About the Author

Wesley R. Gray, Ph.D.

After serving as a Captain in the United States Marine Corps, Dr. Gray earned a PhD, and worked as a finance professor at Drexel University. Dr. Gray’s interest in bridging the research gap between academia and industry led him to found Alpha Architect, an asset management that delivers affordable active exposures for tax-sensitive investors. Dr. Gray has published four books and a number of academic articles. Wes is a regular contributor to multiple industry outlets, to include the following: Wall Street Journal, Forbes, ETF.com, and the CFA Institute. Dr. Gray earned an MBA and a PhD in finance from the University of Chicago and graduated magna cum laude with a BS from The Wharton School of the University of Pennsylvania.


  • karl shewmake

    As an expert in medical diagnosis, and known as the doc who will figure it out, I do not think this study applies to the financial world. Among other arguments, the specificity rule would apply. the financial world has too many variables and is so different than the medical.

  • Hi Karl,
    Thanks for the input. As an actual expert in the field, your comments related to these studies are very helpful.

    In the context of financial markets, it seems that simple models are likely to BEAT experts (e.g., a low-p/e large-cap stock strategy over the past 30 years would place you among a very select group of mutual fund managers that were able to beat the S&P 500).

    At least the doctors in this study gave the model a decent fight!