Models vs. Experts #7: Job Analysis
A comparison of holistic and decomposed judgment strategies in job analyses by job incumbents synthesis
- Cornelius, E. T., & Lyness, K. S.
- Journal of Applied Psychology, 65, 155-163
- An online version of the paper cannot be found
- Want a summary of academic papers with alpha? Check out our free Academic Alpha Database!
The purpose of this study was to compare three alternative judgment strategies used to aid job incumbents in making overall evaluations of worker requirements and motivational characteristics of their own jobs. One hundred fifteen job incumbents across several organizations evaluated their jobs using 13 job analysis scales on two occasions. The results indicated that a “decomposed” judgment strategy, using a mechanical algorithm to determine the overall evaluations, resulted in improved data from incumbents as measured by interrater agreement coefficients. Simple overall clinical, or holistic, judgments proved as effective as the fully decomposed judgments when assessing intrarater reliability across time and when comparing incumbent ratings to job analyst and job supervisor ratings. In all instances, a judgment strategy that involved decomposed estimates followed by holistic overall evaluations proved to be the least effective approach to developing summary evaluations by job incumbents. Other results indicate that the education level of the incumbent is directly related to success on the rating task, whereas length of job experience is not.
What is job analysis? There is a great primer at the following link: http://www.ou.edu/faculty/M/Jorge.L.Mendoza-1/job-analysis-criteria-reliability-validity.pdf . In a nutshell, job analysis helps human resources folks and managers determine the job description (what you gotta do), job specifications (the skills needed to accomplish the job), and performance standards (how to identify if the job is getting done). As one can imagine, job analysis is not a trivial task and theoretically involves some high-level thinking.
Here is how the tests go down:
- Identify a variety of jobs with a large amount of workers (janitor, production worker, bank teller, etc.). These workers are called ‘incumbents’
- Holistic job analysis by incumbents involves them rating their job along 13 dimensions based on a given list of task statements (human, low complexity)
- Decomposed-clinical involves the workers rating each task on 13 scales and then rating the overall job on the same 13 scales (human, high complexity)
- Decomposed-algorithm involves the workers rating each task on 13 scales and then a computer algorithm takes that information and generates a rating for the job as a whole (quant)
- The authors then look at how these three sets of analysis compare over 2 job analysis sessions separated by 3-8 weeks. The tests look at consistency and reliability across time.
Here are the results:
- Quant assessment had an 82% correlation with original
- Low complex assessment had an 80% correlation with original
- High complex assessment had an 60% correlation with original
==> As it gets more complicated, computers win. If the inputs for decision making are relatively simple, humans and computers perform about the same.
Here are some more results. Humans aren’t consistent judgers, even when analyzing the same data. As complexity in the decision making process increases, consistency goes down further.
Buffett’s 40′ foot view can work; computer algorithms can work; but deep dive security analysis probably won’t work.
Thoughts on the paper?
Note: This site provides NO information on our value investing ETFs or our momentum investing ETFs. Please refer to this site.
Join thousands of other readers and subscribe to our blog.
Please remember that past performance is not an indicator of future results. Please read our full disclosures. The views and opinions expressed herein are those of the author and do not necessarily reflect the views of Alpha Architect, its affiliates or its employees. This material has been provided to you solely for information and educational purposes and does not constitute an offer or solicitation of an offer or any advice or recommendation to purchase any securities or other financial instruments and may not be construed as such. The factual information set forth herein has been obtained or derived from sources believed by the author and Alpha Architect to be reliable but it is not necessarily all-inclusive and is not guaranteed as to its accuracy and is not to be regarded as a representation or warranty, express or implied, as to the information’s accuracy or completeness, nor should the attached information serve as the basis of any investment decision. No part of this material may be reproduced in any form, or referred to in any other publication, without express written permission from Alpha Architect.
Definitions of common statistics used in our analysis are available here (towards the bottom)