The Key Literature in Medical Education podcast tackles selection methods this week. From the help of our Aussi friends at onthwards, Marie-Louis Stokes and Anthony Llewellyn, we try to determine whether graphology (i.e. the study of handwriting to differentiate candidates… seriously, this used to be a thing) or behavioural interviewing (i.e. having the candidate perform a representative task relevant to the discipline) or another method can help you weed out the outstanding candidate from your towering stack of applications.
As a McMaster University faculty, I need to give a shout out to the MMI (Multiple Mini Interview) technique that did not get discussed in the podcast. If you’re curious more details here.
For more details on selection techniques, check out the abstract below. For a richer debate on how this classic paper from psychology informs medical education, check out the podcast here.
– Jonathan
————————————————————————–
KeyLIME Session 118 – Article under review:
View/download the abstract here.
Schmidt F, Hunter, JE. The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings. 1998. Psychological Bulletin; 124(2):262-274.
Reviewer: Anthony Llewellyn(@drallewellyn)
Background
This is a classic and often cited study from the psychology literature.
Implicit to the role of many clinician educators is selection. We often spend significant amounts of time devoted to the process of selecting students into medical school and doctors for trainee and higher level jobs. We generally draw our colleagues, support staff and other professionals into the process. The time-related costs of all this activity are often significant.
Selection into medical schools and training posts is competitive in a number of countries. Yet often it seems that a “standard approach” to job selection for trainee positions has been in place for some time, i.e. submission of a paper based job application, if too many applications there is generally a paper-based culling process, followed by an interview with a panel and some form of reference check.
Many lead authors in the medical eudcation space such as Fiona Patterson suggest that there may be better approaches such as: assessment centres; multiple mini interviews; and situational judgement tests.
The Schmidt and Hunter paper, whilst somewhat dated now is one most often cited as a source paper in both medical education articles looking at the selection question as well as in many mainstream human resource and organizational texts.
Purpose
In this paper the authors look at the performance of 19 various types of selection tools for personnel recruitment by summarizing 20 plus years of prior work.
Type of Paper
Systematic review; Meta-analysis; Commentary.
Key Points on Methods
The authors summarize the development of research into personell selection techniques which commenced in the 1900s. It was soon evident that some techniques were more valid than others. Research from post World War II revealed that the findings for the validity of selection techniques were not only valid in certain work categories, when combined in meta-analyses, with very little to no variability. But that equally very little to no variability was found when studies combined results across work categories.
The authors compile meta-analytic findings of 19 personell selection techniques which have been trialled in combination.
The General Mental Ability (GMA) test is identified as the gold standards selection test versus which all other tests have been trialed in combination with this test.
The authors present findings for the GMA in combination with other tests for both job performance post selection as well as performance in a job training program post selection.
The statistical techniques are somewhat valid and would be deemed acceptable for their time period. The authors do not give any information as to how they sourced their studies for the review.
Key Outcomes
The authors note that GMA is noted in the research to be the most valid predictor of future performance when hiring someone who has not performed the job before.
Whilst not an area of comment for this article the reviewers agree that we could consider the GMA as equivalent to the level of mental ability one requires to successfully gain entry into medical school and thereafter we are likely to be dealing with professionals with quite high levels of GMA (or equivalent).
In combination with GMA the following techniques were found to be most powerful in improving selection for job performance:
- Work Sample Tests (24% increase in validity)
- Integrity Tests (27% increase in validity)
- Structured Interviews (24% increase in validity)
In combination with GMA the following techniques were found to be most powerful in improving selection for performance in job training programs:
- Integrity Tests (20% increase in validity)
- Conscientiousness Tests (16% increase in validity)
A number of techniques were found to have no or little increase in validity, including:
- Job experience
- Biographical Data
- Assessment Centres
- Points method for evaluating training and experience
- Years of education
- Interests
- Age
- Graphology
Techniques that were somewhere in the middle included:
- Job knowledge tests
- Job tryouts
- Peer ratings
- Behaviorally consistent method of assessing training and experience
- Reference checks
Key Conclusions
The authors conclude that the cumulative findings show that research knowledge now available make it possible for employers to substantially increase the productivity, output and learning ability of their workforces by using procedures that work well and by avoiding those that do not.
The authors suggest that a combination of assessment of GMA + integrity or GMA + structured interview are the best approach for personnel selection dependent upon the circumstances of selection.
The authors recognize that their study is limited to those studies looking at two techniques in combination and that many selection processes involve a greater number of tools. They further suggest that some techniques are likely to combine aspects of different selection techniques, for e.g. a structured interview may at the same time assess GMA, as well as training and experience using behavioural interviewing techniques.
The authors also suggest that many employers are currently using suboptimal selection techniques.
Spare Keys – other take home points for clinician educator
This article is worth knowing about as it is often cited by Human Resource and Organizational Psychology aspects when looking at the best approach to selection.
You may wish to keep a copy of the tables handy if you are reviewing your selection processes.
It is worth considering that some of the selection techniques which are indicated to significantly improve validity are often not often utilized in selection processes in medicine, even where this may actually make rational sense and might be easily applied.
For e.g. selection processes for procedural training programs could incorporate a basic work sample test, such as demonstrating a procedure on a training mannequin.
Possibly even more worrying is the use of techniques which are seen to be far less valid, for e.g. assigning points for training and experience from an application or resume which is often seen in some medical training selection processes.
It is possible that with increased competition trainees may choose to challenge the validity of selection processes based on evidence (or lack of).
Challenges in adopting new and improved processes for selection into medical school and/or medical positions include:
- Time and resources required to redesign the process
- Convincing those with a stake in the existing process that it is worth changing
.
Listen to the podcast here
