In the United States, there is a movement for a more holistic review of residency candidates that de-emphasizes standardized test scores.
To obtain a residency position, allopathic medical students must take the United States Medical Licensing Exam (USMLE) Steps 1 and 2, and osteopathic medical students must take the Comprehensive Osteopathic Medical Licensing Examinations (COMLEX) Levels 1 and 2, although many take USMLE tests as well.
Program directors use these test scores as a major deciding factor in residency and fellowship selection, and there is a tight correlation between standardized test performance, the competitiveness of the residency, and the eventual salary of physicians graduating from these residencies.
However, these test scores are poor predictors of performance in residency.
USMLE Step I in particular has come under fire for adding unnecessary and unproductive stress to medical students, and there have been multiple calls to either make the test past/fail or to do away with it all together.
There are vocal proponents for both positions. One side: standardized testing with normative comparison causes stress, tests for the wrong things, is expensive, and unfairly penalizes those typically underrepresented in medicine. The other side: medicine is hierarchical and testing should reflect this, a pass/fail strategy penalizes those who score best, medical schools are so variable that standardized testing levels the playing field, and residency programs are rated on board pass rates and therefore they need to know who the failure risks are.
As a program director, I have conducted approximately 4000 interviews over the past decade and, until now, I have always known the test scores of the person I was interviewing.
This year, our team reconsidered this position. Given what we know about implicit bias, it is almost certain that my appreciation for an applicant as a person is greatly influenced by their test score. If so, how can this be a truly holistic review? If I’m interviewing someone with a top score, it would be so easy to overlook any flaws I might see, just as surely as I might be unforgiving knowing an applicant has a low score.
So, this year I’ve decided not to know the scores of our residency applicants before I meet them.
Two associate program directors do review test scores during the application process, but we have a wide range of performance in those we select for interview. I am blinded from this review, as are the chief residents and key interviewing faculty. The only information I request to know is if there is a large gap (typically an increase) between the first and second set of Step scores (I want to know what they learned about learning). Even with this knowledge, I still don’t know the actual scores.
Here’s what I’ve learned so far: I simply cannot tell a person’s test scores by interviewing them. I can, however, get a sense of their humanity, insight, grit, leadership style, interpersonal skill, open mindedness, and willingness to engage.
There may be a bit of a Hawthorne effect here (since I’m studying myself), but I’ve greatly enjoyed the change. Without test scores I can’t decide about a person before they sit down with me. I have to be fully present in every moment, open to wherever the conversation goes. I have to really listen. Hearing someone’s story in this way has been a revelation.
Once the interviews are conducted for the day, our team decides where an applicant should be on the list before we look at the scores. This decision is of course influenced by all the biases we naturally hold as humans, but we try to mitigate this as best we can by having many different people involved in the assessment. Once the list is made, we do review the scores to determine if we should have any significant concerns regarding academic performance. More than 99% of the time we make no changes to the list. The tables are turned – our impression of the person outweighs our impression of their test scores, and not the other way around.
What happens when a program director doesn’t know your test scores?
I don’t know yet.
When I’ve asked the few applicants with significant increases between test performances why there was a gap, I tell them that I don’t know the scores, only the gap. After they answer, I ask them how they feel about me not knowing their scores. Most say they support this. Later it is clear that some people went from excellent scores to 99th percentile scores, and I wonder if they secretly feel cheated by me not having this information during the interview. And what do those with lower scores think?
One caveat about our program that allows us to be score agnostic: we’ve deeply invested in applying adult learning theory to residency training, and most of our residents out-perform their pre-residency scores on the in-training exams they take while in our program. We know that our curriculum can help any person with grit and an open mind improve medical knowledge and test-taking skill (our board pass rate is very high). If we didn’t have this confidence, we don’t think we would be as comfortable with the process described above.
Not knowing test scores ahead of time has certainly changed how we interview, and our list so far is different from any other we’ve constructed. As of this writing we are about halfway through interview season, so we won’t really know what will happen until match day, and ultimately, until graduation day.
I will check back into this space in the future and let you know how we did. If USMLE goes pass/fail, consider this a dry run into the unknown.
Until then, what are your thoughts? Could you and your team consider not knowing?
The views and opinions expressed in this post are those of the author(s) and do not necessarily reflect the official policy or position of The Royal College of Physicians and Surgeons of Canada. For more details on our site disclaimers, please see our ‘About’ page
