This week, the KeyLIME host discuss accreditation systems. While the goal of accreditation is to ensure the quality of medical education, there is limited research that links it to improved student and graduate outcomes. The authors of this article look to answer the question: \”what is the \’evidence\’ for accreditation on a system-wide scale?\” by comparing the performance of students and graduates from medical schools accredited by an agency recognized to individuals who attended schools that did not meet this criteria.
Click here to listen to the episode.
————————————————————————-
KeyLIME Session 358
Reference
van Zanten et. al., Making the grade: licensing examination performance by medical school accreditation status. BMC Med Educ. 2022 Jan 14;22(1):36.
Reviewer
Jason R. Frank (@drjfrank)
Background
Full disclosure: I help oversee a residency accreditation system. As part of that job, I have looked at similar systems around the world. Similar is of course relative, as it is remarkable how much they vary. The one I work in involves extensive standards, program and institutional electronic portfolios, trained surveyors, painstaking visits, trainee reports, and hours of committee adjudication. One I visited involved inviting a prominent professor for dinner, who then wrote up a government-mandated report.
I have heard residents say that accreditation was the only thing that stood between them and an abusive learning environment. I have heard education deans say that accreditation was the most effective way to ensure quality meded. I have heard a department head say that accreditation was a “*$*#&ing waste of time.”
So what is the “evidence” for accreditation on a system-wide scale? The logic model here assumes that a quality institution has quality programs that have quality learning activities that lead to learning that lead to better doctors. Accreditation ensures quality and CQI. Previous studies have correlated accreditation status of worldwide medical schools and USMLE scores, but these were on a small population of schools. The world is seeing a huge growth in the number of medical schools, more accrediting bodies, and a dramatic rise in physician migration across borders.
Purpose
Enter Marta van Zanten, Jack Boulet, and Christine Schiffer from FAIMER and ECFMG in the US: they set out compare the USMLE pass rates for 2018-2020 among international medical graduates (IMGs) from schools that were or were not accredited via the World Federation for Medical Education. This paper appears in BMC Med Ed in 2022.
Key Points on the Methods
Since 2010, the WFME has been accrediting medical school accreditation agencies in the world on a voluntary basis. All schools “accredited” by a WFME recognized agency are considered “accredited” for this study. Currently there are 23 accrediting agencies in 57 countries. Accreditation status was extracted from public websites.
39650 IMGs applied for ECFMG certification in the US between 2018-2020 and wrote one or more of the USMLE exams. The first attempt pass/fail rates were cross tabulated against medical school WFME accreditation status. Data were aggregated and anonymized.
Key Outcomes
Of the 39650 candidates, 48.5% were female. Top 5 countries of medical training of the 158 represented were: India, Grenada, Pakistan, Barbados, and China. Most (81.6%) did medical training in English, while 60.7% indicated that English wasn’t their first language.
Applicants from accredited schools had higher scores on all USMLE components (basic science, clinical knowledge, clinical encounters, communications, and English proficiency). The difference was greatest for the Step 1 written basic science exam (88.4% vs 78.6%).
There was no analysis of accredited school populations vs those who failed WFME accreditation.
Key Conclusions
The authors conclude…that individuals who attended WFME-accreditation medical schools did better on USMLE exams. Since USMLE exam scores are correlated with board certification and residency performance, the authors see this as validity evidence for the positive effect of accreditation on medical schools in general. This, they said, justifies making WFME accreditation required for IMGs going forward.
The authors acknowledge that they did not compare those who failed WFME accreditation, nor accounted for curriculum content or USMLE preparation courses. There was no accounting for socioeconomic context of the schools or candidates. Candidates were clearly a self-selected group.
Spare Keys – other take home points for clinician educators
- Decades of studies provide validity evidence for high stakes exams, as they correlate with other physician career, performance, and patient outcomes.
- However, high stakes exams have also been shown to manifest structural biases that favour certain populations over others.
- Accreditation, like all aspects of #meded, needs rigorous examination to ensure the best possible designs in the various contexts of HPE systems around the world.
- Beware studies that use a blunt measure (pass/fail or accredited/not) to represent complex constructs like “quality education.”
Access KeyLIME podcast archives here
The views and opinions expressed in this post and podcast episode are those of the host(s) and do not necessarily reflect the official policy or position of The Royal College of Physicians and Surgeons of Canada. For more details on our site disclaimers, please see our About Page