On the “Defensibility” of Prospective Summative Entrustment Decisions

By: Holly Caretta-Weyer (@holly_cw)

Imagine yourself in a courtroom having to defend your decision to graduate a trainee to a jury of your peers. What evidence would you bring? What would you say? How would you compose your argument?

You may be thinking right now that this would never happen. But it already is! We graduate trainees every year and render prospective summative entrustment decisions. These may be time-based and by default; however, we are ritually and routinely allowing trainees to go out and enter unsupervised practice. And while we may not explicitly have to defend each and every one of those decision, perhaps this is more in line with how we should be considering our decision-making in light of the transition these trainees are undertaking to go out and care for patients and society without supervision.

To that end, there have been several sobering moments with regard to trainee certification for unsupervised practice over the last several years. First, Jonker, et al¹ identified 43% of study participants who admitted to having certified trainees they did not deem sufficiently competent or entrust with the unsupervised care of their relatives. They cited deficient evaluation criteria, missing data, inability to stop trainees from getting certified, and financial consequences of failing a trainee. Second, the American Program Directors in Internal Medicine identified in 2016 that 52% of program directors had graduated someone in the previous three years that they had concerns about their ability to practice independently.² Finally, Dr. Ben Kinnear asked in an alarming Twitter poll “would you trust your loved ones to each trainee you certify/graduate?” 49% of respondents said “nope, not all of them” and a further 15.4% said “I plead the fifth.”³

Prospective Summative Entrustment Decision-Making Processes

Competency-based medical education (CBME) aims to create a world in which it is the evidence of competence that defines when a trainee is ready for unsupervised practice, not simply time spent in training. In order to make defensible prospective summative entrustment decisions that we can have confidence in and ensure that trainees are prepared to care for society, we need to define robust processes by which these decisions are made.

There are numerous risks and assumptions that we encounter along the way when considering the summative entrustment process and its requisite outcomes. Our aim is for learners to be competent for unsupervised practice and to be able to meet the needs of patients and society. This requires programs of assessment designed to reflect meaningful trainee performance as well as outcomes that align with what is required for competent unsupervised practice. The program of assessment must also provide sufficient data for the competency committee to formulate decisions that they are comfortable with and feel reflect a trainee’s true level of entrustment for unsupervised practice.

How Do We Get There?

By returning to the courtroom analogy, we can think about the work of competency committees as combing through evidence, formulating summative entrustment decisions, and defending their choices about whether a trainee is ready to progress or not at the time of review. But how should this work in a practical sense? Kinnear, et al⁴ provide a scaffolding for optimal competency committee processes. This includes the following key domains:

1. Build a Robust Program of Assessment: A program of assessment encompasses multiple assessment data sources (workplace-based assessment, multi-source feedback, simulation etc), from multiple assessors, in multiple contexts, and at multiple points in time. Assessment should be “fit for purpose,” meaning that we are measuring outcomes of interest and those outcomes are the ones that the competency committee requires to ensure competent unsupervised practice. While objective assessment data in both quantitative and narrative form is essential for these decisions, there are several grounding phenomena that, while not explicitly measured, competency committee members cite as essential for decision-making. Schumacher, et al⁵ in a recent grounded theory study identified these phenomena as trainees’ ability to know their limits and seek help as forming the foundation of prospective summative entrustment decision-making.
2. Optimize Competency Committee Composition: Competency committee members should be varied in their experience and exposure to trainees. Senior faculty and junior faculty, faculty from different clinical sites, faculty who work on different shifts or in different contexts, and those with varied skillsets bring a diversity of perspectives to the summative entrustment decision-making process. This helps ameliorate bias and group think during deliberations and ensure a more robust and defensible decision-making process.
3. Be Deliberate About the Output of the Competency Committee: The work of the competency committee is certainly to make prospective summative entrustment decisions, and this is the primary output. To that end, it is also essential that the committee convey any gaps in the program of assessment that prevent those decisions from being made. However, the committee should also communicate their decision to the trainee and the rationale behind it as well as what needs to be done to make the requisite progress if the answer is “not yet.” Given that the aim is to graduate trainees competent to care for society without supervision, this requires both robust decision-making and provision of individualized learning plans that foster adaptive expertise and continued growth and development.

Finally, when considering how to operationalize this process, it is imperative for certifying boards and organizations to consider a more fluid, ongoing process. Initial certification is only one part of the puzzle at which point a knowledge test is frequently administered. When considering the desire for ongoing growth and development and meeting the needs of society, the rendering of prospective summative entrustment decisions is a key part of that process and must be incorporated as certification processes evolve within the adoption of CBME.

—————————————————————————————————————

About the author: Holly Caretta-Weyer MD MhPE, is the Associate Residency Program Director and Director of Evaluation and Assessment for the Stanford University Emergency Medicine Residency Program as well as EPA/CBME Implementation Lead at The Stanford University School of Medicine.

References

1.Jonker G, Ochtman A, Marty AP, et al. Would you trust your loved ones to this trainee? Certification decisions in postgraduate anesthesia training. Br J Anaesth. 2020 Nov;125(5):e408-410.

2. Association of Program Directors in Internal Medicine, Residency and Fellowship Administration Annual Survey, 2016.

3. Kinnear B. August 12, 2020, 3:00PM, Twitter. (@Midwest_MedPeds)

4. Kinnear B, Warm EJ, Hauer KE. Twelve tips to maximize the value of a clinical competency committee in postgraduate medical education. Med Teach. 2018;40(11):1110-1115.

5. Schumacher DJ, Michelson C, Winn AS, et al. Med Educ. 2022 Mar 8. Online ahead of print.

The views and opinions expressed in this post are those of the author(s) and do not necessarily reflect the official policy or position of The Royal College of Physicians and Surgeons of Canada. For more details on our site disclaimers, please see our ‘About’ page