By: Larry Gruppen (@lgruppen)
We may have abandoned the assumption that everyone with an MD can teach, but we still assume that everyone with an MD can assess. I think that assumption needs to be examined.
The roles of faculty members continue to evolve. Whereas expertise in the domain was once considered sufficient for a teaching role, we have come to recognized that teaching requires additional skills beyond content knowledge, such as knowledge about pedagogy and expertise in teaching methods relevant to one’s own specialized practice.1 The remarkable growth in faculty development efforts is evidence for the recognition that teaching is a special role for which faculty need training.
Similar recognition applies to coaching and mentoring. Like teaching, these were often roles that were taken for granted. Some faculty were considered “natural” mentors or coaches and those that weren’t contributed to the academic mission in other ways. However, the growth of intentional coaching relationships and structured mentoring programs reflect a growing recognition that these important roles cannot be left to chance. Formal training and support are needed to make the benefits of coaching and mentoring available to all learners.
Like teaching, mentoring, and coaching, assessment is also a critical component of medical education. Like the other roles, it requires particular skills, knowledge, and expertise above and beyond medical knowledge. These skills include writing effective test questions, probing underlying knowledge required for skilled performance, setting standards for critical outcome decisions, and limiting the impact of cognitive and implicit biases on judgments, among others. In competency-based programs, assessment skills will also include the development, implementation and use of Entrustable Professional Activities as a key part of the assessment program.
The argument for designating selected faculty members as “assessors” follows the same logic as for the other roles. Special designation in these roles helps to protect time for critical activities, legitimize the value of specialized skills, devote priority and effort to learners, and to provide specialized training for these roles. The possible benefits of designated assessors follow.
Validity evidence for institutional assessments.
Designated assessors would be a benefit to any training program designed to develop complex and critical skills. Documenting the attainment of these skills is part of the educational contract schools have with society. Skilled assessors could provide the necessary validity evidence demonstrating the program’s success.2,3 Assessors could provide or acquire content related evidence for the validity of various assessments. Similarly, they could frame the evidence related to response process and particularly internal structure. Simply having individuals who are comfortable with positing validity arguments in assessment decisions will raise the standard for assessment practice and put the institution in a stronger position for accreditation reviews. Designated assessors could also provide the foundation for a community of practice that would foster improved assessment practices.
Bias reduction and management
We know that faculty, as assessors, have important strengths but also significant limitations. A key strength is their ability to observe and judge the performance of learners, particularly in complex and dynamic clinical situations. Faculty, as experts, can identify and interpret critical aspects of performance that may be invisible to non-experts. However, like all human judges, faculty are subject to a number of cognitive biases, such as premature closure, the representativeness heuristic, personal prejudices and many more. Efforts to ”correct” these biases have generally proven to have limited and fleeting effects.4
Faculty typically intermingle assessment with teaching and this confounding leads to a number of concerns. One is the conflict between identifying and addressing learning gaps and the learner’s reluctance to expose those gaps for fear of being “punished” in the faculty member’s assessment. Conversely, faculty may be reluctant to honestly assess a learner’s strengths and weaknesses if they have had a positive or negative working and teaching relationship. Separating the teaching and assessing roles could be done by having different faculty from those who taught a given set of learners do the assessments.
Competency-based medical education needs designated assessors
Designated assessors are even more important in competency-based programs than in traditional programs, in which time is often taken as a substitute for documented competence. CBME would benefit particularly from having designated assessors who could develop and apply the particular skills required to develop, implement, and use EPA performance to judge competence. There are several examples of innovative assessment methods in competency based programs, most of which require expertise and dedicated time from faculty members. The emergence of clinical competency committees is one manifestation 5–8 of faculty in the role of judges where the input assessment data come from various sources (including other faculty members). Most medical schools use standardized patients as an assessment method and these standardized patients require faculty to develop vignettes, specify key performance metrics, judge passing standards, and oversee the rigor and consistency of the SP assessments.
What would a designated assessor look like?
Many institutions already have a few de facto designated assessors in the form of assistant/associate deans of assessment, directors of standardized patient programs, examination committees, competency assessment committees, and assessment blueprints designers. These individuals often focus on administration of assessment programs, but designated assessors could be expanded to include faculty whose key role is to DO assessments and develop refined skills in outcomes connoisseurship and criticism,9 standard setting technologies, faculty development in assessment, designing and implementing programs of assessment. Broader responsibilities could include expertise in clinical performance assessments in the workplace, specifically when the assessor is separate from the teacher.
The cons of designated assessors
Designated assessors would not be without costs and consequences, however. One hazard is the potential dilution of responsibility – if she’s getting paid to assess and I’m not, it’s not my priority. Assessors clearly need to do more than “ordinary” faculty to justify the investment. Another challenge may be finding candidates for this role if assessment issues don’t appeal to a sufficient number of faculty. Finally, designated assessors are not necessary for any and all assessments, but it may be difficult to determine which assessments should be the focus of the assessors.
The future
If designated assessors are to reflect the history of teachers, coaches and mentors, it will require a political decision on the part of the institution to support these faculty roles as a means of addressing institutional assessment issues. It will also require a community of individuals with. dedication to and expertise in assessment. There is already a significant cohort of assessment experts nationwide that can support this need. Additionally, national organizations, like the National Board of Medical Examiners, are directing resources to training a select cohort of medical school faculty to acquire skills in assessment research and practice. The benefits of designated assessors would apply to medical education at all levels.
—————————————————————————————————
About the author: Larry Gruppen PhD is Chair of the Department of Medical Education and a professor at the University of Michigan Medical School.
References
1. Shulman LS. Those who understand: Knowledge growth in teaching. Educ Res. 1986;15(2):4-14.
2. Messick S. Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. Am Psychol. 1995;50:741-749.
3. Downing SM. Validity: On the meaningful interpretation of assessment data. Med Educ. 2003;37(9):830-837.
4. Gingerich A, Kogan JR, Yeates P, Govaerts MJB, Holmboe ES. Seeing the “Black Box” differently: Assesssor cognition from three research perspectives. Med Educ. 2014;48(11):1055-1068.
5. Ekpenyong A, Padmore JS, Hauer KE. The Purpose, Structure, and Process of Clinical Competency Committees: Guidance for Members and Program Directors. J Grad Med Educ. 2021;13(2 Suppl):45-50.
6. Kinnear B, Warm EEJ, Hauer KKE. Twelve tips to maximize the value of a clinical competency committee in postgraduate medical education. Med Teach. 2018;Epub ahead(11):1-6.
7. Hauer KE, Edgar L, Hogan SO, Kinnear B, Warm E. The Science of Effective Group Process: Lessons for Clinical Competency Committees. J Grad Med Educ. 2021;13(2 Suppl):59-64.
8. Duitsman ME, Fluit CRMG, van Alfen-van der Velden JAEM, et al. Design and evaluation of a clinical competency committee. Perspect Med Educ. 2019;8(1):1-8.
9. Eisner E. Educational connoisseurship and criticism: Their form and functions in educational evaluation. J Aesthetic Educ. 1976;10(3/4):135-150.
The views and opinions expressed in this post are those of the author(s) and do not necessarily reflect the official policy or position of The Royal College of Physicians and Surgeons of Canada. For more details on our site disclaimers, please see our \’About\’ page
