#KeyLIMEPodcast 299: I can’t get no assessment satisfaction

Reference

Schuwirth et., al. A history of assessment in medical education Adv HealthSci Educ Theory Pract. 2020 Dec;25(5):1045-1056

Reviewer

Lara Varpio (@LaraVarpio)

Background

• The authors (two very accomplished scholars who made their careers studying and theorizing about assessment in medical education) offer an interpretive history of how our field’s thinking about assessment has evolved since the 1960s.

Purpose

• The authors present three phases in the development of our thinking about assessment in medical education

Key Points on the Methods

• N/A

Key Outcomes

• Phase one = assessment as measurement. In the 1960s, assessment was deeply informed by the discipline of test psychology. 5 implications:
  • Competence should be captured quantitatively as a single score
  • Quality of assessment = validity and reliability. Reliability = the extent to which scores were reproducible across items, cases, examiners, etc. (internal consistency). Validity => 2 prevailing views: a. each test item was only meaningful to the extent that it contributed numerically to the total score; b. each item as intrinsically meaningful and so the score was a summary statement
  • Assessment test designs mimicked test psychology. Competence = combination of knowledge, skills, attitudes, and problem solving ability. Each of these could be assessed independently of the others
  • Importance of objectivity
  • Purpose of assessment = telling people apart (sufficiently competent or NOT sufficiently competent)
• But :a. subjectivity was NOT the main source of unreliability; b. Objectivity was challenged (assessment necessarily involves human judgement); c. KSAs could not be judged independently of each other.

• Phase 2 = assessment as judgement. 1990s. The purpose of assessment = promoting independence, thoughtfulness and critical thinking. 3 changes:
  • Competence was redefined => competencies, NOT personality traits.
  • Assessment moved back into authentic contexts
  • Recognizing the role of human judgement in assessment processes
• These changes have impact => Validity changes in workplace based assessment. Note: examiners need to have sufficient content expertise AND they have to have expertise in assessment processes. Importance of training examiners to have assessment literacy becomes highlighted
• Construct validity and reliability => recognized as limited.
• Competence was no longer seen as a simple, straightforward phenomenon; instead, competence is complex and multifaceted.

• Third phase = assessment as a system. Now. Education is recognized as more complex, and systems theory has been tapped to help us reconceptualize assessment
• Assessment => whole-system approach.
• When assessment is seen as a system, we seek to keep assessment integrated and holistic. Goal: to engage in meaningful triangulation of information from various sources
• Triangulation information across formats requires a narrative process, not a numeric process.
• Implementing assessment as a system is not easy. It runs contrary to many organizational cultures

Key Conclusions

• What comes next? Can only guess

Spare Keys – Other take home points for Clinician Educators

• If anyone is interested in writing a similar paper => methods point. A paper that explains “this is where we are, this is how we got here, and this is where we should be going next” & highlights turning points in the field’s evolution of thinking => that is called a state-of-the-art literature review

Access KeyLIME podcast archives here