Can prior information about a learner or familiarity with a learner affects clinical ratings? This week’s study takes a look at whether prior performance information can influence the assessment of learners, and also examines how participants use Learner Handover in their assessments.
KeyLIME Session 289
Reference
Shaw et. al., How biased are you? The effect of prior performance information on attending physician ratings and implications for learner handover. Advances in Health Sciences Education. June 2020.
Reviewer
Linda Snell (@LindaSMedEd)
Background
Learner handover, or LH, also known as forward feeding or learner educational handover, is ‘the process of sharing of information about learners between supervisors for the purposes of academic and professional development in medical education.’ There are positive and negative aspects to LH. On the plus side, it can turn multiple low-stakes assessments over time into a continuum (especially in CBME) so learners can build on prior assessments; identify learners with consistently marginal performance; let teachers individualize the learning; facilitate programmatic assessment; be good for patient safety and ease transitions. On the minus side, is the potential to bias future assessments, and ‘self-fulfilling prophecies’, and concern about stigmatization, privacy or litigation.
Many institutions expressly forbid LH or ‘information sharing’ (at least formally), few have explicit policies allowing it.
Can prior information about a learner or familiarity with a learner affects clinical ratings? There is some evidence it can from the MedEd literature (usually personal knowledge of learner), and loads of evidence from outside medicine. Three terms to know:
- “context effect” is used to describe any influence that prior performance information (PPI) or other factors may have on subsequent ratings.
- “contrast effect” bias away from the direction of the PPI
- “assimilation effect” bias towards the direction of the PPI
There seems to be a ‘dose response’, a larger effect noted for more extremely negative or positive PPI; and a larger effect with negative compared to positive PPI.
However, little is known about the effects of indirect PPI or LH in the MedEd context.
Purpose
To determine whether prior performance information either positive or negative (PPI) in the form of LH influences the assessment of learners in the clinical context.
Second question aim to ascertain how participants use LH in their assessments.
Key Points on the Methods
Purposive sampling of faculty members and final year residents acting as junior attendings in 1 department at 1 university. Sample size calc 42.
Quasi randomized to ensure similarity of factors gender, experience, specialty, into 3 groups, given + or – or no LH on 6 ‘learners’. LH form created to summarize learners’ skills in medical expert, communicator/collaborator and professionalism. Negative LH described a learner who has several areas requiring attention, positive LH described a learner with superior performance. Forms piloted but not validated.
Each participant saw same 6 videos of an acted clinical encounter portraying average performance and performed a Mini-CEX.
Descriptive and comparative stats, effect sizes, effect of order of LH;
Also had a post video questionnaire asking how raters used the LH provided, their impressions on LH and its credibility and whether participants were able to deduce the true purpose of the study.
Key Outcomes
Population – good mix of GIM, other subspecialists; sex; experience
No interaction of order and LH conditions
LH effect on mini-CEX and overall competency score: significant difference between negative LH and positive LH. Effect size large the difference between negative LH and control and positive and control not statistically significant.
LH use: 78% intervention participants were able to deduce the true nature of the study.
71% stated they avoided using LH by trying to ignore the information or reported not using the information at all. 25% used the LH to focus their assessment.
LH views mixed: in favour of LH would use the information to support struggling learners (50%), obtain multiple perspectives of learner performance (12%), monitor patient safety concerns (14%) or focus observation energies when time is limited (9%). Introduction of bias was the most commonly identified concern (33%)
Key Conclusions
The authors demonstrated an assimilation effect related to LH depicting learners at extremes of performance despite the presence of mixed expert and novice raters and awareness of the potential for bias, supporting the need for careful consideration if LH is to be implemented to facilitate CBME.
Spare Keys – other take home points for clinician educators
Should we use LH for low stakes formative assessments? Even ‘low stakes’ can be viewed as high stakes by learners.
LH may affect rater cognition – influence what is actively observed, or how information is processed and categorized
Access KeyLIME podcast archives here
The views and opinions expressed in this post and podcast episode are those of the host(s) and do not necessarily reflect the official policy or position of The Royal College of Physicians and Surgeons of Canada. For more details on our site disclaimers, please see our ‘About’ page