“Rater cognition (i.e. how an individual constructs an assessment of performance) has been a critical element in ensuring the validity of direct observation assessments” and is also the topic of this week’s KeyLIME session! Interested? Read on, and check out the podcast here (or on iTunes!)
————————————————————————–
KeyLIME Session 157:
Listen to the podcast
Read the episode abstract here.
Gomez-Garibello C. Young M.Emotions and Assessment: Considerations for Rater-Based Judgments of Entrustment. Medical Education. 2017. [ePub ahead of print)
Reviewer: Jon Sherbino (@sherbino)
Emotions and Assessment: Considerations for Rater-Based Judgments of Entrustment
Background
(This episode is part of an agreement between Medical Education – the journal – and KeyLIME. We get access to select manuscripts in advance of publication for review on the podcast. The KeyLIME hosts have complete editorial autonomy to choose a manuscript and provide a review without journal approval. KeyLIME is grateful for this collaboration with Medical Education.)
The assessment literature has seen strong a migration towards emphasizing direct observation and the use of expert raters in the last decade. Rater cognition (i.e. how an individual constructs an assessment of performance) has been a critical element in ensuring the validity of direct observation assessments. The influence of a cohort, the frame of reference of the observer, the confounding between global assessments and deconstructed checklists, numeric versus narrative inputs, even gender are all elements that influence the individual rater’s “score.” We’ve discussed this previously on KeyLIME: <138, 133, 122, 86, 78>
What I haven’t yet read in the literature is the influence of the social and psychological dynamic between trainee and observer. Or more specifically, what is the effect of emotion on assessment. Emotion influences how we identify, process and act on information. This paper gives us a glimpse of the effect of emotion on rater’s observations. If you’re interested, read on. Better yet, listen on… and click to the podcast.
Purpose
“We … specifically consider the role that emotions could play in rater-based assessment, with a focal lens on EPAs.”
Key Points on Methods
Not really relevant as this is a narrative review. See our previous discussions that suggest narrative reviews are at high risk of confirmation bias, where authors may unintentionally present summaries of the literature that support their view point.
Nonetheless, narrative reviews provide digestible structure to large and complex topics that typically span a number of disciplines.
Key Outcomes
From General Education Literature:
1. Control Value Theory suggests that learner’s perceived control and value of an educational task and their emotion (pride, frustration) associated with the outcome on the task influence future performance. (e.g. test anxiety).
a. This theory is predominantly interpreted through the lens of the learner; however, the emotions of the teacher are also to be considered when determining the influence on learning (e.g. teacher enthusiasm and learner engagement).
2. A level of arousal is necessary to optimize performance, but extremely high or low arousal will impede performance.
3. Decision making is influenced by: mood of the rater, incidental emotions (emotion elicited by the decision) and expected emotions (anticipated emotions as a result of the outcome of the decision).
From Rater Cognition Literature:
1. There is no literature specific to emotions and rater-based assessment in HPE. However, one can speculate on the influence of emotion and it’s relationship to trust (a social construct) that is at the heart of many direct observations, particularly EPA assessments.
a. Emotions could potentially change arousal in raters leading to a shift in attention to trainee performance (e.g. bias).
b. Emotions could potentially add random noise to the assessment.
c. Emotions may be a legitimate source of information in assessment decisions. Excluding or systematically controlling for emotion may inappropriately decontextualize a direct observation.
Recommendations:
The authors suggest that the classic model of assessment (Pellegrino’s Triangle), which includes observation, cognition and interpretation, incorporate emotion as a key component of a new tetrad.
Key Conclusions
The authors conclude…
“Recognizing emotions as a key component of assessment and decision-making processes moves us away from conceptualizing emotions as merely non-cognitive sources of variation in developing assessment judgments… The inclusion of emotions as an element within the assessment process suggests a move away from the idea of a rater as a ‘measurement instrument’ and instead embeds the assessment process within a social and emotional context.”
Access KeyLIME podcast archives here
Check us out on iTunes