On KeyLIME this week, Linda discuses Workplace-based assessment (WBA), a hot topic in Med Ed. Despite it being essential to CBME and used (in part or whole) in many clinical learning settings, our understanding of all it has to offer learners is limited, with ‘varied and conflicting evidence’. Read on, and check out the podcast here (or on iTunes!)
————————————————————————–
KeyLIME Session 159:
Listen to the podcast
Read the episode abstract here.
Barrett A1,2, Galvin R3, Scherpbier AJ4, Teunissen PW4,5, O’Shaughnessy A1, Horgan M2. Is the learning value of workplace-based assessment being realised? A qualitative study of trainer and trainee perceptions and experiences. Postgrad Med J. 2017 Mar;93(1097):138-142.
Reviewer: Linda Snell (@LindaSMedEd)
Background
Workplace-based assessment (WBA) is a hot topic in Med Ed, but despite it being essential to CBME and used (in part or whole) in many clinical learning settings, our understanding of all it has to offer learners is limited, with ‘varied and conflicting evidence’. WBA can have impact on changing ‘observable’ behaviours (e.g. procedural skills, and other ‘simple learning’). We do not know if WBA has the potential to impact aspects of learning that are more difficult to observe, (e.g. critical thinking, affirmation of good practice).
In PGME, WBA was introduced as a potential solution to the issue of infrequent observation and feedback, and to inform trainee learning and development.
How WBAs play out in actual practice remains unclear, due to variable implementation and tools.
A number of systematic reviews concluded that WBA does not show a demonstrable change in practice, but ‘practice’ here was defined narrowly; other potentially important and less tangible learning effects, such as reinforcement of positive behaviours and affirmation of good practice, may be undetected.
Purpose
“to explore perceptions of learning value connected with WBA from the perspectives of trainers and trainees, and to understand how their WBA experiences have shaped those perceptions.”
Type of Paper
Research: qualitative methodology using template analysis
Key Points on Methods
Conceptual framework Teunissen’s ETR (experience, trajectory, reification) – here experiences of WBA, development of patterns of working with WBA over time, make WBA concrete, a reality, in practice such that it impacts learning.
Two research questions:
1. What are trainers’ and trainees’ perceptions of the learning value of WBA?
2. How have trainer and trainee WBA experiences shaped these perceptions?’
Individual interviews, PG trainers and trainees in 6 programs in RCP Ireland. In these programs WBA required since 2010, using WBA tools like MiniCEX, OSATS, DOPS, recorded in an e-portfolio.
Consolidated Criteria for Reporting Qualitative Research standardized reporting guidelines.
Participants had to have used at least 1 WBA in past year!
Data analyzed using a template analysis method – reading and re-reading transcripts for high-level themes and subsequent ‘subthemes’. Visual mapping of identified themes using post-it notes, which allows for ‘clustering’ of the identified themes. Trainees and trainers analyses done separately and then common and differing themes examined.
Key Outcomes
9 trainees, 8 trainers
2 sections: WBA experiences and learning value of WBA
Common themes:
-Purpose – tick-box exercise, done post hoc and not prospectively- perception of lack of authenticity
-Who is responsible for WBA? – the trainers thought it was the trainee and the trainees thought it was a joint effort
-Feedback – trainers thought that they provided it, and the trainees thought it was a rare occurrence.
Trainee themes:
-WBA ‘forces observation’ and may represent ‘protected training time’
-The ‘learning value’ of WBA was more hypothetical than actual – an unrealized concept. Trainees liked the idea of being able to follow a learning trajectory.
-Using the ETR framework identified limitations of WBA implementation; explicitly looking at whether, or how, WBA supports learners’ development, or trajectories in ETR terms, revealed the relative absence of this expected effect, highlighting need to explore the educational impact of WBA.
Key Conclusions
The authors conclude that there is a “need to re-conceptualize WBA as a continuous practice, in which assessment, feedback and deliberate practice become normalized. Ongoing WBA practice should be supported by well-designed tools rather than thought of as a stand-alone set of tools restricted by poor implementation practices.”
Spare Keys– other take home points for clinician educators
Find a data analysis method that works – does template analysis work?
Find a conceptual frame that fits – does the ETA framework fit??
Shout out
Thanks to Zakia Dimassi at the American University of Beirut for recommending this paper.
Access KeyLIME podcast archives here
Check us out on iTunes