The Key Literature in Medical Education series continues on the ICE blog.
You can subscribe to the podcast on iTunes here or listen directly via the webpage here. The podcast goes into greater details, debate, argument (thanks @Drjfrank…) but if you want a quick synopsis, than the abstract below will do!
– Jonathan (@sherbino)
KeyLIME Session 67 – Article under review:
View/download the abstract here.
Shalhoub J1, Vesey AT, Fitzgerald JEF. What Evidence is There for the Use of Workplace-Based Assessment in Surgical Training?, Journal of Surgical Education, (14): 1-10
Reviewer
Jonathan Sherbino (@sherbino)
Background
As we have discussed many times on KeyLIME, authentic assessment (based on observation of performance in a clinical context) is key to the determination of competence. Inferred competent performance (i.e. based on knowledge-based tests or in simulated environments) is insufficient for contemporary unsupervised medical practice. Of course, this does not imply that other assessment instruments do not have a useful role. Rather, work-based assessment WBA instruments (WBAs) must be incorporated into a larger assessment program that also includes knowledge-based tests or simulation-based instruments. Instruments must be fit for purpose with a collective program determining global competence of a trainee.
Obstacles to incorporating WBAs include confusion over design, challenges to validity, and aversion to change.
Purpose
“This article aims to review and discuss the existing empirical educational evidence that surround the introduction and ongoing use of WBAs in surgical training.”
Type of paper
Narrative review
Key Points on the Methods
A simple PubMed search was performed. It was expanded via a hand search of institutional reports of WBAs.
This study did not follow the PRISMA guidelines (see here http://www.prisma-statement.org/) for systematic reviews. Therefore, there is a high probability of bias in the results.
Key Outcomes
Satisfaction survey data from British surgical trainees indicated a dislike of WBAs when they were first introduced. However, this trend was reversed upon a subsequent resurvey three years later.
One trial of procedure-based assessment of surgical trainees indicated good traditional psychometric qualities.
One consensus report from a national surgical education body (Intercollegiate Surgical Curriculum Program) argued that WBAs inhibit the formative relationship between learner and teacher with an impact on an individual’s progression within Maslow’s hierarchy of needs.
Outside of the surgical literature a number of studies and systematic reviews comment on the psychometric qualities of different instruments (i.e. miniCEX, MSF) and the formative aspect of WBAs (e.g. feedback).
Key Conclusions
The authors conclude…”Limited evidence has potentially undermined the introduction of WBAs in surgical training to date. There are misunderstandings regarding their use as either summative or formative educational tools. These shortcomings are an opportunity for further work.”
Spare Keys – other take home points for clinician educators
WBAs are a classic example of the proprietary naming of a slightly modified common instrument. What is the difference between a mini-CEX or STACER? An OSAT or DOPS or PBA? All have in common a context, a scope of performance, and a scoring template. The resistance to WBAs from teaching faculty may be partially ameliorated by opting for clarity and simplicity in design. Moreover, all clinical teachers should be reminded that the history of medical education is founded on the practice of experts observing novices in action and providing judgments about performance.
Access KeyLIME podcast archives here
