Exams play a very important role in medical education. See Episode 10 of our podcast that suggests that an exam (i.e. retrieval practice) is more effective than concept mapping or “simple studying” (i.e. re-reading/highlighting material).
This episode discusses: is a closed-book exam that requires a learner to have encoded and recall the required information a superior educational process to an open-book exam that requires a learner to appraise and apply an array of widely available information?
See below for the conclusions reached or listed to the podcast here.
————————————————————————–
KeyLIME Session 134 – Article under review:
View/download the abstract here.
Durning SJ, Dong T, Ratcliffe T1, Schuwirth L, Artino AR Jr, Boulet JR, Eva K. Comparing Open-Book and Closed-Book Examinations: A Systematic Review. Academic Medicine. 2016 Apr; 91(4):583-99.
Reviewer: Jonathan Sherbino (@sherbino)
Background
The educational effect of an exam (i.e. anticipation of the exam to motivate a learner to prepare) and the formative effect of an exam (i.e. feedback to a learner to reveal gaps in knowledge to assist in ongoing learning) are key educational principles. From a neuroscience point of view, the act of recalling information reinforces cognitive schemas and synaptic connections, allowing more ready access to the information in the future.
In clinical practice I frequently access on-line resources to guide my clinical practice. Should the exams administered to residents in my program replicate this clinical practice? Yet, the information I access builds on a foundation of information that I have learned (encoded) and recall to narrow the scope of the question that I am searching for answers.
This podcast can’t unpack all of the issues partially submerged in the paragraph above. But it does provide the best evidence to help structure our debate. So, let’s dive in. Closed-book or open-book exams? And what is this “book” thing anyway?
Purpose
“To compare the relative utility of open-book examinations (OBEs) and closed-book examinations (CBEs) given the rapid expansion and accessibility of knowledge.”
Type of Paper
Systematic review
Key Points on Methods
- Scoping search to provide preliminary outcome measures and inclusion/exclusion criteria
- English language, any design with an empirical comparison
- MEDLINE, Embase, ERIC, PsycINFO to 2013
- Biblio search
- Contacted experts
- Google search
- 3 author independent review
- De novo quality assessment
- Research question
- Hypothesis
- theoretical framework
- trustworthiness of results
- study rigor
- appropriate data analysis
Key Outcomes
- 4192 –> 37 included
- Poor to moderate quality
- 92% single institution
- 65% college-level
- 57% end-of-course exam (medium level stakes)
- Exam prep
- Increased preparation time with CBE
- No difference in strategy/tactic
- Test anxiety
- Poor data
- Students overestimate the reduction of anxiety with OBE
- Exam performance
- Confounder:
- OBE leads to different questions types (i.e. Application rather than retrieval of knowledge)
- Slight improved performance CBE
- Psychometrics
- Improved reliability of CBE because much longer testing time required per question for OBE, therefore fewer questions available to be answered
- Testing effects
- Test-enhanced learning equivalent
- Public perception
- Students prefer OBE
- But indicate exam is harder and requires training for the format
- Teachers concerned with increasing resources required to design OBE
**learners have limited experience with OBE
**limited studies that allowed internet resources and Google search
**few regarding physicians
Key Conclusions
The authors conclude…
“Given the data available, there does not appear to be sufficient evidence for exclusively using CBE or OBE. As such, a combined approach could become a more significant part of testing protocols as licensing bodies seek ways to assess competencies other than the maintenance of medical knowledge.”
Spare Keys – other take home points for clinician educator
This is a great example (that can be readily replicated) of how to perform a systematic search of the literature. I would have loved more details on how to perform a Google Scholar search, as only 62 articles were identified. When I did a quick search, I found 714k. If anyone has an authoritative / consensus approach, I would love to hear about it
Access KeyLIME podcast archives here
Check us out on iTunes!
