As part of the ALiEM Faculty Incubator program, teams of 2-4 incubator participants authored a primer on a key education theory, linking the abstract to practical scenarios. For the third year, these posts are being serialized on our blog, as a joint collaboration with ALiEM. You can view the first e-book here – the second is nearing completion and will soon be released. You can view all the blog posts from series 1 and 2 here.
The ALiEM team loves hearing your feedback prior to publication. No comment is too big or too small and they will be used to refine each primer prior to the eBook publication. (note: the blog posts themselves will remain unchanged)
This is the tenth and FINAL post of Volume 3. You can find the previous posts here: Bolman and Deal’s Four-Frame Model; Validity; Mayer’s Cognitive Theory of Multimedia Learning; The Kirkpatrick Model: Four Levels of Learning Evaluation; Curriculum Development; Programmatic Assessment; Realist Evaluation;Kotter’s Stages of Change; and, Mastery Learning. Please share your feedback with the ALiEM team! 🙂
Self-Directed Assessment Seeking
Authors: Nilantha Lenora (@SriLankanERDoc); Laila Abubshait (@layla_abubshait); Manu Ayyan (@Drmanuayyan)
Editor: Benjamin H. Schnapp (@schnappadap)
Main Authors or Originators: Kevin Eva; Glenn Regehr
Other important authors or works:
- Boud D. Avoiding the traps: seeking good practice in the use of self-assessment and reflection in professional courses. Soc Work Educ. 1999;18(2):121-132.
- Boud D. Enhancing Learning Through Self-Assessment. Routledge; 1995.
Sargeant J, Mann K, van der Vleuten C, Metsemakers J. “Directed” self-assessment: - Practice and feedback within a social context: J Contin Educ Health Prof. 24/2008;28(1):47-54.
- Sargeant J, Armson H, Chesluk B, et al. The processes and dimensions of informed self-assessment: a conceptual model. Acad Med. 2010;85(7):1212-1220.
- Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psychol. 1999;77(6):1121-1134.
Part 1: The Hook
Adam is a PGY1 in a four year Emergency Medicine (EM) residency program. He ranked near the bottom of his class in medical school and had below average USMLE scores. Many of his recent shift evaluations have expressed concerns about his clinical knowledge and skills. Adam rationalized these poor shift evaluations as flawed with a variety of justifications, including having difficult patients who were “poor historians,” feeling sleep deprived, and just plain “bad luck” with unusual presentations of diseases. He was called to attend his first meeting with his program director (PD) six months into his intern year and asked to complete a self-evaluation prior to the meeting.
Adam enjoys his current EM training experience. Adam feels extremely confident in his medical knowledge, procedural skills, and ED workups. He also feels he is above average and one of the top residents in his class. He sees a high number of patients per hour and is proactive in seeking procedures. In accordance with this positive self-outlook on his abilities, he feels his self-assessment will be more accurate than faculty evaluations and should describe his many strengths as a resident. As such, he filled out the self-assessment form with a very positive description of his abilities.
At his meeting, the program director told him that his self-assessment is very different from the faculty evaluations of him. Specifically, faculty evaluations mirrored his shift evaluations and expressed concern regarding his below average clinical knowledge and skills.
Adam is extremely perplexed at the discrepancy and tells his PD: “But how? I don’t understand. I know I’m better than my peers and I feel I’m one of the top interns in my class!”
Part 2: The Meat
Overview
With respect to health professions education, Eva and Regehr define traditional self-assessment as a “personal, unguided reflection on performance for the purposes of generating an individually-derived summary of one’s own level of knowledge, skill, and understanding in a particular area.”1 Individuals themselves are the source of information and look inward to generate an assessment of their own knowledge and abilities. Self-assessment can be further categorized into three different perspectives:2
● Summative (assessing an overall performance or one’s abilities in general)
● Predictive (assessing one’s ability to perform in new situations)
● Concurrent (assessing ongoing performance while conducting an activity)
Much of the health professions literature has adhered to this traditional definition when referring to and studying self-assessment. Traditional self-assessment can take on many forms including formal self-assessment questionnaires, checklists, journal or diary entries, patient chart reviews, or reviews of videotaped performances as common examples.
Within education theory, self-assessment is key to diagnosing one’s learning needs, an early step in the process of self-directed learning as described by Knowles.3 Self-assessment can also complement learning as suggested by constructivism theory, with a self-assessment of existing knowledge serving as a foundation for new information that can be built upon and as an aid in integrating new knowledge.4
Additionally, professional societies rely on accurate self-assessment as an essential premise for self-regulation in health professionals.1 An individual is expected to take responsibility and assess their own knowledge base and clinical practice, identify areas for improvement, pursue educational opportunities addressing these areas, and then put this new knowledge into action as performance improvement.5 Stemming from this, self-assessment has been incorporated into undergraduate and graduate medical education,6 continuous professional development,7 and maintenance of certification programs for health professionals.8
Background
Intuitively, self-assessments should have the most fidelity and be the most accurate representation of our current knowledge, skills, and abilities. We have more information about ourselves than those external to us and therefore should create the most accurate self-assessments of ourselves as well.9
The cumulative literature on self-assessment tells a different story however, one that shows the accuracy of traditional self-assessment to be quite poor when compared to external standards.
In his review of the self-assessment literature in medical education from 1970 to 1991, Gordon found that the validity of self-assessment was low to moderate when compared with external criteria. He also theorized that it is a skill that can be improved and should be integrated with other sources of evaluative data to improve its validity and accuracy.10 Davis, et al. arrived at a similar conclusion, finding physician self-assessment inaccurate when compared with external observations as a reference standard.11
A possible explanation for the inaccuracies in self-assessment arose from landmark research conducted by Kruger and Dunning.12 In a series of experiments, they demonstrated that the skills required to perform well in a given area are the same skills needed to accurately assess one’s performance in that area. In other words, by lacking the requisite skills that define competence, the “unskilled” are unable to identify the presence or absence of those same skills in themselves, making them the least accurate when it comes to self-assessment. Other papers have replicated similar findings, highlighting a flaw in using self-assessment alone as the sole determinant for self-directed learning efforts.13
Around the same time, Boud offered new perspective on the traditional notion of self-assessment. Instead of relying on the individual alone to conduct a self-assessment, Boud proposed that self-assessment should not be an unguided task left to the individual by themself. He suggested that self-assessment should also incorporate external information from peers, instructors, and other sources of information outside the individual to guide and add validity to the self-assessment process.14,15
Eva and Regehr dubbed Boud’s conception of self-assessment “self-directed assessment seeking,” defined as the “pedagogical process of explicitly seeking external sources of information for formative and summative assessments of one’s current level of performance and practice improvement”.1
Based on consistent evidence demonstrating that traditional self-assessment has many limitations, Eva and Regehr suggest that seeking feedback and incorporating information from external sources is more valuable for directing performance improvement than relying on an individual’s assessment alone.1
Modern takes or advances
Sargeant, et al. proposed a conceptual model of informed self-assessment with three main components. These components include:
1) Gathering both internal and external sources of information
2) Integrating the two sources of information with the help of a facilitator
3) Responding to the information by the individual – either rejecting the results or accepting and applying them towards performance improvement.16
Different types of external data and strategies exist to inform the modern notion of self-assessment and overcome the cognitive biases limiting its accuracy.
External feedback, for example, can inform, add value, and improve the accuracy of self assessment once incorporated with it. Learners should be actively seeking feedback from senior colleagues, supervisors, program directors, teaching faculty, and peers to be included with their self-assessment. Eva and Regehr argue that we should focus our efforts towards obtaining trustworthy feedback, improving our abilities to act on this feedback without feeling threatened, and teaching others how to share feedback in a way that will improve it’s acceptance by the recipients.1
Multi-source feedback is feedback compiled from different types of individuals observing and offering assessments on different aspects of a learner’s performance. It is extremely useful for areas that are difficult to gauge via self-assessment, including dimensions such as professionalism, communication, and interpersonal relationships.17 For example, to assess resident professionalism, a program director might obtain data from nurses, techs, and physicians from other specialties. The aggregation of several sources improves the reliability and validity of the information obtained.
Validation of self-assessment using external standards may also be a valuable tool. Clinical guidelines, consensus-based performance standards, and bench marking with physicians of similar practice profiles can serve as a “reality check” to improve the effectiveness of self-assessments as well.18
Guided reflection with the help of a trusted facilitator or mentor can also function as a bridge between one’s self-perception and external sources of feedback, even serving to reconcile the two when there is discordance between them.19
Other examples of where this theory might apply in both the classroom & clinical setting
In post-graduate medical education, the ACGME core competency of Practice-Based Learning and Improvement requires residents to demonstrate self-directed learning.20 Self-assessment can play a role in this by identifying a resident’s professional strengths and weaknesses in order to guide such self-directed learning efforts.
The six ACGME core competencies21 or ACGME Milestones22 themselves can serve as the organizing structure for a resident’s self-assessment. Each could serve as an accepted external standard to improve the validity of their self-assessment. A trusted faculty advisor can then add more accuracy to the assessment by incorporating external data such as multi-source feedback and faculty evaluations via guided reflection with the resident.
An example of self-assessment applied within the clinical realm exists with the Maintenance of Certification (MOC) program for the Royal College of Physicians and Surgeons in Canada. The MOC program rewards physicians with continuing professional development credits for self-assessment activities that assess clinical knowledge and performance against objective measures. It also encourages reflection on performance at the individual level as well as with peers.8
Another example exists in the United Kingdom’s National Health Service concept of ‘appraisal.’ Here, an external appraiser (i.e. a mentor or senior clinician) reviews an individual physician’s self-assessment and portfolio with them and can also incorporate external data such as multi-source feedback.11, 23
Annotated Bibliography of Key Papers
Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Perspect Soc Psychol 1999; 77:1121-34.12
This landmark paper used a series of experiments to highlight the limitations of self-assessment. Individuals who performed poorly were also the least able to accurately self-assess their performance in these experiments. Their findings suggest that knowledge and competence within a given area (or lack thereof) also defines one’s ability to accurately assess competence in that area as well.
Eva KW, Regehr G. “I’ll never play professional football” and other fallacies of self-assessment. J Contin Educ Health Prof. 2008 Winter;28(1):14-19.1
This article draws distinctions between self-assessment as an ability, self-directed assessment seeking as a pedagogical strategy, and self-monitoring, which is a moment-by-moment awareness of the likelihood that one maintains the skill and knowledge to act in a particular situation.
Regehr G, Eva K. Self-assessment, self-direction, and the self-regulating professional. Clin Orthop Relat Res. 2006 Aug; 449:34-38.5
In this paper, the authors review the literature in adult education, medical education, and cognitive psychology to highlight two critical flaws in self-assessment and self-regulation. They challenge the assumption that the process of self-assessment as conceptualized in this model can lead to the identification of gaps in skill or knowledge.
Sargeant J, Armson H, Chesluk B, et al. The processes and dimensions of informed self-assessment: a conceptual model. Acad Med 2010; 85: 1212-20.16
This qualitative study of self-assessment used focus groups to create a multidimensional conceptual model of informed self-assessment. This model integrates internal sources of information (i.e. an individual self-assessment) with external sources (i.e. evaluation forms), and finishes with deliberate reflection by learners on the integrated assessment with the guidance of a trusted mentor.
Colthart I, Bagnall G, Evans A, Allbutt H, Haig A, Illing J, McKinstry B. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide No. 10. Medical Teacher 30(2): 124–45.24
This systematic review suggests that the accuracy of self-assessment can be enhanced by feedback, particularly video and verbal, and by providing explicit assessment criteria and benchmarking guidance. Self-assessment needs to be used as one tool amongst others to provide a more complete appraisal of competence in healthcare practice.
Limitations
Many poor performers believe at baseline that they are above average in comparison to their peers, a superiority bias also known as the Lake Wobegon effect. Self-assessment abilities can also vary depending on the context, with our abilities more accurate in subject areas we have skills in and less accurate in subjects we are unfamiliar with.2
Certain areas such as interpersonal skills, communication skills, and professionalism are inherently difficult to accurately assess by oneself and are more conducive to assessment via a method like multisource feedback.17
Another limitation is the assumption within self-assessment theory that once a knowledge or performance gap is brought to a learner’s attention they will readily accept this gap and be motivated to seek out opportunities to address it. Individuals can have several cognitive mechanisms functioning as a “psychological immune system” serving to fight off, rationalize, and dismiss unfavorable assessments while maintaining a positive and optimistic outlook on one’s abilities, limiting motivation to address performance gaps.2,25
Part 3: The Denouement
Adam’s glowing self-assessment in the face of multiple sources of negative feedback is an example of the Dunning-Kruger effect. His lack of competence limits the accuracy of his efforts to assess his own abilities as a resident. In fact, he overestimates his abilities and feels he is above average relative to his peers.
The discordance between Adam’s assessment of his abilities and those around him created a layer of cognitive dissonance. He displayed many of the psychological defense mechanisms that inherently protect us from negative information and he proceeded to deny, dismiss, and rationalize many of the objective external sources of data he was presented with.
To add more validity to Adam’s self-assessment, his PD decided that Adam’s new self-assessment could be organized around the ACGME Milestones as an accepted external standard to serve as a benchmark. To improve the accuracy of his self-assessment, it could also be informed by external data as well, such as multisource feedback from nurses and techs in the department and faculty evaluations.
Adam’s PD also decided that a trusted faculty advisor who can help Adam reconcile the gap between his self-assessment and the external sources of data would be essential. The advisor presented the external data to Adam in a non-threatening manner while trying to maintain Adam’s self-efficacy. Guided reflection led by the advisor served as an important bridge between Adam’s self-assessment and the external sources of data. With help, Adam identified learning goals and an action plan for remediation. Adam followed this plan across his PGY-2 year and did indeed rise to become one of the top residents in his class by the end of residency.
PLEASE ADD YOUR PEER REVIEW IN THE COMMENTS SECTION BELOW
References
1. Eva KW, Regehr G. “I’ll never play professional football” and other fallacies of self-assessment. J Contin Educ Health Prof. 2008;28(1):14-19.
2. Eva KW, Regehr G. Self-Assessment in the Health Professions: A Reformulation and Research Agenda. Acad Med. 2005;80(10):S46.
3. Manning G. Self-directed learning: A key component of adult learning theory. Business and Public Administration Studies. 2007;2(2):104.
4. Bourke R, Mentis M. Self-assessment as a lens for learning. In: ResearchGate. ; 2007:322.
5. Regehr G, Eva K. Self-assessment, Self-direction, and the Self-regulating Professional: Clin Orthop Relat Res. 05/2006;PAP.
6. Hildebrand C, Trowbridge E, Roach MA, Sullivan AG, Broman AT, Vogelman B. Resident self-assessment and self-reflection: University of Wisconsin-Madison’s Five-Year Study. J Gen Intern Med. 2009;24(3):361-365.
7. Duffy FD, Lynn LA, Didura H, et al. Self-assessment of practice performance: development of the ABIM Practice Improvement Module (PIM). J Contin Educ Health Prof. 2008;28(1):38-46.
8. Silver I, Campbell C, Marlow B, Sargeant J. Self-assessment and continuing professional development: the Canadian perspective. J Contin Educ Health Prof. 2008;28(1):25-31.
9. Eva KW, Regehr G. Effective feedback for maintenance of competence: from data delivery to trusting dialogues. CMAJ. 2013;185(6):463-464.
10. Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med. 1991;66(12):762-769.
11. Davis DA, Mazmanian PE, Fordis M, Harrison RV, Thorpe KE, Perrier L. Accuracy of Physician Self-assessment Compared With Observed Measures of Competence: A Systematic Review. JAMA. 2006;296(9):1094-1102.
12. Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psychol. 1999;77(6):1121-1134.
13. Sadosty AT, Bellolio MF, Laack TA, Luke A, Weaver A, Goyal DG. Simulation-based emergency medicine resident self-assessment. J Emerg Med. 2011;41(6):679-685.
14. Boud D. Enhancing Learning Through Self-Assessment. Routledge; 1995.
15. Boud D. Avoiding the traps: seeking good practice in the use of self assessment and reflection in professional courses. Soc Work Educ. 1999;18(2):121-132.
16. Sargeant J, Armson H, Chesluk B, et al. The processes and dimensions of informed self-assessment: a conceptual model. Acad Med. 2010;85(7):1212-1220.
17. Donnon T, Al Ansari A, Al Alawi S, Violato C. The reliability, validity, and feasibility of multisource feedback physician assessment: a systematic review. Acad Med. 2014;89(3):511-516.
18. Galbraith RM, Hawkins RE, Holmboe ES. Making self-assessment more effective. J Contin Educ Health Prof. 2008;28(1):20-24.
19. Sargeant J, Mann K, van der Vleuten C, Metsemakers J. “Directed” self-assessment: Practice and feedback within a social context: J Contin Educ Health Prof. 24/2008;28(1):47-54.
20. NEJM Knowledge+ Team. Practice-based Learning and Improvement: ACGME Core Competencies. NEJM Knowledge+. https://knowledgeplus.nejm.org/blog/practice-based-learning-and-improvement/. Published July 28, 2016. Accessed October 16, 2018.
21. Li S-TT, Paterniti DA, Tancredi DJ, et al. Resident Self-Assessment and Learning Goal Development: Evaluation of Resident-Reported Competence and Future Goals. Acad Pediatr. 2015;15(4):367-373.
22. Meier AH, Gruessner A, Cooney RN. Using the ACGME Milestones for Resident Self-Evaluation and Faculty Engagement. J Surg Educ. 2016;73(6):e150-e157.
23. Conlon M. Appraisal: the catalyst of personal development. BMJ. 2003;327(7411):389-391.
24. Colthart I, Bagnall G, Evans A, et al. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Med Teach. 2008;30(2):124-145.
25. Gilbert DT, Wilson TD. Miswanting: Some Problems in the Forecasting of Future Affective States. In: Thinking and Feeling: The Role of Affect in Social Cognition. Cambridge University Press; 2001:178-197.