#CBME: Key Principles in Programmatic Assessment? Part 2

(  This is the concluding high level summary of  the current literature on programmatic assessment from an very early career Clinician Educator – medical student – as part of her literature review for a medical education elective. Part 1 is here.  -Jonathan (@sherbino))

By Larissa Hatin

Key Principle # 4: Rich Data

Qualitative assessment is more than merely using words instead of numbers on a rating scale. The use of narrative, real time information provides the learner with rich data to inform a tailored learning plan, identifies strengths and weaknesses and also helps guide residents in self-reflective behaviour. Narrative comments are more instructive than a 5.7 out of 7. Narrative descriptions provide the level of specificity needed by the learner to make improvements and develop specific learning.

The McMAP system highlights this principle by making narrative feedback a mandatory aspect. Resident focus groups agree that this new program increased the amount of formative feedback given from staff physicians. As well, the McMAP data report is reviewed at the end of each month-long rotation and provides a report to the rotation preceptor. This report is discussed with the resident and results in tailored advice for the learner that allows them to continue to improve with a specific task or focus in mind.

Key Principle # 5: Engage the trainee

Assessment programs should actively engage the resident, not only so that they know the areas they might be deficient, but also to give them the opportunity to collaborate on their ongoing learning. This also helps to develop in the learner the skill of guide self-assessment, a key feature for future practice and ongoing learning.

The McMAP program requires each resident to perform a self-assessment of their collated month’s-end data submitted to their portfolio.  This self-assessment is reviewed by a faculty advisor, facilitating a discussion of perceived and observed needs.

Key Principle # 6: Leverage the Wisdom of Crowds

“No single individual should make judgements about the competence of a trainee in isolation, especially for summative decisions”⁶. Science (e.g. peer review) and the judicial system (e.g. jury) is built on the process of collective decision making.  Likewise, high stakes decisions about learner advancement should leverage the process of group decision making.

Like many other residency programs, the McMAP system has a competence committee that meets every two months to review resident portfolios. Resident, faculty and community members are represented.  The purpose of the committee is to identify trends in resident performance and make recommendations about trainee advancement or remediation to the residency training committee.  The review, documentation and recommendation for learner advancement is informed by aggregated information using a systematic and transparent process. In this way, high-stakes decisions are made using large amounts of information on the residence performance with the goal of assisting in the development of a resident’s competence.

In summary, there are six key principles that inform programmatic assessment. But the fundamental concept is simple: assessment for learning. Programs that are constructed with the goal of making every assessment valuable for the learner will result in better outcomes and better care. As a medical student this is exactly what I want to hear

References

1. Frank JR, Snell LS, Cate TO, Holmboe ES, Carraccio C, Swing SR et al. (2010). Competency-based medical education: theory to practice. Med Teach 32: 638-645.
2. Royal College of Physicians and Surgeons of Ontario. (2017). About Competence by Design. Retrieved from: http://www.royalcollege.ca/rcsite/cbd/competence-by-design-cbd-e
3. Schuwirth LW, Van der Vleuten CP. (2011). Programmatic assessment: From assessment of learning to assessment for learning. Med Teach 33: 478-485.
4. Van der Vleuten CP, Schuwirth LW, Driessen EW, Dijkstra J, Tigelaar D, Baartman LK, van Tartwijk J. (2012). A model for programmatic assessment fit for purpose. Med Teach 34: 205-214.
5. Dijkstra J, Van der Vleuten CP, Schuwirth LW. (2010). A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract 15: 379-393.
6. Holmboe ES, Sherbino J., Long DM, Swing SR, Frank JR. (2010). The role of assessment in competency-based medical education. Med Teach 32: 676-682.
7. Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, et al. (2011). Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach 33: 206-214.
8. Van der Vleuten CP, Schuwirth LW, Driessen EW, Govaerts MJ, Heeneman S. (2015). Twelve tips for programmatic assessment. Med Teach 37: 641-646.
9. Chan T, Sherbino J, McMAP collaborators. (2015). The McMaster Modular Assessment Program (McMAP): a theoretically grounded work-based assessment system for an Emergency Medicine residency program. Acad Med 90: 900-905.