Jon introduces a paper that looks to fidelity of implementation over the lifecycle of an innovation by introducing an alternative model for evaluating the longevity of an innovation: the Eco-Normalization theory.
————————————————————————–
KeyLIME Session 331
Reference
Hamza, D.,Regehr. F. 2021. Eco-Normalization: Evaluating the Longevity of an Innovation in Context. Academic medicine: journal of the Association of American Medical Colleges. August 3. Online ahead of print.
Reviewer
Jon Sherbino (@sherbino)
Background
Program evaluation is one of those unspoken words in HPE. It’s kind of like saying Voldemort. Everyone acknowledges it’s importance, but few scholars or scientists want to commit to it? Why? Perhaps evaluating the effectiveness of an innovation is less satisfying that developing an implementing the innovation. More likely, I believe it reflects the complexity of program evaluation.
What is an example of a successful or challenging program evaluation you have led or contributed too?
Program evaluation has advanced though. Early in HPE the focus was on the output of the innovation, using an experiment to determine effectiveness. This was replaced by a focus on process, which was challenging as theories guiding such evaluation lacked sufficient data about the innovation to draw conclusions. Most recently the magic word in program evaluation is “fidelity”, where the inner workings of an innovation are monitored within their real-world context of application.
Fidelity of implementation can suffer similar fatal flaws as its predecessors if attention to mechanistic measurements (e.g. completion of work-place assessments) is excessive. Such an approach may fail to reveal adoption of underlying values these measurements are meant to represent (e.g. adoption of a feedback culture). However, the biggest challenge facing current approaches to fidelity of implementation relate to the use of a snapshot (or a biopsy in time) of the effectiveness of an implementation. Such approaches ignore the trajectory or course of the innovation. This can be problematic if the cross-section used for evaluation occurs early when resources and engagement are still high. Recall that up to 75% of innovations are abandoned or devolve over time.
Enter Hamza and Regehr, who tackle fidelity of implementation over the lifecycle of an innovation.
Purpose
“The purpose of this paper is to develop an alternative model for evaluating the longevity of an innovation, borrowing from and building on the theories and models of implementation science.”
Key Points on the Methods
SA critical review was conducted, including a literature review and grey literature review via Google Scholar. Models from implementation science were used to organized themes and factors to the new model the authors developed.
Key Outcomes
Three existing models were identified and integrated into the proposed new Eco-Normalization theory.
- Normalization Process Theory emphasizes the coherence (shared understanding of why and how required to change) and participation required between individuals and groups as they engage in the change required of an innovation.
- The Consolidated Framework for Implementation Research (my fav!) looks at the ecosystem in which the innovation must invite change, including the characteristics of the innovation, the outer setting, the inner setting, individual characteristics and the process of implementation.
- Reflexive Monitoring in Action promotes collective reflection of prior experiences to gain learnings that promote long-term systems change to achieve the goals of the innovation.(it is akin to PDSA, but with a more radical goal of change and longer time horizon than the typical incremental change of PDSA cycles.)
Eco-Normalization
• The model seeks to evaluate the normalization of values and practices, not at the individual level, but at the ecosystem (e.g. merge NPT and CFIR)
• Primary-level interactions involve the design of the innovation, local aspirations of stakeholders and aspirations of the system.
• Secondary-level interactions include the compatibility of the innovation with the practices of the people doing the work, the ways in which the system impacts stakeholders, and how the features of the innovation fit with the features of the system.
This leads to the 6 questions of Eco-Normalization program evaluation.
Primary
- Does the innovation align with the grand aspirations of change?
- Do the system goals align with the grand aspirations of change?
- Do the stakeholders’ local aspirations align with the grand aspirations of change?
Secondary - Does the innovation interact with the system in a way that will lead to the aspirations of change?
- Does the innovation evoke meaning to actions and agency of the people doing the work that will lead to aspirations of change?
- Does the system support the people doing the work in ways that will lead to aspirations of change?
Key Conclusions
The authors conclude…
“ [The Econ-Normalization model] frames the innovation as a dynamic rather than static set of practices, it treats the system and the workers on the ground as agentive partners with their own goals and agendas, and it acknowledges the interactions between all these factors as the ecosystem seeks its “new normal” following the perturbation of change.”
Spare Keys – Other take home points for Clinician Educators
A critical review is an emerging and authentic type of synthesis. Grant and Booth’s (2009) typology of syntheses defines a critical review as one that critically evaluates the literature to identify significant items, synthesizes findings using a conceptual framework, and produces an analysis that typically results in an hypothesis or model.
Access KeyLIME podcast archives here
The views and opinions expressed in this post and podcast episode are those of the host(s) and do not necessarily reflect the official policy or position of The Royal College of Physicians and Surgeons of Canada. For more details on our site disclaimers, please see our ‘About’ page