Search

Introduction: People experiencing health-related social needs (HRSNs), such as transportation insecurity, are less likely to undergo preventive health screenings. They are more likely to have worse health outcomes overall, including a higher rate of late-stage cancer diagnoses. If primary care clinicians are aware of HRSNs, they can tailor preventive care, including cancer screening approaches. Accordingly, recent guidelines recommend that clinicians “adjust” care based on HRSNs. This study assessed the level of clinician awareness of patient-reported HRSNs and congruence between clinician perception and patient-reported HRSNs. Methods: We surveyed patients aged 50 to 85 years and their clinicians in 3 primary care clinics that routinely screen patients for HRSNs. Patients and clinicians reported the presence/absence of 6 HRSNs, including food, transportation, housing and financial insecurity for medications/healthcare, financial insecurity for utilities, and social isolation. Kappa statistics assessed the concordance of reported HRSNs between patients and clinicians. Results: Across 237 paired patient-clinician surveys, mean patient age was 65 years, and 62% and 13% of patients were female and Latinx/Hispanic, respectively. Concordance between clinician- and patient-reported HRSNs varied by HRSN, with the lowest agreement for food insecurity (kappa = .08; 95% CI: 0.00, 0.17; P = .01) and highest agreement for transportation insecurity (kappa = .39; 95% CI: 0.18, 0.59; P

Abstract For the fields of implementation science and health equity, understanding and being responsive to local contexts is of utmost importance to better inform the development, implementation, and evaluation of healthcare and public health interventions to increase their uptake and sustainment. Contexts are multi-level and include political, historical, economic, and social factors that influence health, as well as organizational characteristics, reflecting the richness of members’ views, resources, values, and needs. Poor alignment between solutions and those contextual characteristics could have an impact on inequities. The PRISM (Practical Robust Implementation and Sustainability Model) is a context-based implementation science framework that incorporates RE-AIM outcomes (Reach, Effectiveness, Adoption, Implementation, Maintenance) and offers guidance to researchers, practitioners, and their patient and community partners on how to conceptualize, assess, and address contextual domains with a focus on health equity. Drawing from systems thinking, participatory engagement, and health equity principles, this commentary expands on previous work to 1) offer a novel perspective on how to align an intervention’s core functions and forms with the PRISM’s contextual domains, and 2) foster an ongoing and iterative engagement process with diverse partners throughout the research and practice process using a co-creation approach. We recommend intervention-to-context alignment through iterative cycles. To that end, we present the RE-AIM Framework’s ‘outcomes cascade’ to illustrate touch points of opportunity and gaps within and across each of the five RE-AIM outcomes to illustrate ‘where things go wrong’. We present a case study to illustrate and offer recommendations for research and practice efforts to increase contextual responsiveness, and enhance alignment with context before, during, and after implementation efforts and to ensure equity is being addressed. We strive to make a conceptual contribution to advance the field of pragmatic research and implementation of evidence-based practices through the application of the contextually-based PRISM framework with a focus on health equity.

Abstract Background The field of implementation science was developed to address the significant time delay between establishing an evidence-based practice and its widespread use. Although implementation science has contributed much toward bridging this gap, the evidence-to-practice chasm remains a challenge. There are some key aspects of implementation science in which advances are needed, including speed and assessing causality and mechanisms. The increasing availability of artificial intelligence applications offers opportunities to help address specific issues faced by the field of implementation science and expand its methods. Main text This paper discusses the many ways artificial intelligence can address key challenges in applying implementation science methods while also considering potential pitfalls to the use of artificial intelligence. We answer the questions of “why” the field of implementation science should consider artificial intelligence, for “what” (the purpose and methods), and the “what” (consequences and challenges). We describe specific ways artificial intelligence can address implementation science challenges related to (1) speed, (2) sustainability, (3) equity, (4) generalizability, (5) assessing context and context-outcome relationships, and (6) assessing causality and mechanisms. Examples are provided from global health systems, public health, and precision health that illustrate both potential advantages and hazards of integrating artificial intelligence applications into implementation science methods. We conclude by providing recommendations and resources for implementation researchers and practitioners to leverage artificial intelligence in their work responsibly. Conclusions Artificial intelligence holds promise to advance implementation science methods (“why”) and accelerate its goals of closing the evidence-to-practice gap (“purpose”). However, evaluation of artificial intelligence’s potential unintended consequences must be considered and proactively monitored. Given the technical nature of artificial intelligence applications as well as their potential impact on the field, transdisciplinary collaboration is needed and may suggest the need for a subset of implementation scientists cross-trained in both fields to ensure artificial intelligence is used optimally and ethically.

Abstract Background This pilot study examined the preliminary effectiveness of the PREVENT digital intervention that supports health care teams in delivering health behavior counseling on cancer survivors’ motivation to change behavior, their physical activity and food intake behaviors, and cardiovascular health (CVH). Methods Clinicians (physicians, nurse practitioners) at three urban cancer survivorship clinics were trained to use PREVENT. Patients were randomized to the PREVENT intervention or a wait-list routine care control group. Eligibility criteria for patients included: between ages 12–39, overweight or obese, were at least 6-months post-active cancer treatment, and had sufficient English proficiency. Results Fifty-five participants were enrolled; 27 were randomized to the PREVENT intervention and 28 to wait-list routine care control. The majority of the participants (82%) identified as non-Hispanic white, with an average age of 19.8 (SD ± 5.2) years. Patients that received the PREVENT intervention had greater increases in their self-efficacy, vigorous activity and number of food recommendations met than those who received routine clinical care. Changes in willingness, knowledge, and CVH outcomes were not significant. Conclusions The PREVENT digital intervention may provide improvements in preventive behaviors among AYA cancer survivors by supporting care teams with delivering evidence-based, tailored behavior change recommendations and resources to support patient health. Trial registration This trial ( NCT04623190 ) was registered on 11/02/2022.

Abstract Background Implementation science and health services outcomes research each focus on many constructs that are likely interrelated. Both fields would be informed by increased understanding of these relationships. However, there has been little to no investigation of the relationships between implementation outcomes and service outcomes, despite general acknowledgement that both types of outcomes are important in the pathway to individual and population health outcomes. Given the lack of objective data about the links between implementation and service outcomes, an initial step in elucidating these relationships is to assess perceptions of these relationships among researchers and practitioners in relevant fields. The purpose of this paper is to assess perceived relationships between Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework outcomes and service outcomes, testing five a priori hypotheses about which perceived relationships may be strongest. Methods A cross-sectional online survey was administered to a convenience sample of implementation scientists, health services researchers, and public health and medical practitioners from a variety of settings. Respondents provided information on their discipline, training, practice and research settings, and levels of experience in health service outcomes research, implementation science, and the RE-AIM framework. Next, they rated perceived relationships between RE-AIM and service outcomes. Repeated measures analysis of variance were used to test a priori hypotheses. Exploratory analyses assessed potential differences in mean ratings across groups of respondents categorized by discipline, setting, and levels of implementation science, health services, and RE-AIM experience. Results Surveys were completed by 259 respondents, most of whom were employed in academic and medical settings. The majority were doctoral-level researchers and educators or physicians. Reported levels of experience with implementation research, health services research, and the RE-AIM framework varied. The strongest perceived relationships overall were between Implementation/Fidelity and Effectiveness (as a service outcome); Maintenance and Efficiency; Reach and Equity; Adoption and Equity; Implementation/Adaptation and Patient-Centeredness; Adoption and Patient-Centeredness; and Implementation/Fidelity and Safety. All but one of the a priori hypotheses were supported. No significant differences in ratings of perceived relationships were observed among subgroups of respondents. Conclusions This study is an initial step in developing conceptual understanding of the links between implementation outcomes, health services outcomes, and health outcomes. Our findings on perceived relationships between RE-AIM and services outcomes suggest some areas of focus and identify several areas for future research to advance both implementation science and health services research toward common goals of improving health outcomes.

Abstract Background To increase uptake of implementation science (IS) methods by researchers and implementers, many have called for ways to make it more accessible and intuitive. The purpose of this paper is to describe the iPRISM webtool (Iterative, Practical, Robust Implementation and Sustainability Model) and how this interactive tool operationalizes PRISM to assess and guide a program’s (a) alignment with context, (b) progress on pragmatic outcomes, (c) potential adaptations, and (d) future sustainability across the stages of the implementation lifecycle. Methods We used an iterative human-centered design process to develop the iPRISM webtool. Results We conducted user-testing with 28 potential individual and team-based users who were English and Spanish speaking from diverse settings in various stages of implementing different types of programs. Users provided input on all aspects of the webtool including its purpose, content, assessment items, visual feedback displays, navigation, and potential application. Participants generally expressed interest in using the webtool and high likelihood of recommending it to others. The iPRISM webtool guides English and Spanish-speaking users through the process of iteratively applying PRISM across the lifecycle of a program to facilitate systematic assessment and alignment with context. The webtool summarizes assessment responses in graphical and tabular displays and then guides users to develop feasible and impactful adaptations and corresponding action plans. Equity considerations are integrated throughout. Conclusions The iPRISM webtool can intuitively guide individuals and teams from diverse settings through the process of using IS methods to iteratively assess and adapt different types of programs to align with the context across the implementation lifecycle. Future research and application will continue to develop and evaluate this IS resource.

Abstract Background Logic models map the short-term and long-term outcomes that are expected to occur with a program, and thus are an essential tool for evaluation. Funding agencies, especially in the United States (US), have encouraged the use of logic models among their grantees. They also use logic models to clarify expectations for their own funding initiatives. It is increasingly recognized that logic models should be developed through a participatory approach which allows input from those who carry out the program being evaluated. While there are many positive examples of participatory logic modeling, funders have generally not engaged grantees in developing the logic model associated with their own initiatives. This article describes an instance where a US funder of a multi-site initiative fully engaged the funded organizations in developing the initiative logic model. The focus of the case study is Implementation Science Centers in Cancer Control (ISC3), a multi-year initiative funded by the National Cancer Institute. Methods The reflective case study was collectively constructed by representatives of the seven centers funded under ISC3. Members of the Cross-Center Evaluation (CCE) Work Group jointly articulated the process through which the logic model was developed and refined. Individual Work Group members contributed descriptions of how their respective centers reviewed and used the logic model. Cross-cutting themes and lessons emerged through CCE Work Group meetings and the writing process. Results The initial logic model for ISC3 changed in significant ways as a result of the input of the funded groups. Authentic participation in the development of the logic model led to strong buy-in among the centers, as evidenced by their utilization. The centers shifted both their evaluation design and their programmatic strategy to better accommodate the expectations reflected in the initiative logic model. Conclusions The ISC3 case study demonstrates how participatory logic modeling can be mutually beneficial to funders, grantees and evaluators of multi-site initiatives. Funded groups have important insights about what is feasible and what will be required to achieve the initiative’s stated objectives. They can also help identify the contextual factors that either inhibit or facilitate success, which can then be incorporated into both the logic model and the evaluation design. In addition, when grantees co-develop the logic model, they have a better understanding and appreciation of the funder’s expectations and thus are better positioned to meet those expectations.

Abstract Introduction: The slow adoption of evidence-based interventions reflects gaps in effective dissemination of research evidence. Existing studies examining designing for dissemination (D4D), a process that ensures interventions and implementation strategies consider adopters’ contexts, have focused primarily on researchers, with limited perspectives of practitioners. To address these gaps, this study examined D4D practice among public health and clinical practitioners in the USA. Methods: We conducted a cross-sectional study among public health and primary care practitioners in April to June 2022 (analyzed in July 2022 to December 2022). Both groups were recruited through national-level rosters. The survey was informed by previous D4D studies and pretested using cognitive interviewing. Results: Among 577 respondents, 45% were public health and 55% primary care practitioners, with an overall survey response rate of 5.5%. The most commonly ranked sources of research evidence were email announcements for public health practitioners (43.7%) and reading academic journals for clinical practitioners (37.9%). Practitioners used research findings to promote health equity (67%) and evaluate programs/services (66%). A higher proportion of clinical compared to public health practitioners strongly agreed/agreed that within their work setting they had adequate financial resources (36% vs. 23%, p < 0.001) and adequate staffing (36% vs. 24%, p = 0.001) to implement research findings. Only 20% of all practitioners reported having a designated individual or team responsible for finding and disseminating research evidence. Conclusions: Addressing both individual and modifiable barriers, including organizational capacity to access and use research evidence, may better align the efforts of researchers with priorities and resources of practitioners.

Abstract Background Despite lung cancer being a leading cause of death in the United States and lung cancer screening (LCS) being a recommended service, many patients eligible for screening do not receive it. Research is needed to understand the challenges with implementing LCS in different settings. This study investigated multiple practice members and patient perspectives impacting rural primary care practices related to LCS uptake by eligible patients. Methods This qualitative study involved primary care practice members in multiple roles (clinicians n = 9, clinical staff n = 12 and administrators n = 5) and their patients (n = 19) from 9 practices including federally qualified and rural health centers (n = 3), health system owned (n = 4) and private practices (n = 2). Interviews were conducted regarding the importance of and ability to complete the steps that may result in a patient receiving LCS. Data were analyzed using a thematic analysis with immersion crystallization then organized using the RE-AIM implementation science framework to illuminate and organize implementation issues. Results Although all groups endorsed the importance of LCS, all also struggled with implementation challenges. Since assessing smoking history is part of the process to identify eligibility for LCS, we asked about these processes. We found that smoking assessment and assistance (including referral to services) were routine in the practices, but other steps in the LCS portion of determining eligibility and offering LCS were not. Lack of knowledge about screening and coverage, patient stigma, and resistance and practical considerations such as distance to LCS testing facilities complicated completion of LCS compared to screening for other types of cancer. Conclusions Limited uptake of LCS results from a range of multiple interacting factors that cumulatively affect consistency and quality of implementation at the practice level. Future research should consider team-based approaches to conduct of LCS eligibility and shared decision making.

Abstract Background The Practical, Robust Implementation and Sustainability Model (PRISM) was developed in 2008 as a contextually expanded version of the broadly used Reach, Adoption, Effectiveness, Implementation, and Maintenance (RE-AIM) framework. PRISM provides researchers a pragmatic and intuitive model to improve translation of research interventions into clinical and community practice. Since 2008, the use of PRISM increased across diverse topics, populations, and settings. This citation analysis and scoping systematic review aimed to assess the use of the PRISM framework and to make recommendations for future research. Methods A literature search was conducted using three databases (PubMed, Web of Science, Scopus) for the period of 2008 and September 2020. After exclusion, reverse citation searches and invitations to experts in the field were used to identify and obtain recommendations for additional articles not identified in the original search. Studies that integrated PRISM into their study design were selected for full abstraction. Unique research studies were abstracted for information on study characteristics (e.g., setting/population, design), PRISM contextual domains, and RE-AIM outcomes. Results A total of 180 articles were identified to include PRISM to some degree. Thirty-two articles representing 23 unique studies integrated PRISM within their study design. Study characteristics varied widely and included studies conducted in diverse contexts, but predominately in high-income countries and in clinical out-patient settings. With regards to use, 19 used PRISM for evaluation, 10 for planning/development, 10 for implementation, four for sustainment, and one for dissemination. There was substantial variation across studies in how and to what degree PRISM contextual domains and RE-AIM outcomes were operationalized and connected. Only two studies directly connected individual PRISM context domains with RE-AIM outcomes, and another four included RE-AIM outcomes without direct connection to PRISM domains. Conclusions This is the first systematic review of the use of PRISM in various contexts. While there were low levels of ‘integrated’ use of PRISM and few reports on linkage to RE-AIM outcomes, most studies included important context domains of implementation and sustainability infrastructure and external environment. Recommendations are provided for more consistent and comprehensive use of and reporting on PRISM to inform both research and practice on contextual factors in implementation.

Abstract Introduction It is not realistic for most clinicians to perform the multitude of recommended preventive primary care services. This is especially true in low resource and rural settings, creating challenges to delivering high-quality care. This study collected stakeholder input from clinicians on which services they most need to improve. Methods The authors conducted a survey of primary care physicians 9–12/2021, with an emphasis on rural practices, to assess areas in which clinicians felt the greatest needs for improvement. The survey focused on primary prevention (behavior change counseling) and cancer screening, and contrasted needs for improvement for these services vs. other types of screening, and between clinicians in rural vs. non-rural practices. Results There were 326 respondents from 4 different practice-based research networks, a wide range of practice types, 49 states and included 177 clinicians in rural settings. Respondents rated the need to improve delivery of primary prevention counseling services highest, with needs for nutrition and dietary assessment and counseling rated highest followed by physical activity and with almost no differences between rural and nonrural. Needs for improvement in cancer screenings were rated higher than non-cancer screenings, except for blood pressure screening. Conclusions Both rural and nonrural primary care clinicians feel a need for improvement, especially with primary prevention activities. Although future research is needed to replicate these findings with different populations and other types of preventive service activities, greater priority should be given to development of practical, stakeholder informed assistance and resources for primary care to conduct primary prevention.

Abstract Background Lung ultrasound (LUS) is a clinician-performed evidence-based imaging modality that has multiple advantages in the evaluation of dyspnea caused by multiple disease processes, including COVID-19. Despite these advantages, few hospitalists have been trained to perform LUS. The aim of this study was to increase adoption and implementation of LUS during the 2020 COVID-19 pandemic by using recurrent assessments of RE-AIM outcomes to iteratively revise our implementation strategies. Methods In an academic hospital, we implemented guidelines for the use of LUS in patients with COVID-19 in July 2020. Using a novel “RE-AIM dashboard,” we used an iterative process of evaluating the high-priority outcomes of Reach, Adoption, and Implementation at twice monthly intervals to inform revisions of our implementation strategies for LUS delivery (i.e., Iterative RE-AIM process). Using a convergent mixed methods design, we integrated quantitative RE-AIM outcomes with qualitative hospitalist interview data to understand the dynamic determinants of LUS Reach, Adoption, and Implementation. Results Over the 1-year study period, 453 LUSs were performed in 298 of 12,567 eligible inpatients with COVID-19 (Reach = 2%). These 453 LUS were ordered by 43 out of 86 eligible hospitalists (LUS order adoption = 50%). However, the LUSs were performed/supervised by only 8 of these 86 hospitalists, 4 of whom were required to complete LUS credentialing as members of the hospitalist procedure service (proceduralist adoption 75% vs 1.2% non-procedural hospitalists adoption). Qualitative and quantitative data obtained to evaluate this Iterative RE-AIM process led to the deployment of six sequential implementation strategies and 3 key findings including (1) there were COVID-19-specific barriers to LUS adoption, (2) hospitalists were more willing to learn to make clinical decisions using LUS images than obtain the images themselves, and (3) mandating the credentialing of a strategically selected sub-group may be a successful strategy for improving Reach. Conclusions Mandating use of a strategically selected subset of clinicians may be an effective strategy for improving Reach of LUS. Additionally, use of Iterative RE-AIM allowed for timely adjustments to implementation strategies, facilitating higher levels of LUS Adoption and Reach. Future studies should explore the replicability of these preliminary findings.

Abstract Background As the field of implementation science wrestles with the need for system decision-makers to anticipate the budget impact of implementing new programs, there has been a push to report implementation costs more transparently. For this purpose, the method of time-driven activity-based costing (TDABC) has been heralded as a pragmatic advance. However, a recent TDABC review found that conventional methods for estimating staff time remain resource-intensive and called for simpler alternatives. Our objective was to conceptually compare conventional and emerging TDABC approaches to measuring staff time. Methods Our environmental scan of TDABC methods identified several categories of approaches for staff time estimation; across these categories, staff time was converted to cost as a pro-rated fraction of salary/benefits. Conventional approaches used a process map to identify each step of program delivery and estimated the staff time used at each step in one of 3 ways: (a) uniform estimates of time needed for commonly occurring tasks (self-report), (b) retrospective “time diary” (self-report), or (c) periodic direct observation. In contrast, novel semi-automated electronic health record (EHR) approaches “nudge” staff to self-report time for specific process map step(s)—serving as a contemporaneous time diary. Also, novel EHR-based automated approaches include timestamps to track specific steps in a process map. We compared the utility of these TDABC approach categories according to the 5 R’s model that measures domains of interest to system decision-makers: relevance, rapidity, rigor, resources, and replicability, and include two illustrative case examples. Results The 3 conventional TDABC staff time estimation methods are highly relevant to settings but have limited rapidity, variable rigor, are rather resource-intensive, and have varying replicability. In contrast to conventional TDABC methods, the semi-automated and automated EHR-based approaches have high rapidity, similar rigor, similar replicability, and are less resource-intensive, but have varying relevance to settings. Conclusions This synthesis and evaluation of conventional and emerging methods for staff time estimation by TDABC provides the field of implementation science with options beyond the current approaches. The field remains pressed to innovatively and pragmatically measure costs of program delivery that rate favorably across all of the 5 R’s domains.

Abstract Background Evidence, in multiple forms, is a foundation of implementation science. For public health and clinical practice, evidence includes the following: type 1 evidence on etiology and burden; type 2 evidence on effectiveness of interventions; and type 3: evidence on dissemination and implementation (D&I) within context. To support a vision for development and use of evidence in D&I science that is more comprehensive and equitable (particularly for type 3 evidence), this article aims to clarify concepts of evidence, summarize ongoing debates about evidence, and provide a set of recommendations and tools/resources for addressing the “how-to” in filling evidence gaps most critical to advancing implementation science. Main text Because current conceptualizations of evidence have been relatively narrow and insufficiently characterized in our opinion, we identify and discuss challenges and debates about the uses, usefulness, and gaps in evidence for implementation science. A set of questions is proposed to assist in determining when evidence is sufficient for dissemination and implementation. Intersecting gaps include the need to (1) reconsider how the evidence base is determined, (2) improve understanding of contextual effects on implementation, (3) sharpen the focus on health equity in how we approach and build the evidence-base, (4) conduct more policy implementation research and evaluation, and (5) learn from audience and stakeholder perspectives. We offer 15 recommendations to assist in filling these gaps and describe a set of tools for enhancing the evidence most needed in implementation science. Conclusions To address our recommendations, we see capacity as a necessary ingredient to shift the field’s approach to evidence. Capacity includes the “push” for implementation science where researchers are trained to develop and evaluate evidence which should be useful and feasible for implementers and reflect community or stakeholder priorities. Equally important, there has been inadequate training and too little emphasis on the “pull” for implementation science (e.g., training implementers, practice-based research). We suggest that funders and reviewers of research should adopt and support a more robust definition of evidence. By critically examining the evolving nature of evidence, implementation science can better fulfill its vision of facilitating widespread and equitable adoption, delivery, and sustainment of scientific advances.

Abstract Background Understanding the cost and/or cost-effectiveness of implementation strategies is crucial for organizations to make informed decisions about the resources needed to implement and sustain evidence-based interventions (EBIs). This economic evaluation protocol describes the methods and processes that will be used to assess costs and cost-effectiveness across implementation strategies used to improve the reach, adoption, implementation, and organizational maintenance of an evidence-based pediatric weight management intervention- Building Health Families (BHF). Methods A within-trial cost and cost-effectiveness analysis (CEA) will be completed as part of a hybrid type III effectiveness-implementation trial (HEI) designed to examine the impact of an action Learning Collaborative (LC) strategy consisting of network weaving, consultee-centered training, goal-setting and feedback, and sustainability action planning to improve the adoption, implementation, organizational maintenance, and program reach of BHF in micropolitan and surrounding rural communities in the USA, over a 12-month period. We discuss key features of implementation strategy components and the associated cost collection and outcome measures and present brief examples on what will be included in the CEA for each discrete implementation strategy and how the results will be interpreted. The cost data will be collected by identifying implementation activities associated with each strategy and using a digital-based time tracking tool to capture the time associated with each activity. Costs will be assessed relative to the BHF program implementation and the multicomponent implementation strategy, included within and external to a LC designed to improve reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) of BHF. The CEA results will be reported by RE-AIM outcomes, using the average cost-effectiveness ratio or incremental cost-effectiveness ratio. All the CEAs will be performed from the community perspective. Discussion The proposed costing approach and economic evaluation framework for dissemination and implementation strategies and EBI implementation will contribute to the evolving but still scant literature on economic evaluation of implementation and strategies used and facilitate the comparative economic analysis. Trial registration ClinicalTrials.gov NCT04719442 . Registered on January 22, 2021.

Abstract Background This report describes how we refined a protocol for a pragmatic comparative effectiveness study of two models of an evidence-based diabetes shared medical appointment intervention and used the PRECIS-2 rating system to evaluate these adaptations. Methods We report primary data collected between June and August 2019, and protocol refinements completed between 2018 and 2020. Twenty-two members of the study team collaborated in protocol refinement and completed the PRECIS-2 ratings of study pragmatism. We discuss study design refinements made to achieve the desired level of pragmatism vs. experimental control for each of the nine PRECIS-2 dimensions. Study team members received training on PRECIS-2 scoring and were asked to rate the study protocol on the nine PRECIS-2 dimensions. Ratings were compared using descriptive statistics. Results In general, the PRECIS-2 ratings revealed high levels of pragmatism, but somewhat less pragmatic ratings on the categories of Delivery and Organization (costs and resources). This variation was purposeful, and we provide the rationale for and steps taken to obtain the targeted level of pragmatism on each PRECIS-2 dimension, as well as detail design changes made to a) make the design more pragmatic and b) address COVID-19 issues. There was general agreement among team members and across different types of stakeholders on PRECIS-2 ratings. Conclusions We discuss lessons learned from use of PRECIS-2 and experiences in refining the study to be maximally pragmatic on some dimensions and less so on other dimensions. This paper expands on prior research by describing actions to achieve higher levels of pragmatism and revise our protocol fit to the changed context. We make recommendations for future use of PRECIS-2 to help address changing context and other strategies for the planning of and transparent reporting on pragmatic research and comparative effectiveness research. Trial registration Clinicaltrials.gov Registration ID: NCT03590041 .

Background: Take home naloxone (THN) programs have been shown to effectively reverse opioid overdose events with limited adverse events, yet often miss young adults who use opioids. To identify opportunities for naloxone expansion, we conducted interviews with young adults who had used opioids. We explored young adults’ experience with current THN programs, and perspectives on ideal THN programs and emerging naloxone public health vending machine (PHVM) programs shown to increase access to sterile syringes in young adults. Methods: We interviewed 16 young adults receiving substance treatment services within an integrated safety net healthcare system. Participants were 18–30 years of age with a history of nonmedical prescription opioid use. Interviews obtained the patient perspective of current THN, ideal THN and PHVM programs. Interviews were transcribed and coded by team-based methods. Themes were developed using an inductive-deductive iterative approach and defined through consensus. Results: Treatment was often the first exposure to naloxone. Participants recommended easy to access programs for ideal naloxone distribution and had overall positive feedback on PHVMs. Three key themes were identified to improve naloxone uptake: knowledge, convenience, and privacy. Participants identified safety, lack of police presence, and low costs as important vending machine features. Conclusions: Our results identified implementation opportunities to increase naloxone uptake including convenient location and hours, privacy, and using trusted sources of information to improve program awareness. PHVMs present an opportunity to maximize these opportunities and increase access to naloxone in young adults.

Objectives: It is important to understand the unique perspectives and values that motivate patients and clinicians in rural primary care settings to participate in clinical care activities. Our objective was to explore perspectives, preferences, and values related to primary care that could influence implementation of evidence-based programs. Methods: Qualitative study utilizing semi-structured interviews and using immersion/crystallization and thematic analysis. Participants were primary care practice members (clinicians, clinical staff, and administrators) and their patients in rural Colorado. Results: Twenty-six practice members and 23 patients across 9 practices participated. There were 4 emergent themes that were consistent across practice members and some patients. Patient perspectives are located in parenthesis. They included: (1) Focus on quality patient care, patient satisfaction, and continuity of care (patients appreciated quality and compassionate care), (2) Importance of prevention and wellness (patients appreciated help with preventing health problems), (3) Clinician willingness and ability to meet patient preferences for care (patients described comfort with local care), and (4) Passion for serving underserved, uninsured, or vulnerable populations (patients described their vulnerabilities). There were differences in how the perspectives were operationalized by practice member role, illustrating the importance of different ways of addressing these values. Conclusions: Successful implementation requires consideration of context, and much of context is understanding what is important to those involved in the primary care experience. This study sheds light on salient values of rural primary care practice members and their patients, which may inform interventions designed with and for this setting.

Abstract Background Uncontrolled hypertension represents a substantial and growing burden in Guatemala and other low and middle-income countries. As a part of the formative phase of an implementation research study, we conducted a needs assessment to define short- and long-term needs and opportunities for hypertension services within the public health system. Methods We conducted a multi-method, multi-level assessment of needs related to hypertension within Guatemala’s public system using the World Health Organization’s health system building blocks framework. We conducted semi-structured interviews with stakeholders at national (n = 17), departmental (n = 7), district (n = 25), and community (n = 30) levels and focus groups with patients (3) and frontline auxiliary nurses (3). We visited and captured data about infrastructure, accessibility, human resources, reporting, medications and supplies at 124 health posts and 53 health centers in five departments of Guatemala. We conducted a thematic analysis of transcribed interviews and focus group discussions supported by matrix analysis. We summarized quantitative data observed during visits to health posts and centers. Results Major challenges for hypertension service delivery included: gaps in infrastructure, insufficient staffing and high turnover, limited training, inconsistent supply of medications, lack of reporting, low prioritization of hypertension, and a low level of funding in the public health system overall. Key opportunities included: prior experience caring for patients with chronic conditions, eagerness from providers to learn, and interest from patients to be involved in managing their health. The 5 departments differ in population served per health facility, accessibility, and staffing. All but 7 health posts had basic infrastructure in place. Enalapril was available in 74% of health posts whereas hydrochlorothiazide was available in only 1 of the 124 health posts. With the exception of one department, over 90% of health posts had a blood pressure monitor. Conclusions This multi-level multi-method needs assessment using the building blocks framework highlights contextual factors in Guatemala’s public health system that have been important in informing the implementation of a hypertension control trial. Long-term needs that are not addressed within the scope of this study will be important to address to enable sustained implementation and scale-up of the hypertension control approach.

Abstract Understanding the resources needed to achieve desired implementation and effectiveness outcomes is essential to implementing and sustaining evidence-based practices (EBPs). Despite this frequent observation, cost and economic measurement and reporting are rare, but becoming more frequent in implementation science, and when present is seldom reported from the perspective of multiple stakeholders (e.g., the organization, supervisory team), including those who will ultimately implement and sustain EBPs. Incorporating a multi-level framework is useful for understanding and integrating the perspectives and priorities of the diverse set of stakeholders involved in implementation. Stakeholders across levels, from patients to delivery staff to health systems, experience different economic impacts (costs, benefit, and value) related to EBP implementation and have different perspectives on these issues. Economic theory can aid in understanding multi-level perspectives and approaches to addressing potential conflict across perspectives. This paper provides examples of key cost components especially important to different types of stakeholders. It provides specific guidance and recommendations for cost assessment activities that address the concerns of various stakeholder groups, identifies areas of agreement and conflict in priorities, and outlines theoretically informed approaches to understanding conflicts among stakeholder groups and processes to address them. Involving stakeholders throughout the implementation process and presenting economic information in ways that are clear and meaningful to different stakeholder groups can aid in maximizing benefits within the context of limited resources. We posit that such approaches are vital to advancing economic evaluation in implementation science. Finally, we identify directions for future research and application. Considering a range of stakeholders is critical to informing economic evaluation that will support appropriate decisions about resource allocation across contexts to inform decisions about successful adoption, implementation, and sustainment. Not all perspectives need to be addressed in a given project but identifying and understanding perspectives of multiple groups of key stakeholders including patients and direct implementation staff not often explicitly considered in traditional economic evaluation are needed in implementation research.

Abstract Background Nearly half of patients do not take their cardiovascular medications as prescribed, resulting in increased morbidity, mortality, and healthcare costs. Mobile and digital technologies for health promotion and disease self-management offer an opportunity to adapt behavioral “nudges” using ubiquitous mobile phone technology to facilitate medication adherence. The Nudge pragmatic clinical trial uses population-level pharmacy data to deliver nudges via mobile phone text messaging and an artificial intelligent interactive chat bot with the goal of improving medication adherence and patient outcomes in three integrated healthcare delivery systems. Methods The Theory of mHealth, the Expanded RE-AIM/PRISM, and the PRECIS-2 frameworks were used for program planning, implementation, and evaluation, along with a focus on dissemination and cost considerations. During the planning phase, the Nudge study team developed and piloted a technology-based nudge message and chat bot of optimized interactive content libraries for a range of diverse patients. Inclusion criteria are very broad and include patients in one of three diverse health systems who take medications to treat hypertension, atrial fibrillation, coronary artery disease, diabetes, or hyperlipidemia. A target of approximately 10,000 participants will be randomized to one of 4 study arms: usual care (no intervention), generic nudge (text reminder), optimized nudge, and optimized nudge plus interactive AI chat bot. The PRECIS-2 tool indicated that the study protocol is very pragmatic, although there is variability across PRECIS-2 dimensions. Discussion The primary effectiveness outcome is medication adherence defined by the proportion of days covered (PDC) using pharmacy refill data. Implementation outcomes are assessed using the RE-AIM framework, with a particular focus on reach, consistency of implementation, adaptations, cost, and maintenance/sustainability. The project has limitations including limited power to detect some subgroup effects, medication complications (bleeding), and longer-term outcomes (myocardial infarction). Strengths of the study include the diverse healthcare systems, a feasible and generalizable intervention, transparent reporting using established pragmatic research and implementation science frameworks, strong stakeholder engagement, and planning for dissemination and sustainment. Trial registration ClinicalTrials.gov NCT03973931 . Registered on 4 June 2019. The study was funded by the NIH; grant number is 4UH3HL144163-02 issued 4/5/19.

Responding rapidly to emerging public health crises is vital to reducing their escalation, spread, and impact on population health. These responses, however, are challenging and disparate processes for researchers and practitioners. Researchers often develop new interventions that take significant time and resources, with little exportability. In contrast, community-serving systems are often poorly equipped to properly adopt new interventions or adapt existing ones in a data-driven way during crises' onset and escalation. This results in significant delays in deploying evidence-based interventions (EBIs) with notable public health consequences. This prolonged timeline for EBI development and implementation results in significant morbidity and mortality that is costly and preventable. As public health emergencies have demonstrated (e.g., COVID-19 pandemic), the negative consequences often exacerbate existing health disparities. Implementation science has the potential to bridge the extant gap between research and practice, and enhance equity in rapid public health responses, but is underutilized. For the field to have a greater “real-world” impact, it needs to be more rapid, iterative, participatory, and work within the timeframes of community-serving systems. This paper focuses on rapid adaptation as a developing implementation science area to facilitate system responses during public health crises. We highlight frameworks to guide rapid adaptation for optimizing existing EBIs when responding to urgent public health issues. We also explore the economic implications of rapid adaptation. Resource limitations are frequently a central reason for implementation failure; thus, we consider the economic impacts of rapid adaptation. Finally, we provide examples and propose directions for future research and application.

Introduction: DNA-based population screening has been proposed as a public health solution to identify individuals at risk for serious health conditions who otherwise may not present for medical care. The clinical utility and public health impact of DNA-based population screening is a subject of active investigation. Geisinger, an integrated healthcare delivery system, was one of the first healthcare systems to implement DNA screening programs (MyCode Community Health Initiative (MyCode) and clinical DNA screening pilot) that leverage exome data to identify individuals at risk for developing conditions with potential clinical actionability. Here, we demonstrate the use of an implementation science framework, RE-AIM (Reach, Effectiveness, Adoption, Implementation and Maintenance), to conduct a post-hoc evaluation and report outcomes from these two programs to inform the potential impact of DNA-based population screening.Methods: Reach and Effectiveness outcomes were determined from the MyCode research program, while Adoption and Implementation outcomes were measured using the clinical DNA screening pilot. Reach was defined as the number of patients who were offered and consented to participate in MyCode. Effectiveness of DNA screening was measured by reviewing MyCode program publications and synthesizing findings from themes. Adoption was measured by the total number of DNA screening tests ordered by clinicians at the clinical pilot sites. Implementation was assessed by interviewing a subset of clinical pilot clinicians about the deployment of and recommended adaptations to the pilot that could inform future program dissemination.Results:Reach: As of August 2020, 68% (215,078/316,612) of individuals approached to participate in the MyCode program consented. Effectiveness: Published evidence reported from MyCode demonstrates that DNA screening identifies at-risk individuals more comprehensively than clinical ascertainment based on phenotypes or personal/family history. Adoption: From July 2018 to June 2021, a total of 1,026 clinical DNA screening tests were ordered by 60 clinicians across the three pilot clinic sites. Implementation: Interviews with 14 clinicians practicing at the pilot clinic sites revealed motivation to provide patients with DNA screening results and yielded future implementation strategies.Conclusion: The RE-AIM framework offers a pragmatic solution to organize, analyze, and report outcomes across differently resourced and designed precision health programs that include genomic sequencing and return of clinically actionable genomic information.

Abstract Background Hypertension is a major risk factor for cardiovascular disease (CVD). Despite advances in hypertension prevention and treatment, the proportion of patients who are aware, treated and controlled is low, particularly in low-income and middle-income countries (LMICs). We will evaluate an adapted version of a multilevel and multicomponent hypertension control program in Guatemala, previously proven effective and feasible in Argentina. The program components are: protocol-based hypertension treatment using a standardized algorithm; team-based collaborative care; health provider education; health coaching sessions; home blood pressure monitoring; blood pressure audit; and feedback. Methods Using a hybrid type 2 effectiveness-implementation design, we will evaluate clinical and implementation outcomes of the multicomponent program in Guatemala over an 18-month period. Through a cluster randomized trial, we will randomly assign 18 health districts to the intervention arm and 18 to enhanced usual care across five departments, enrolling 44 participants per health district and 1584 participants in total. The clinical outcomes are (1) the difference in the proportion of patients with controlled hypertension (

Abstract Background Shared medical appointments (SMAs) have been shown to be an efficient and effective strategy for providing diabetes self-management education and self-management support. SMA features vary and it is not known which features are most effective for different patients and practice settings. The Invested in Diabetes study tests the comparative effectiveness of SMAs with and without multidisciplinary care teams and patient topic choice for improving patient-centered and clinical outcomes related to diabetes. Methods This study compares the effectiveness of two SMA approaches using the Targeted Training for Illness Management (TTIM) curriculum. Standardized SMAs are led by a health educator with a set order of TTIM topics. Patient-driven SMAs are delivered collaboratively by a multidisciplinary care team (health educator, medical provider, behavioral health provider, and a peer mentor); patients select the order and emphasis on TTIM topics. Invested in Diabetes is a cluster randomized pragmatic trial involving approximately 1440 adult patients with type 2 diabetes. Twenty primary care practices will be randomly assigned to either standardized or patient-driven SMAs. A mixed-methods evaluation will include quantitative (practice- and patient-level data) and qualitative (practice and patient interviews, observation) components. The primary patient-centered outcome is diabetes distress. Secondary outcomes include autonomy support, self-management behaviors, clinical outcomes, patient reach, and practice-level value and sustainability. Discussion Practice and patient stakeholder input guided protocol development for this pragmatic trial comparing SMA approaches. Implementation strategies from the enhanced Replicating Effective Programs framework will help ensure practices maintain fidelity to intervention protocols while tailoring workflows to their settings. Invested in Diabetes will contribute to the literature on chronic illness management and implementation science using the RE-AIM model. Trial registration ClinicalTrials.gov, NCT03590041. Registered on 5 July 2018.

Implementation science is concerned with the study of adoption, implementation and maintenance of evidence-based interventions and use of implementation strategies to facilitate translation into practice. Ways to conceptualize and overcome challenges to implementing evidence-based practice may enhance the field of implementation science. The concept of mental models may be one way to view such challenges and to guide selection, use, and adaptation of implementation strategies to deliver evidence-based interventions. A mental model is an interrelated set of beliefs that shape how a person forms expectations for the future and understands the way the world works. Mental models can shape how an individual thinks about or understands how something or someone does, can, or should function in the world. Mental models may be sparse or detailed, may be shared among actors in implementation or not, and may be substantially tacit, that is, of limited accessibility to introspection. Actors' mental models can determine what information they are willing to accept and what changes they are willing to consider. We review the concepts of mental models and illustrate how they pertain to implementation of an example intervention, shared decision making. We then describe and illustrate potential methods for eliciting and analyzing mental models. Understanding the mental models of various actors in implementation can provide crucial information for understanding, anticipating, and overcoming implementation challenges. Successful implementation often requires changing actors' mental models or the way in which interventions or implementation strategies are presented or implemented. Accurate elicitation and understanding can guide strategies for doing so.

Abstract There are complex issues in understanding and categorizing implementation science theories, models, and frameworks. Systematic reviews of these models are important undertakings for synthesizing current knowledge. The issues involved are even more challenging when reviewing a large number of frameworks and when some of the frameworks have evolved significantly over time. This paper addresses how the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework was described in the recent Esmail (2020) review and identifies four mischaracterizations. This is followed by a more general discussion of how advances or extensions of frameworks after an original source publication or influential review tend to be overlooked. We discuss why inadvertent mischaracterization of what a framework is and is not, and what it can and cannot be used for, can have deleterious consequences. Finally, we suggest initial ideas about what could be done to prevent or alleviate some of these problems by reviewers, framework developers, and scholars at large.

Abstract Background Prehospital (ambulance) care can reduce morbidity and mortality from trauma. Yet, there is a dearth of effective evidence-based interventions and implementation strategies. Emergency Medical Services Traumatic Shock Care (EMS-TruShoC) is a novel bundle of five core evidence-based trauma care interventions. High-Efficiency EMS Training (HEET) is an innovative training and sensitization program conducted during clinical shifts in ambulances. We assess the feasibility of implementing EMS-TruShoC using the HEET strategy, and feasibility of assessing implementation and clinical outcomes. Findings will inform a main trial. Methods We conducted a single-site, prospective cohort, multi-methods pilot implementation study in Western Cape EMS system of South Africa. Of the 120 providers at the study site, 12 were trainers and the remaining were eligible learners. Feasibility of implementation was guided by the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework. Feasibility of assessing clinical outcomes was assessed using shock indices and clinical quality of care scores, collected via abstraction of patients’ prehospital trauma charts. Thresholds for progression to a main trial were developed a priori. Results The average of all implementation indices was 83% (standard deviation = 10.3). Reach of the HEET program was high, with 84% learners completing at least 75% of training modules. Comparing the proportion of learners attaining perfect scores in post- versus pre-implementation assessments, there was an 8-fold (52% vs. 6%) improvement in knowledge, 3-fold (39% vs. 12%) improvement in skills, and 2-fold (42% vs. 21%) increase in self-efficacy. Clinical outcomes data were successfully calculated—there were clinically significant improvements in shock indices and quality of prehospital trauma care in the post- versus pre-implementation phases. Adoption of HEET was good, evidenced by 83% of facilitator participation in trainings, and 100% of surveyed stakeholders indicating good programmatic fit for their organization. Stakeholders responded that HEET was a sustainable educational solution that aligned well with their organization. Implementation fidelity was very high; 90% of the HEET intervention and 77% of the implementation strategy were delivered as originally planned. Participants provided very positive feedback, and explained that on-the-job timing enhanced their participation. Maintenance was not relevant to assess in this pilot study. Conclusions We successfully implemented the EMS-TruShoC educational intervention using the HEET training strategy in a single-site pilot study conducted in a low-resource international setting. All clinical outcomes were successfully calculated. Overall, this pilot study suggests high feasibility of our future, planned experimental trial.

Health information technology (HIT) has not been broadly adopted for use in outpatient healthcare settings to effectively address obesity in youth, especially among disadvantaged populations that face greater barriers to good health. A well-designed HIT tool can deliver behavior change recommendations and provide community resources to address this gap, and the Obesity-Related Behavioral Intervention Trials (ORBIT) model can guide its development and refinement. This article reports the application of the ORBIT model to (1) describe the characteristics and design of a novel HIT tool (the PREVENT tool) using behavioral theory, (2) illustrate the use of stakeholder-centered “designing for dissemination and sustainability” principles, and (3) discuss the practical implications and directions for future research. Two types of stakeholder engagement (customer discovery and user testing) were conducted with end users (outpatient healthcare teams). Customer discovery interviews (n = 20) informed PREVENT tool components and intervention targets by identifying (1) what healthcare teams (e.g., physicians, dietitians) identified as their most important “jobs to be done” in helping adolescents who are overweight/obese adopt healthy behaviors, (2) their most critical “pains” and “gains” related to overweight/obesity treatment, and (3) how they define success compared to competing alternatives. Interviews revealed the need for a tool to help healthcare teams efficiently deliver tailored, evidence-based behavior change recommendations, motivate patients, and follow-up with patients within the constraints of clinic schedules and workflows. The PREVENT tool was developed to meet these needs. It facilitates prevention discussions, delivers tailored, evidence-based recommendations for physical activity and food intake, includes an interactive map of community resources to support behavior change, and automates patient follow-up. Based on Self-Determination Theory, the PREVENT tool engages the patient to encourage competence and autonomy to motivate behavior change. The use of this intentional, user-centered design process should increase the likelihood of the intended outcomes (e.g., behavior change, weight stabilization/loss) and ultimately increase uptake, implementation success, and long-term results. After initial tool development, user-testing interviews (n = 13) were conducted using a think-aloud protocol that provided insight into users' (i.e., healthcare teams) cognitive processes, attitudes, and challenges when using the tool. Overall, the PREVENT tool was perceived to be useful, well-organized, and visually appealing.

Abstract Introduction: Understanding, categorizing, and using implementation science theories, models, and frameworks is a complex undertaking. The issues involved are even more challenging given the large number of frameworks and that some of them evolve significantly over time. As a consequence, researchers and practitioners may be unintentionally mischaracterizing frameworks or basing actions and conclusions on outdated versions of a framework. Methods: This paper addresses how the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework has been described, summarizes how the model has evolved over time, and identifies and corrects several misconceptions. Results: We address 13 specific areas where misconceptions have been noted concerning the use of RE-AIM and summarize current guidance on these issues. We also discuss key changes to RE-AIM over the past 20 years, including the evolution to Pragmatic Robust Implementation and Sustainability Model, and provide resources for potential users to guide application of the framework. Conclusions: RE-AIM and many other theories and frameworks have evolved, been misunderstood, and sometimes been misapplied. To some degree, this is inevitable, but we conclude by suggesting some actions that reviewers, framework developers, and those selecting or applying frameworks can do to prevent or alleviate these problems.

This report describes how stakeholder groups informed a web-based care planning tool’s development for addressing root causes of poor health. Stakeholders included community members (n = 6), researchers (n = 6), community care providers (n = 9), and patients (n = 17). Feedback was solicited through focus groups, semi-structured interviews, and user experience observations and then qualitatively analyzed to identify themes. Each group contributed a unique perspective. Researchers wanted evidence-based content; community members and providers focused on making goals manageable; patients wanted care team support and simple action-oriented language. Our findings highlight the benefits of stakeholder input. Blending perspectives from multiple groups results in a more robust intervention design.

Abstract Background: Little is known about practices used to disseminate findings to non-research, practitioner audiences. This study describes the perspectives, experience and activities of dissemination & implementation (D&I) scientists around disseminating their research findings. Methods: The study explored D&I scientists’ experiences and recommendations for assessment of dissemination activities to non-research audiences. Existing list serves were used to recruit scientists. Respondents were asked three open-ended questions on an Internet survey about dissemination activities, recommendations for changing evaluation systems and suggestions to improve their own dissemination of their work. Results: Surveys were completed by 159 scientists reporting some training, funding and/or publication history in D&I. Three themes emerged across each of the three open-ended questions. Question 1 on evaluation generated the themes of: 1a) promotional review; 1b) funding requirements and 1c) lack of acknowledgement of dissemination activities. Question 2 on recommended changes generated the themes of: 2a) dissemination as a requirement of the academic promotion process; 2b) requirement of dissemination plan and 2c) dissemination metrics. Question 3 on personal changes to improve dissemination generated the themes of: 3a) allocation of resources for dissemination activities; 3b) emerging dissemination channels and 3c) identify and address issues of priority for stakeholders. Conclusions: Our findings revealed different types of issues D&I scientists encounter when disseminating findings to clinical, public health or policy audiences and their suggestions to improve the process. Future research should consider key requirements which determine academic promotion and grant funding as an opportunity to expand dissemination efforts.

Introduction Vaccination against influenza and pertussis in pregnancy offers a ‘two-for-one’ opportunity to protect mother and child. Pregnant patients have increased risk of severe disease from influenza and newborns have increased risk of severe disease from both influenza and pertussis. Obstetricians need communication tools to support their self-efficacy and effectiveness in communicating the importance of immunisation during pregnancy and ultimately improving maternal vaccination rates.Methods and analysis We describe the protocol for a pragmatic study testing the feasibility and potential impact of a clinician communication strategy on maternal vaccination uptake. This study will be conducted in five prenatal care settings in Colorado, USA. The Motivational Interviewing for Maternal Immunisation strategy involves training prenatal care providers to use motivational interviewing in the vaccine conversation with pregnant patients. Our primary outcomes will be the adoption and implementation of the intervention measured using the Enhanced RE-AIM/Practical Robust Implementation and Sustainability Model for dissemination and implementation. Secondary outcomes will include provider time spent, fidelity to Motivational Interviewing and self-efficacy measured through audio recorded visits and provider surveys, patients’ visit experience based on audio recorded visits and follow-up interviews, and maternal vaccine uptake as measured through chart reviews.Ethics and dissemination This study is approved by the following institutional review boards: Colorado Multiple Institutional Review Board. Results will be disseminated through peer-reviewed manuscripts and conference presentations.Trial registration number NCT04302675.

The Centers for Medicare and Medicaid Services (CMS) has mandated shared decision making (SDM) using patient decision aids for three conditions (lung cancer screening, atrial fibrillation, and implantable defibrillators). These forward-thinking approaches are in response to a wealth of efficacy data demonstrating that decision aids can improve patient decision making. However, there has been little focus on how to implement these approaches in real-world practice. This article demonstrates how using an implementation science framework may help programs understand multilevel challenges and opportunities to improve adherence to the CMS mandates. Using the PRISM (Pragmatic Robust Implementation and Sustainability Model) framework, we discuss general challenges to implementation of SDM, issues specific to each mandate, and how to plan for, enhance, and assess SDM implementation outcomes. Notably, a theme of this discussion is that successful implementation is context-specific and to truly have successful and sustainable changes in practice, context variability, and adaptation to context must be considered and addressed.

RE-AIM is a widely adopted, robust implementation science (IS) framework used to inform intervention and implementation design, planning, and evaluation, as well as to address short-term maintenance. In recent years, there has been growing focus on the longer-term sustainability of evidence-based programs, policies and practices (EBIs). In particular, investigators have conceptualized sustainability as the continued health impact and delivery of EBIs over a longer period of time (e.g., years after initial implementation) and incorporated the complex and evolving nature of context. We propose a reconsideration of RE-AIM to integrate recent conceptualizations of sustainability with a focus on addressing dynamic context and promoting health equity. In this Perspective, we present an extension of the RE-AIM framework to guide planning, measurement/evaluation, and adaptations focused on enhancing sustainability. We recommend consideration of: (1) extension of “maintenance” within RE-AIM to include recent conceptualizations of dynamic, longer-term intervention sustainability and “evolvability” across the life cycle of EBIs, including adaptation and potential de-implementation in light of changing and evolving evidence, contexts, and population needs; (2) iterative application of RE-AIM assessments to guide adaptations and enhance long-term sustainability; (3) explicit consideration of equity and cost as fundamental, driving forces that need to be addressed across RE-AIM dimensions to enhance sustainability; and (4) use or integration of RE-AIM with other existing frameworks that address key contextual factors and examine multi-level determinants of sustainability. Finally, we provide testable hypotheses and detailed research questions to inform future research in these areas.

Introduction: Implementation science frameworks have helped advance translation of research to practice. They have been widely used for planning and post-hoc evaluation, but seldom to inform and guide mid-course adjustments to intervention and implementation strategies.Materials and Methods: This study developed an innovative methodology using the RE-AIM framework and related tools to guide mid-course assessments and adaptations across five diverse health services improvement projects in the Veterans Health Administration (VA). Using a semi-structured guide, project team members were asked to assess the importance of and progress on each RE-AIM dimension (i.e., reach, effectiveness, adoption, implementation, maintenance) at the current phase of their project. Based on these ratings, each team identified one or two RE-AIM dimensions for focused attention. Teams developed proximal goals and implementation strategies to improve progress on their selected dimension(s). A follow-up meeting with each team occurred approximately 6 weeks after the goal setting meeting to evaluate the usefulness of the iterative process. Results were evaluated using both descriptive quantitative analyses and qualitative assessments from interviews and meeting notes.Results: A median of seven team members participated in the two meetings. Qualitative and descriptive data revealed that the process was feasible, understandable and useful to teams in adjusting their interventions and implementation strategies. The RE-AIM dimensions identified as most important were adoption and effectiveness, and the dimension that had the largest gap between importance and rated progress was reach. The dimensions most frequently selected for improvement were reach and adoption. Examples of action plans were summarizing stakeholder interviews for leadership, revising exclusion criteria, and conducting in-service trainings. Follow-up meetings indicated that teams found the process very useful and were able to implement the action plans they set.Discussion: The iterative use of RE-AIM to support adjustments during project implementation proved feasible and useful across diverse projects in the VA setting. Building on this and related examples, future research should replicate these findings and further develop the methodology, as well as explore the optimal frequency and timing for these iterative applications of RE-AIM. More generally, greater focus on more rapid and iterative use of implementation science frameworks is encouraged to facilitate successful translation of research to practice.

The National Working Group on RE-AIM Planning and Evaluation Framework (herein Workgroup) was established in 2004 to support the application of the framework and advance dissemination and implementation science (D&I). Workgroup members developed and disseminated products and resources (and continue to do so) to advocate for consistent application of RE-AIM and allow for cross study comparisons. The purpose of this paper is to summarize key Workgroup activities, products, and services (e.g., webinars, consultations, planning tools) and enhance bidirectional communication between the Workgroup and RE-AIM users. The ultimate goal of this work is to serve as a forum for dissemination to improve the balance between RE-AIM user demand (needs) and the currently limited RE-AIM Workgroup supply (consultation and resources) to demonstrate and expand the utility of RE-AIM as a D&I planning and evaluation framework. A summary of resources is provided as well as specific examples of how the Workgroup has been responsive to user needs.

A dramatically increased interest in dissemination and implementation (D&I) science, with relatively few training programs for D&I scientists, highlights the need for innovative ways to deliver educational materials, training, and resources. We described nine interactive, web-based D&I science resources appropriate for trainees and Clinical and Translational Science Awards. We used audience feedback and design thinking to develop resources iteratively. Primary target users are T3–T4 researchers, although T2 researchers can benefit from “designing for dissemination” resources. Workforce development resources were used in D&I science workshops, as stand-alone, self-directed resources, and for consultations and trainings. We assessed resource design (purpose, functionality), usage, user experience and engagement. Educational resources addressed included: D&I science basics, pragmatic trials, getting proposals funded, designing for dissemination, and D&I science theory selection. We reviewed the purpose, functionality, status, and usage of these interactive resources. All resources engaged users; provided interactive feedback for learners; and linked users to additional learning resources. Online resources can be valuable for preparing clinical and translational mentees for research consultations, as follow-up training activities, and as D&I workforce development resources. The resources described are publicly available and we encourage their use, further development, and evaluation by Clinical and Translational Science Awards and other programs.

Abstract Background There have been over 430 publications using the RE-AIM model for planning and evaluation of health programs and policies, as well as numerous applications of the model in grant proposals and national programs. Full use of the model includes use of qualitative methods to understand why and how results were obtained on different RE-AIM dimensions, however, recent reviews have revealed that qualitative methods have been used infrequently. Having quantitative and qualitative methods and results iteratively inform each other should enhance understanding and lessons learned. Methods Because there have been few published examples of qualitative approaches and methods using RE-AIM for planning or assessment and no guidance on how qualitative approaches can inform these processes, we provide guidance on qualitative methods to address the RE-AIM model and its various dimensions. The intended audience is researchers interested in applying RE-AIM or similar implementation models, but the methods discussed should also be relevant to those in community or clinical settings. Results We present directions for, examples of, and guidance on how qualitative methods can be used to address each of the five RE-AIM dimensions. Formative qualitative methods can be helpful in planning interventions and designing for dissemination. Summative qualitative methods are useful when used in an iterative, mixed methods approach for understanding how and why different patterns of results occur. Conclusions In summary, qualitative and mixed methods approaches to RE-AIM help understand complex situations and results, why and how outcomes were obtained, and contextual factors not easily assessed using quantitative measures.

Despite the many advantages of lung ultrasound (LUS) in the diagnosis and management of patients with dyspnea, its adoption among hospitalists has been slow. We performed semi-structured interviews of hospitals from four diverse health systems in the United States to understand determinants of adoption within a range of clinical settings. We used the diffusion of innovation theory to guide a framework analysis of the data. Of the 27 hospitalists invited, we performed 22 interviews from four hospitals of diverse types. Median years post-residency of interviewees was 10.5 [IQR:5-15]. Four main themes emerged: (1) There are important clinical advantages to LUS despite operator dependence, (2) LUS enhances patient and clinician experience, (3) Investment of clinician time to learn and perform LUS is a barrier to adoption but yields improved efficiency for the health system and (4) Mandated training and use may be necessary to achieve broad adoption as monetary incentives are less effective. Despite the perceived benefits of LUS for patients, clinicians and health systems, a significant barrier to broad LUS adoption is the experience of time scarcity by hospitalists. Future implementation strategies should focus on changes to the clinical environment that address clinician barriers to learning and adoption of new skills.

Precision health and big data approaches have great potential, yet such benefits will be realized only when social and behavioral determinants of health and patient preferences are combined with genomic information. Literature review and co-author experiences informed this commentary. Validated health behavior, mental health, and patient preference measures were collected and summarized in real time. Integration of such data into existing data sets will advance precision health, patient-centered care, research, and policy.

Background: The RE-AIM framework has been widely used in health research but it is unclear the extent to which this framework is also used for planning and evaluating health-related programs in clinical and community settings. Our objective was to evaluate how RE-AIM is used in the “real-world” and identify opportunities for improving use outside of research contexts.Methods: We used purposive and snowball sampling to identify clinical and community health programs that used RE-AIM for planning and/or evaluation. Recruitment methods included surveys with email follow-up to funders, implementers, and RE-AIM working group members. We identified 17 programs and conducted structured in-depth interviews with key informants (n = 18). Across RE-AIM dimensions, respondents described motivations, uses, and measures; rated understandability and usefulness; discussed benefits and challenges, strategies to overcome challenges, and resources used. We used descriptive statistics for quantitative ratings, and content analysis for qualitative data.Results: Program content areas included chronic disease management and prevention, healthy aging, mental health, or multiple, often behavioral health-related topics. During planning, most programs considered reach (n = 9), adoption (n = 11), and implementation (n = 12) while effectiveness (n = 7) and maintenance (n = 6) were considered less frequently. In contrast, most programs evaluated all RE-AIM dimensions, ranging from 13 programs assessing maintenance to 15 programs assessing implementation and effectiveness. On five-point scales, all RE-AIM dimensions were rated as easy to understand (Overall M = 4.7 ± 0.5), but obtaining data was rated as somewhat challenging (Overall M = 3.4 ± 0.9). Implementation was the most frequently used dimension to inform program design (M = 4.7 ± 0.6) relative to the other dimensions (3.0–3.9). All dimensions were considered similarly important for decision-making (average M = 4.1 ± 1.4), with the exception of maintenance (M = 3.4 ± 1.7). Qualitative corresponded to the quantitative findings in that RE-AIM was reported to be a practical, easy to understand, and well-established implementation science framework. Challenges included understanding differences among RE-AIM dimensions and data acquisition. Valuable resources included the RE-AIM website and collaborating with an expert.Discussion: RE-AIM is an efficient framework for planning and evaluation of clinical and community-based projects. It provides structure to systematically evaluate health program impact. Programs found planning for and assessing maintenance difficult, providing opportunities for further refinement.

The RE-AIM planning and evaluation framework was conceptualized two decades ago. As one of the most frequently applied implementation frameworks, RE-AIM has now been cited in over 2,800 publications. This paper describes the application and evolution of RE-AIM as well as lessons learned from its use. RE-AIM has been applied most often in public health and health behavior change research, but increasingly in more diverse content areas and within clinical, community, and corporate settings. We discuss challenges of using RE-AIM while encouraging a more pragmatic use of key dimensions rather than comprehensive applications of all elements. Current foci of RE-AIM include increasing the emphasis on cost and adaptations to programs and expanding the use of qualitative methods to understand “how” and “why” results came about. The framework will continue to evolve to focus on contextual and explanatory factors related to RE-AIM outcomes, package RE-AIM for use by non-researchers, and integrate RE-AIM with other pragmatic and reporting frameworks.

There is growing interest from multiple specialties, including internal medicine, to incorporate diagnostic point of care ultrasound (POCUS) into standard clinical care. However, few internists currently use POCUS. The objective of this study was to understand the current determinants of POCUS adoption at both the health system and clinician level at a U.S. academic medical center from the perspective of multi-level stakeholders. We performed semi-structured interviews of multi-level stakeholders including hospitalists, subspecialists, and hospital leaders at an academic medical center in the U.S. Questions regarding the determinants of POCUS adoption were asked of study participants. Using the framework method, team-based analysis of interview transcripts were guided by the contextual domains of the Practical Robust Implementation and Sustainability Model (PRISM). Thirty-one stakeholders with diverse roles in POCUS adoption were interviewed. Analysis of interviews revealed three overarching themes that stakeholders considered important to adoption by clinicians and health systems: clinical impact, efficiency and cost. Subthemes included two that were deemed essential to high-fidelity implementation: the development of credentialing policies and robust quality assurance processes. These findings identify potential determinants of system and clinician level adoption that may be leveraged to achieve high-fidelity implementation of POCUS applications that result in improved patient outcomes.

After publication of our article [1] the authors have notified us that the title for Figure 1 was incorrectly captioned.

Shared decision making (SDM) is not widely practiced in routine care due to a variety of organizational, provider, patient, and contextual factors. This article explores how implementation science—which encourages attention to the multilevel contextual factors that influence the adoption, implementation, and sustainment of health care practices—can provide useful insights for increasing SDM use in routine practice. We engaged with stakeholders representing different organizations and geographic locations over three phases: 1) multidisciplinary workgroup meeting comprising researchers and clinicians ( n = 11); 2) survey among a purposive sample of 47 patient advocates, clinicians, health care system leaders, funders, policymakers, and researchers; and 3) working session among diverse stakeholders ( n = 30). The workgroup meeting identified priorities for action and research, which included targeting multiple audiences and levels, shifting culture toward valuing and supporting SDM, and considering contextual factors influencing SDM implementation. Survey respondents provided recommendations for increasing adoption, implementation, and maintenance of SDM in practice including providing tools to support SDM, obtaining stakeholders’ involvement, and raising awareness of the importance of SDM. Stakeholders in the working session provided recommendations on the design of a guide for implementation of SDM in clinical settings, strategies to disseminate educational curricula on SDM, and strategies to influence policies to increase SDM use. These specific recommendations serve as a call to action to pursuing specific promising strategies aimed at increasing SDM use in practice and enhance understanding of the perspectives of diverse stakeholders at multiple levels from an implementation science perspective that appear fruitful for further study and application.

BackgroundMany health outcomes and implementation science studies have demonstrated the importance of tailoring evidence-based care interventions to local context to improve fit. By adapting to local culture, history, resources, characteristics, and priorities, interventions are more likely to lead to improved outcomes. However, it is unclear how best to adapt evidence-based programs and promising innovations. There are few guides or examples of how to best categorize or assess health-care adaptations, and even fewer that are brief and practical for use by non-researchers.Materials and methodsThis study describes the importance and potential of assessing adaptations before, during, and after the implementation of health systems interventions. We present a promising multilevel and multimethod approach developed and being applied across four different health systems interventions. Finally, we discuss implications and opportunities for future research.ResultsThe four case studies are diverse in the conditions addressed, interventions, and implementation strategies. They include two nurse coordinator-based transition of care interventions, a data and training-driven multimodal pain management project, and a cardiovascular patient-reported outcomes project, all of which are using audit and feedback. We used the same modified adaptation framework to document changes made to the interventions and implementation strategies. To create the modified framework, we started with the adaptation and modification model developed by Stirman and colleagues and expanded it by adding concepts from the RE-AIM framework. Our assessments address the intuitive domains of Who, How, When, What, and Why to classify and organize adaptations. For each case study, we discuss how the modified framework was operationalized, the multiple methods used to collect data, results to date and approaches utilized for data analysis. These methods include a real-time tracking system and structured interviews at key times during the intervention. We provide descriptive data on the types and categories of adaptations made and discuss lessons learned.ConclusionThe multimethod approaches demonstrate utility across diverse health systems interventions. The modified adaptations model adequately captures adaptations across the various projects and content areas. We recommend systematic documentation of adaptations in future clinical and public health research and have made our assessment materials publicly available.

The RE-AIM Framework is a planning and evaluation model that has been used in a variety of settings to address various programmatic, environmental, and policy innovations for improving population health. In addition to the broad application and diverse use of the framework, there are lessons learned and recommendations for the future use of the framework across clinical, community, and corporate settings. The purposes of this article are to: (A) provide a brief overview of the RE-AIM Framework and its pragmatic use for planning and evaluation; (B) offer recommendations to facilitate the application of RE-AIM in clinical, community, and corporate settings; and (C) share perspectives and lessons learned about employing RE-AIM dimensions in the planning, implementation, and evaluation phases within these different settings. In this article, we demonstrate how the RE-AIM concepts and elements within each dimension can be applied by researchers and practitioners in diverse settings, among diverse populations and for diverse health topics.