Search

Aim: Although the US FDA encourages manufacturers of medical devices to submit real-world evidence (RWE) to support regulatory decisions, the ability of real-world data (RWD) to generate evidence suitable for decision making remains unclear. The 2017 Medical Device User Fee Amendments (MDUFA IV), authorized the National Evaluation System for health Technology Coordinating Center (NESTcc) to conduct pilot projects, or ‘Test-Cases’, to assess whether current RWD captures the information needed to answer research questions proposed by industry stakeholders. We synthesized key lessons about the challenges conducting research with RWD and the strategies used by research teams to enhance their ability to generate evidence from RWD based on 18 Test-Cases conducted between 2020 and 2022. Materials & methods: We reviewed study protocols and reports from each Test-Case team and conducted 49 semi-structured interviews with representatives of participating organizations. Interview transcripts were coded and thematically analyzed. Results: Challenges that stakeholders encountered in workingwith RWD included the lack of unique device identifiers, capturing key data elements and their appropriate meaning in structured data, limited reliability of diagnosis and procedure codes in structured data, extracting information from unstructured electronic health record (EHR) data, limited capture of long-term study end points, missing data and data sharing. Successful strategies included using manufacturer and supply chain data, leveraging clinical registries and registry reporting processes to collect and aggregate data, querying standardized EHR data, implementing natural language processing algorithms and using multidisciplinary research teams. Conclusion: The Test-Cases identified numerous challenges working with RWD but also opportunities to address these challenges and improve researchers’ ability to use RWD to generate evidence on medical devices.

Abstract Background The World Trade Center (WTC) Health Program (“Program”) seeks to assess the inventory, quality, and impact of its funded research in the context of all clinical and translational research involving WTC populations. This paper presents a protocol for ongoing scoping reviews of WTC-related health research. Methods Using terms relevant to the September 11 attacks, we will search OVID MEDLINE, PsycINFO, Scopus, Web of Science, CINAHL, and Embase for records of peer-reviewed publications. Title, abstract, and full text screening will be used to exclude records according to a priori criteria. Data abstraction will be performed on all articles that meet inclusion criteria using a standardized query form that was developed in collaboration with NIOSH. A team of reviewers will be trained to abstract data from included articles. Articles will be double-reviewed, and disagreements will be adjudicated. Results We will summarize existing research involving WTC populations. The summary will assess the extent, nature, and signals of impact of WTC-related health research. Conclusions Our review will lay the groundwork for additional study of research impact by identifying population, clinical, and translational topics that can be assessed through future focused reviews. It will also support planning activities by Program policy makers and stakeholders as they work to achieve the Program’s research goals. Systematic review registration This publication serves as documentation of the protocol.

Abstract Background Involving collaborators and partners in research may increase relevance and uptake, while reducing health and social inequities. Collaborators and partners include people and groups interested in health research: health care providers, patients and caregivers, payers of health research, payers of health services, publishers, policymakers, researchers, product makers, program managers, and the public. Evidence syntheses inform decisions about health care services, treatments, and practice, which ultimately affect health outcomes. Our objectives are to: A. Identify, map, and synthesize qualitative and quantitative findings related to engagement in evidence syntheses B. Explore how engagement in evidence synthesis promotes health equity C. Develop equity-oriented guidance on methods for conducting, evaluating, and reporting engagement in evidence syntheses Methods Our diverse, international team will develop guidance for engagement with collaborators and partners throughout multiple sequential steps using an integrated knowledge translation approach: 1. Reviews. We will co-produce 1 scoping review, 3 systematic reviews and 1 evidence map focusing on (a) methods, (b) barriers and facilitators, (c) conflict of interest considerations, (d) impacts, and (e) equity considerations of engagement in evidence synthesis. 2. Methods study, interviews, and survey. We will contextualise the findings of step 1 by assessing a sample of evidence syntheses reporting on engagement with collaborators and partners and through conducting interviews with collaborators and partners who have been involved in producing evidence syntheses. We will use these findings to develop draft guidance checklists and will assess agreement with each item through an international survey. 3. Consensus. The guidance checklists will be co-produced and finalised at a consensus meeting with collaborators and partners. 4. Dissemination. We will develop a dissemination plan with our collaborators and partners and work collaboratively to improve adoption of our guidance by key organizations. Conclusion Our international team will develop guidance for collaborator and partner engagement in health care evidence syntheses. Incorporating partnership values and expectations may result in better uptake, potentially reducing health inequities.

Abstract Established in 2015, the Multi-Stakeholder Engagement (MuSE) Consortium is an international network of over 120 individuals interested in stakeholder engagement in research and guidelines. The MuSE group is developing guidance for stakeholder engagement in the development of health and healthcare guideline development. The development of this guidance has included multiple meetings with stakeholders, including patients, payers/purchasers of health services, peer review editors, policymakers, program managers, providers, principal investigators, product makers, the public, and purchasers of health services and has identified a number of key issues. These include: (1) Definitions, roles, and settings (2) Stakeholder identification and selection (3) Levels of engagement, (4) Evaluation of engagement, (5) Documentation and transparency, and (6) Conflict of interest management. In this paper, we discuss these issues and our plan to develop guidance to facilitate stakeholder engagement in all stages of the development of health and healthcare guideline development.

OBJECTIVES/GOALS: Community and other stakeholder engagement (CSE) is critical for relevant and equitable clinical research, yet implementation poses challenges. This study delineates the perspectives of scientists and diverse stakeholders regarding facilitators and challenges in CSE, its perceived value, and their recommendations for successful CSE. METHODS/STUDY POPULATION: The Tufts CTSI Pilot Studies Program requires applicants to propose a plan for CSE while implementing the award, including which stakeholders (SHs)—community members, clinicians, and others affected by the research--will be involved and at what stages. This qualitative study assessed the experiences of both Principal Investigators (PIs) and SHs engaged in pilot projects from three cohorts of awardees (2019-21). Recruitment targeted one PI and one SH per project. Semi-structured interviews explored their CSE experiences, including facilitators, challenges, meaningfulness, perceived impact, intent to participate in CSE in future studies, as well as recommendations for funders, research support organizations, and investigators. Inductive consensus-based coding and thematic analysis was employed. RESULTS/ANTICIPATED RESULTS: Fourteen PIs from different pilot projects and a SH from five of these projects participated. Almost all PIs (92%) had over six years of experience, but two-thirds (67%) had little or no experience with CSE. Four SHs self-identified as representatives of community organizations and one as a clinician scientist. CSE was a “win-win” for both PIs and SHs, and all PIs intended to involve SHs in other research studies. Three facilitators were identified as fostering effective CSE (e.g., PI access to CSE expertise while conducting the project), while four challenges hindered it (e.g., limits on SH capacity and CSE funding). SHs advised scientists to build authentic, sustained relationships, and PIs and SHs provided three actionable recommendations for funders and research support organizations to deepen and expand CSE. DISCUSSION/SIGNIFICANCE: Perspectives of scientists and SHs engaged in research projects are vital for expanding and sustaining effective CSE in research. Funders and research support organizations can enhance their strategies for CSE integration in clinical and translational research by incorporating these diverse views to ensure the research achieves maximal impact.

Aim: We developed the Patient-Engaged Health Technology Assessment strategy for survey-based goal collection from patients to yield patient-important outcomes suitable for use in multi-criteria decision analysis. Methods: Rheumatoid arthritis patients were recruited from online patient networks for proofof- concept testing of goal collection and prioritization using a survey. A Project Steering Committee and Expert Panel rated the feasibility of scaling to larger samples. Results: Survey respondents (n = 47) completed the goal collection exercise. Finding effective treatments was rated by respondents as the most important goal, and reducing stiffness was rated as the least important. Feedback from our steering committee and expert panel support the approach’s feasibility for goal identification and ranking. Conclusion: Goals relevant for treatment evaluation can be identified and rated for importance by patients to permit wide input from patients with lived experience of disease.

OBJECTIVES/GOALS: The training needs of clinical & translational scientists are evolving. Implementation of new curriculum content requires assessment of need, fit with current curriculum, incentives and barriers to implementation. We used implementation science methods to plan the implementation of a dissemination and implementation science training toolkit. METHODS/STUDY POPULATION: The Tufts Clinical & Translational Science (CTS) Graduate Program is the training core of the Tufts CTSI and its associated TL1. To assess barriers and facilitators to implementing the University of Washington’s Dissemination and Implementation (D&I) Science Toolkit for TL1 Programs into the Tufts CTS Graduate Program, we used the Consolidated Framework for Implementation Research (CFIR). We developed an interview guide based on CFIR constructs or categories of factors that might influence implementation of the D&I Toolkit and interviewed 8 stakeholders including leadership, trainees and faculty. We compared D&I Toolkit content to the current curriculum on stakeholder engagement (SE) and research study quality improvement (QI) to identify how the D&I Toolkit could be adapted to fit trainee and program needs. RESULTS/ANTICIPATED RESULTS: Stakeholder interviews and review of the current curriculum identified facilitators and barriers to implementing the D&I Toolkit in the Tufts CTS Graduate Program. The program has a strong SE focus and fosters the use of research study QI methods. Interviews identified student and faculty desire to implement the D&I content yet time constraints and overlap with the current curriculum were identified. The following plan to adapt the D&I Toolkit content to fit contextual factors was developed and executed: 1. Adapt didactic content to fit into a 90-min session. 2. Highlight the synergy between the current SE curriculum and the role of stakeholders in D&I science. 3. Modify course materials to highlight examples of disseminating local research results. 4. Limit the experiential learning component to a SE plan. DISCUSSION/SIGNIFICANCE: TL1 training programs must balance competing demands in selecting curriculum content. The CFIR framework was used to systematically assess implementation barriers by engaging program faculty, students and leaders to identify implementation strategies. This process could be useful when evaluating the addition of other educational content.

Abstract Background There is a need for the development of comprehensive, global, evidence‐based guidance for stakeholder engagement in guideline development. Stakeholders are any individual or group who is responsible for or affected by health‐ and healthcare‐related decisions. This includes patients, the public, providers of health care and policymakers for example. As part of the guidance development process, Multi‐Stakeholder Engagement (MuSE) Consortium set out to conduct four concurrent systematic reviews to summarise the evidence on: (1) existing guidance for stakeholder engagement in guideline development, (2) barriers and facilitators to stakeholder engagement in guideline development, (3) managing conflicts of interest in stakeholder engagement in guideline development and (4) measuring the impact of stakeholder engagement in guideline development. This protocol addresses the second systematic review in the series. Objectives The objective of this review is to identify and synthesise the existing evidence on barriers and facilitators to stakeholder engagement in health guideline development. We will address this objective through two research questions: (1) What are the barriers to multi‐stakeholder engagement in health guideline development across any of the 18 steps of the GIN‐McMaster checklist? (2) What are the facilitators to multi‐stakeholder engagement in health guideline development across any of the 18 steps of the GIN‐McMaster checklist? Search Methods A comprehensive search strategy will be developed and peer‐reviewed in consultation with a medical librarian. We will search the following databases: MEDLINE, Cumulative Index to Nursing & Allied Health Literature (CINAHL), EMBASE, PsycInfo, Scopus, and Sociological Abstracts. To identify grey literature, we will search the websites of agencies who actively engage stakeholder groups such as the AHRQ, Canadian Institutes of Health Research (CIHR) Strategy for Patient‐Oriented Research (SPOR), INVOLVE, the National Institute for Health and Care Excellence (NICE) and the PCORI. We will also search the websites of guideline‐producing agencies, such as the American Academy of Pediatrics, Australia's National Health Medical Research Council (NHMRC) and the WHO. We will invite members of the team to suggest grey literature sources and we plan to broaden the search by soliciting suggestions via social media, such as Twitter. Selection Criteria We will include empirical qualitative and mixed‐method primary research studies which qualitatively report on the barriers or facilitators to stakeholder engagement in health guideline development. The population of interest is stakeholders in health guideline development. Building on previous work, we have identified 13 types of stakeholders whose input can enhance the relevance and uptake of guidelines: Patients, caregivers and patient advocates; Public; Providers of health care; Payers of health services; Payers of research; Policy makers; Program managers; Product makers; Purchasers; Principal investigators and their research teams; and Peer‐review editors/publishers. Eligible studies must describe stakeholder engagement at any of the following steps of the GIN‐McMaster Checklist for Guideline Development. Data Collection and Analysis All identified citations from electronic databases will be imported into Covidence software for screening and selection. Documents identified through our grey literature search will be managed and screened using an Excel spreadsheet. A two‐part study selection process will be used for all identified citations: (1) a title and abstract review and (2) full‐text review. At each stage, teams of two review authors will independently assess all potential studies in duplicate using a priori inclusion and exclusion criteria. Data will be extracted by two review authors independently and in duplicate according to a standardised data extraction form. Main Results The results of this review will be used to inform the development of guidance for multi‐stakeholder engagement in guideline development and implementation. This guidance will be official GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group guidance. The GRADE system is internationally recognised as a standard for guideline development. The findings of this review will assist organisations who develop healthcare, public health and health policy guidelines, such as the World Health Organization, to involve multiple stakeholders in the guideline development process to ensure the development of relevant, high quality and transparent guidelines.

Abstract Background Stakeholder engagement has become widely accepted as a necessary component of guideline development and implementation. While frameworks for developing guidelines express the need for those potentially affected by guideline recommendations to be involved in their development, there is a lack of consensus on how this should be done in practice. Further, there is a lack of guidance on how to equitably and meaningfully engage multiple stakeholders. We aim to develop guidance for the meaningful and equitable engagement of multiple stakeholders in guideline development and implementation. Methods This will be a multi-stage project. The first stage is to conduct a series of four systematic reviews. These will (1) describe existing guidance and methods for stakeholder engagement in guideline development and implementation, (2) characterize barriers and facilitators to stakeholder engagement in guideline development and implementation, (3) explore the impact of stakeholder engagement on guideline development and implementation, and (4) identify issues related to conflicts of interest when engaging multiple stakeholders in guideline development and implementation. Discussion We will collaborate with our multiple and diverse stakeholders to develop guidance for multi-stakeholder engagement in guideline development and implementation. We will use the results of the systematic reviews to develop a candidate list of draft guidance recommendations and will seek broad feedback on the draft guidance via an online survey of guideline developers and external stakeholders. An invited group of representatives from all stakeholder groups will discuss the results of the survey at a consensus meeting which will inform the development of the final guidance papers. Our overall goal is to improve the development of guidelines through meaningful and equitable multi-stakeholder engagement, and subsequently to improve health outcomes and reduce inequities in health.

IntroductionStakeholder engagement can play an important role in increasing public trust and the understanding of scientific research and its impact. Frameworks for stakeholder identification exist, but these frameworks may not apply well to basic science and early stage translational research.MethodsFour Clinical and Translational Science Award (CTSA) hubs led six focus groups and two semi-structured interviews using a semi-structured discussion guide to learn from basic science researchers about stakeholder engagement in their work. The 24 participants represented fourteen clinical and academic disciplines.ResultsEarly stage translational researchers reported engagement with a broad array of stakeholders. Those whose research has a clinical focus reported working with a more diverse range of stakeholders than those whose work did not. Common barriers to stakeholder engagement were grouped into three major themes: a poor definition of concepts, absence of guidance, and limited resources.DiscussionThe National Center for Advancing Translational Sciences (NCATS), the consortium of CTSAs, and the individual CTSA "hubs" are three actors that can help early stage translational researchers develop shared terms of reference, build the necessary skills, and assemble the appropriate resources for engaging stakeholders in Clinical and Translational Research. Getting this right will involve a coordinated push by all three entities.

Abstract An important challenge in comparative effectiveness research is the lack of infrastructure to support pragmatic clinical trials, which compare interventions in usual practice settings and subjects. These trials present challenges that differ from those of classical efficacy trials, which are conducted under ideal circumstances, in patients selected for their suitability, and with highly controlled protocols. In 2012, we launched a 1‐year learning network to identify high‐priority pragmatic clinical trials and to deploy research infrastructure through the NIH Clinical and Translational Science Awards Consortium that could be used to launch and sustain them. The network and infrastructure were initiated as a learning ground and shared resource for investigators and communities interested in developing pragmatic clinical trials. We followed a three‐stage process of developing the network, prioritizing proposed trials, and implementing learning exercises that culminated in a 1‐day network meeting at the end of the year. The year‐long project resulted in five recommendations related to developing the network, enhancing community engagement, addressing regulatory challenges, advancing information technology, and developing research methods. The recommendations can be implemented within 24 months and are designed to lead toward a sustained national infrastructure for pragmatic trials.