Search

Abstract Background Health guideline developers engage with interested people and groups to ensure that guidelines and their recommendations are relevant and useful to those who will be affected by them. These ‘interest‐holders’ include patients, payers/purchasers of health services, payers of health research, peer review editors, product makers, programme managers, policymakers, providers, principal investigators, and the public. The Guidelines International Network (GIN) and McMaster University Guideline Development Checklist describes 146 steps of the guideline process organized into 18 topics. While one topic focuses on engagement, it does not describe how to engage with interest‐holders. In addition, interest‐holder input could be sought throughout the guideline development process. This scoping review is part of a series of four related reviews. The three other reviews address barriers and facilitators to engagement in guideline development, managing conflicts of interest in guideline development, and assessing the impact of interest‐holder engagement on guideline development. The four reviews will inform the development of guidance for multi‐interest‐holder engagement in guideline development; the GIN‐McMaster Guideline Development Checklist Extension for Engagement. Objectives The objective of this scoping review is to identify, describe, and summarise existing guidance and methods for multi‐interest‐holder engagement throughout the health guideline development process. Search Methods We conducted one comprehensive search for studies of engagement in guidelines to meet the inclusion criteria of one or more of the four systematic reviews in this series. We searched MEDLINE (OVID), CINAHL (EBSCO), EMBASE (OVID), PsycInfo (OVID) and SCOPUS databases up to September 2022. We did not include limits for date, study design, or language. We searched websites of agencies and organizations that engage interest‐holder groups, such as the Agency for Healthcare Research and Quality (AHRQ), CIHR Strategy for Patient‐Oriented Research (SPOR), National Institute for Health and Care Research (NIHR) Be Part of Research, Guidelines International Network (G‐I‐N), the National Institute for Health and Care Excellence, and the PatientCentred Outcomes Research Institute (PCORI). We handsearched the websites of guideline producing agencies. We solicited additional grey literature from the members of the MuSE Consortium. Selection Criteria Studies were included in this review if they reported on engagement of any of our identified groups, patients, payers/funders of research, payers/purchasers of health services, policymakers, programme managers, providers, principal investigators/researchers, peer review editors, product makers in the development of a health guideline. Titles and abstracts of identified citations were screened independently, in duplicate. The full text of potentially relevant papers were screened for eligibility into one or more of the four reviews in the series. Screening was done independently, by two reviewers. The team held weekly meetings with all reviewers involved in screening to discuss and resolve conflicts. Data Collection and Analysis Two reviewers extracted relevant data into a pilot‐tested data extraction form using Excel. We used the GIN‐McMaster guideline development checklist as a framework for extracting the available guidance for each of our identified interest‐holder groups throughout the development process. We presented descriptive statistics of the number of papers reporting guidance for each groups across the steps of the guideline process. We synthesized the relevant text using a qualitative meta‐summary approach. Main Results We included 16 papers (from 17 reports). These papers were from Australia, Denmark, the Netherlands, the UK, and the USA, and eight papers were international (countries not specified). The papers provided guidance for at least one of our interest‐holder groups for at least one stage of guideline development. We mapped this guidance to the GIN‐McMaster Guideline Development Checklist to identify the available guidance for each of our interest‐holder groups across all stages of the guideline development process. Guidance was available for patient engagement in 15 of the 16 papers. At least two papers provided guidance for each of the 18 topics of the GIN‐McMaster Guideline Development Checklist. For healthcare providers, 9 papers provided guidance for their engagement across 10 of the 18 guideline development topics. Guidance for engaging with the public was provided for 14 of the 18 topics and reported in 4 of our included papers. For payers/purchasers of health services, policymakers, product makers, programme managers, and principal investigators, 2–3 papers provided guidance for these groups across 4–7 topics of the GIN‐McMaster checklist. We did not identify any specific guidance for payers of health research or for editors of peer‐reviewed journals. Authors' Conclusions Guidance for interst‐holder engagement in guidelines is available but has focused primarily on patients. We will utilize the guidance identified in this scoping review to inform the GIN‐McMaster Guideline Development Checklist Extension for engagement. Combined with the information obtained from the other systematic reviews in this series, we will address the gaps in guidance for the other identified interest‐holder groups.

Abstract This is the protocol for a Campbell systematic review. The objectives are as follows. The objective of this review is to identify and synthesize empirical research on the impacts of interest‐holder engagement on the guideline development process and content. Our research questions are as follows: (1) What are the empirical examples of impact on the process in health guideline development across any of the 18 steps of the GIN‐McMaster checklist? (2) What are the empirical examples of impact on the content in health guideline development across any of the 18 steps of the GIN‐McMaster checklist?

Aim: Although the US FDA encourages manufacturers of medical devices to submit real-world evidence (RWE) to support regulatory decisions, the ability of real-world data (RWD) to generate evidence suitable for decision making remains unclear. The 2017 Medical Device User Fee Amendments (MDUFA IV), authorized the National Evaluation System for health Technology Coordinating Center (NESTcc) to conduct pilot projects, or ‘Test-Cases’, to assess whether current RWD captures the information needed to answer research questions proposed by industry stakeholders. We synthesized key lessons about the challenges conducting research with RWD and the strategies used by research teams to enhance their ability to generate evidence from RWD based on 18 Test-Cases conducted between 2020 and 2022. Materials & methods: We reviewed study protocols and reports from each Test-Case team and conducted 49 semi-structured interviews with representatives of participating organizations. Interview transcripts were coded and thematically analyzed. Results: Challenges that stakeholders encountered in workingwith RWD included the lack of unique device identifiers, capturing key data elements and their appropriate meaning in structured data, limited reliability of diagnosis and procedure codes in structured data, extracting information from unstructured electronic health record (EHR) data, limited capture of long-term study end points, missing data and data sharing. Successful strategies included using manufacturer and supply chain data, leveraging clinical registries and registry reporting processes to collect and aggregate data, querying standardized EHR data, implementing natural language processing algorithms and using multidisciplinary research teams. Conclusion: The Test-Cases identified numerous challenges working with RWD but also opportunities to address these challenges and improve researchers’ ability to use RWD to generate evidence on medical devices.

Abstract Background The World Trade Center (WTC) Health Program (“Program”) seeks to assess the inventory, quality, and impact of its funded research in the context of all clinical and translational research involving WTC populations. This paper presents a protocol for ongoing scoping reviews of WTC-related health research. Methods Using terms relevant to the September 11 attacks, we will search OVID MEDLINE, PsycINFO, Scopus, Web of Science, CINAHL, and Embase for records of peer-reviewed publications. Title, abstract, and full text screening will be used to exclude records according to a priori criteria. Data abstraction will be performed on all articles that meet inclusion criteria using a standardized query form that was developed in collaboration with NIOSH. A team of reviewers will be trained to abstract data from included articles. Articles will be double-reviewed, and disagreements will be adjudicated. Results We will summarize existing research involving WTC populations. The summary will assess the extent, nature, and signals of impact of WTC-related health research. Conclusions Our review will lay the groundwork for additional study of research impact by identifying population, clinical, and translational topics that can be assessed through future focused reviews. It will also support planning activities by Program policy makers and stakeholders as they work to achieve the Program’s research goals. Systematic review registration This publication serves as documentation of the protocol.

Abstract Background Involving collaborators and partners in research may increase relevance and uptake, while reducing health and social inequities. Collaborators and partners include people and groups interested in health research: health care providers, patients and caregivers, payers of health research, payers of health services, publishers, policymakers, researchers, product makers, program managers, and the public. Evidence syntheses inform decisions about health care services, treatments, and practice, which ultimately affect health outcomes. Our objectives are to: A. Identify, map, and synthesize qualitative and quantitative findings related to engagement in evidence syntheses B. Explore how engagement in evidence synthesis promotes health equity C. Develop equity-oriented guidance on methods for conducting, evaluating, and reporting engagement in evidence syntheses Methods Our diverse, international team will develop guidance for engagement with collaborators and partners throughout multiple sequential steps using an integrated knowledge translation approach: 1. Reviews. We will co-produce 1 scoping review, 3 systematic reviews and 1 evidence map focusing on (a) methods, (b) barriers and facilitators, (c) conflict of interest considerations, (d) impacts, and (e) equity considerations of engagement in evidence synthesis. 2. Methods study, interviews, and survey. We will contextualise the findings of step 1 by assessing a sample of evidence syntheses reporting on engagement with collaborators and partners and through conducting interviews with collaborators and partners who have been involved in producing evidence syntheses. We will use these findings to develop draft guidance checklists and will assess agreement with each item through an international survey. 3. Consensus. The guidance checklists will be co-produced and finalised at a consensus meeting with collaborators and partners. 4. Dissemination. We will develop a dissemination plan with our collaborators and partners and work collaboratively to improve adoption of our guidance by key organizations. Conclusion Our international team will develop guidance for collaborator and partner engagement in health care evidence syntheses. Incorporating partnership values and expectations may result in better uptake, potentially reducing health inequities.

Abstract Established in 2015, the Multi-Stakeholder Engagement (MuSE) Consortium is an international network of over 120 individuals interested in stakeholder engagement in research and guidelines. The MuSE group is developing guidance for stakeholder engagement in the development of health and healthcare guideline development. The development of this guidance has included multiple meetings with stakeholders, including patients, payers/purchasers of health services, peer review editors, policymakers, program managers, providers, principal investigators, product makers, the public, and purchasers of health services and has identified a number of key issues. These include: (1) Definitions, roles, and settings (2) Stakeholder identification and selection (3) Levels of engagement, (4) Evaluation of engagement, (5) Documentation and transparency, and (6) Conflict of interest management. In this paper, we discuss these issues and our plan to develop guidance to facilitate stakeholder engagement in all stages of the development of health and healthcare guideline development.

OBJECTIVES/GOALS: Community and other stakeholder engagement (CSE) is critical for relevant and equitable clinical research, yet implementation poses challenges. This study delineates the perspectives of scientists and diverse stakeholders regarding facilitators and challenges in CSE, its perceived value, and their recommendations for successful CSE. METHODS/STUDY POPULATION: The Tufts CTSI Pilot Studies Program requires applicants to propose a plan for CSE while implementing the award, including which stakeholders (SHs)—community members, clinicians, and others affected by the research--will be involved and at what stages. This qualitative study assessed the experiences of both Principal Investigators (PIs) and SHs engaged in pilot projects from three cohorts of awardees (2019-21). Recruitment targeted one PI and one SH per project. Semi-structured interviews explored their CSE experiences, including facilitators, challenges, meaningfulness, perceived impact, intent to participate in CSE in future studies, as well as recommendations for funders, research support organizations, and investigators. Inductive consensus-based coding and thematic analysis was employed. RESULTS/ANTICIPATED RESULTS: Fourteen PIs from different pilot projects and a SH from five of these projects participated. Almost all PIs (92%) had over six years of experience, but two-thirds (67%) had little or no experience with CSE. Four SHs self-identified as representatives of community organizations and one as a clinician scientist. CSE was a “win-win” for both PIs and SHs, and all PIs intended to involve SHs in other research studies. Three facilitators were identified as fostering effective CSE (e.g., PI access to CSE expertise while conducting the project), while four challenges hindered it (e.g., limits on SH capacity and CSE funding). SHs advised scientists to build authentic, sustained relationships, and PIs and SHs provided three actionable recommendations for funders and research support organizations to deepen and expand CSE. DISCUSSION/SIGNIFICANCE: Perspectives of scientists and SHs engaged in research projects are vital for expanding and sustaining effective CSE in research. Funders and research support organizations can enhance their strategies for CSE integration in clinical and translational research by incorporating these diverse views to ensure the research achieves maximal impact.

Aim: We developed the Patient-Engaged Health Technology Assessment strategy for survey-based goal collection from patients to yield patient-important outcomes suitable for use in multi-criteria decision analysis. Methods: Rheumatoid arthritis patients were recruited from online patient networks for proofof- concept testing of goal collection and prioritization using a survey. A Project Steering Committee and Expert Panel rated the feasibility of scaling to larger samples. Results: Survey respondents (n = 47) completed the goal collection exercise. Finding effective treatments was rated by respondents as the most important goal, and reducing stiffness was rated as the least important. Feedback from our steering committee and expert panel support the approach’s feasibility for goal identification and ranking. Conclusion: Goals relevant for treatment evaluation can be identified and rated for importance by patients to permit wide input from patients with lived experience of disease.

OBJECTIVES/GOALS: The training needs of clinical & translational scientists are evolving. Implementation of new curriculum content requires assessment of need, fit with current curriculum, incentives and barriers to implementation. We used implementation science methods to plan the implementation of a dissemination and implementation science training toolkit. METHODS/STUDY POPULATION: The Tufts Clinical & Translational Science (CTS) Graduate Program is the training core of the Tufts CTSI and its associated TL1. To assess barriers and facilitators to implementing the University of Washington’s Dissemination and Implementation (D&I) Science Toolkit for TL1 Programs into the Tufts CTS Graduate Program, we used the Consolidated Framework for Implementation Research (CFIR). We developed an interview guide based on CFIR constructs or categories of factors that might influence implementation of the D&I Toolkit and interviewed 8 stakeholders including leadership, trainees and faculty. We compared D&I Toolkit content to the current curriculum on stakeholder engagement (SE) and research study quality improvement (QI) to identify how the D&I Toolkit could be adapted to fit trainee and program needs. RESULTS/ANTICIPATED RESULTS: Stakeholder interviews and review of the current curriculum identified facilitators and barriers to implementing the D&I Toolkit in the Tufts CTS Graduate Program. The program has a strong SE focus and fosters the use of research study QI methods. Interviews identified student and faculty desire to implement the D&I content yet time constraints and overlap with the current curriculum were identified. The following plan to adapt the D&I Toolkit content to fit contextual factors was developed and executed: 1. Adapt didactic content to fit into a 90-min session. 2. Highlight the synergy between the current SE curriculum and the role of stakeholders in D&I science. 3. Modify course materials to highlight examples of disseminating local research results. 4. Limit the experiential learning component to a SE plan. DISCUSSION/SIGNIFICANCE: TL1 training programs must balance competing demands in selecting curriculum content. The CFIR framework was used to systematically assess implementation barriers by engaging program faculty, students and leaders to identify implementation strategies. This process could be useful when evaluating the addition of other educational content.

Abstract Background There is a need for the development of comprehensive, global, evidence‐based guidance for stakeholder engagement in guideline development. Stakeholders are any individual or group who is responsible for or affected by health‐ and healthcare‐related decisions. This includes patients, the public, providers of health care and policymakers for example. As part of the guidance development process, Multi‐Stakeholder Engagement (MuSE) Consortium set out to conduct four concurrent systematic reviews to summarise the evidence on: (1) existing guidance for stakeholder engagement in guideline development, (2) barriers and facilitators to stakeholder engagement in guideline development, (3) managing conflicts of interest in stakeholder engagement in guideline development and (4) measuring the impact of stakeholder engagement in guideline development. This protocol addresses the second systematic review in the series. Objectives The objective of this review is to identify and synthesise the existing evidence on barriers and facilitators to stakeholder engagement in health guideline development. We will address this objective through two research questions: (1) What are the barriers to multi‐stakeholder engagement in health guideline development across any of the 18 steps of the GIN‐McMaster checklist? (2) What are the facilitators to multi‐stakeholder engagement in health guideline development across any of the 18 steps of the GIN‐McMaster checklist? Search Methods A comprehensive search strategy will be developed and peer‐reviewed in consultation with a medical librarian. We will search the following databases: MEDLINE, Cumulative Index to Nursing & Allied Health Literature (CINAHL), EMBASE, PsycInfo, Scopus, and Sociological Abstracts. To identify grey literature, we will search the websites of agencies who actively engage stakeholder groups such as the AHRQ, Canadian Institutes of Health Research (CIHR) Strategy for Patient‐Oriented Research (SPOR), INVOLVE, the National Institute for Health and Care Excellence (NICE) and the PCORI. We will also search the websites of guideline‐producing agencies, such as the American Academy of Pediatrics, Australia's National Health Medical Research Council (NHMRC) and the WHO. We will invite members of the team to suggest grey literature sources and we plan to broaden the search by soliciting suggestions via social media, such as Twitter. Selection Criteria We will include empirical qualitative and mixed‐method primary research studies which qualitatively report on the barriers or facilitators to stakeholder engagement in health guideline development. The population of interest is stakeholders in health guideline development. Building on previous work, we have identified 13 types of stakeholders whose input can enhance the relevance and uptake of guidelines: Patients, caregivers and patient advocates; Public; Providers of health care; Payers of health services; Payers of research; Policy makers; Program managers; Product makers; Purchasers; Principal investigators and their research teams; and Peer‐review editors/publishers. Eligible studies must describe stakeholder engagement at any of the following steps of the GIN‐McMaster Checklist for Guideline Development. Data Collection and Analysis All identified citations from electronic databases will be imported into Covidence software for screening and selection. Documents identified through our grey literature search will be managed and screened using an Excel spreadsheet. A two‐part study selection process will be used for all identified citations: (1) a title and abstract review and (2) full‐text review. At each stage, teams of two review authors will independently assess all potential studies in duplicate using a priori inclusion and exclusion criteria. Data will be extracted by two review authors independently and in duplicate according to a standardised data extraction form. Main Results The results of this review will be used to inform the development of guidance for multi‐stakeholder engagement in guideline development and implementation. This guidance will be official GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group guidance. The GRADE system is internationally recognised as a standard for guideline development. The findings of this review will assist organisations who develop healthcare, public health and health policy guidelines, such as the World Health Organization, to involve multiple stakeholders in the guideline development process to ensure the development of relevant, high quality and transparent guidelines.