Search

Objective: The evaluation of health benefits and harms of an intervention with GRADE Evidence to Decision (EtD) frameworks includes judgments if the effects are "trivial", "small", "moderate" or "large". Such judgments ideally require the a priori establishment of decision thresholds (DTs), whose empirical derivation for single outcomes has been previously described. In this article, we provide a methodological approach to estimate DTs for composite endpoints based on disutilities., Study Design and Setting: We generated an approach that involves the computation of pooled disutilities, in which the disutility of each outcome comprised in the composite endpoint is weighted by the respective relative frequency. We also applied a modelling approach based on probability distributions to account for uncertainty in the estimates. We present a practical example of the determination of DTs associated with the development of at least one adverse event following treatment with intranasal medications for rhinitis that we used in the Allergic Rhinitis in Asthma (ARIA) guidelines., Results: We provide the methodological steps of a modelling-based approach to compute pooled disutilities and, as a result, DTs for composite endpoints. We have developed a webtool (https://compositedt.med.up.pt/) that allows for a simple implementation of the proposed approach. Applying our approach to a practical example, we concluded that rhinitis nasal medications, compared to placebo, were associated with a trivial harm from adverse events., Conclusion: We propose an approach for estimating DTs for composite endpoints, which may be particularly valuable whenever composite endpoints are used for clinical research, clinical practice and decision-making., (Copyright © 2024. Published by Elsevier Inc.)

Introduction: GRADE and other Evidence to Decision (EtD) frameworks are widely used by guideline development groups (GDG) and other decision-makers. When GDGs judge the magnitude of desirable and undesirable health outcomes on EtDs, they typically categorize them as trivial, small, moderate or large. However, generic judgment or decision thresholds (DTs) that could guide the user about such estimates of effect size or serve as references for interpretation of findings are not yet available. The objective of this study was to empirically derive DTs for EtD judgments about the magnitude of dichotomously assessed health benefits and harms., Methods: We conducted a methodological randomised controlled trial to derive empirical DTs across conditions and health outcomes. We invited stakeholders including clinicians, epidemiologists, decision scientists, health research methodologists, experts in Health Technology Assessment (HTA), members of GDGs, patient representatives, and the public to participate in the trial. We employed randomly assigned case scenarios to elicit ranges of absolute risk differences (ARD) judged as small and moderate effects from study participants. We then used the collected data to derive empirical decision thresholds. We also investigated the validity of our DTs by measuring the agreement between judgments that were made by GDGs in the past and the judgments that our DTs approach would suggest if applied to the same guideline data., Results: A total of 445 stakeholders accessed the survey of which 409 were randomised and 288 rated at least one case scenario. Based on these participants, the study findings support our a priori hypothesis of a difference in the DTs for trivial, small, moderate, and large effects and are suggestive of a relation between raters' judgments and the joint measure of absolute effects and outcome values. The results permit the use and calculation of DTs for a variety of scenarios and we present three ways of how to use the results practically., Conclusions: In this trial we confirmed that empirically derived DTs discriminate between judgments on the EtDs. These DTs can be used for judgments about desirable and undesirable health effects in systematic reviews or to initiate and inform a discussion with a GDG. This ensures consistency in judgments across different guideline questions and promotes transparency in judgments., (Copyright © 2024. Published by Elsevier Inc.)

Objectives: The PANELVIEW questionnaire identifies the strengths and weaknesses of the process and methods used for developing health guidelines from the guideline development group's perspective. To expand its use, PANELVIEW ideally should be available in different languages. We aimed to cross-culturally adapt PANELVIEW into Spanish and assess its acceptability, validity, and reliability., Study Design and Setting: To translate and culturally adapt PANELVIEW to Spanish, we followed International Society for Pharmacoeconomics and Outcomes Research's Translation and Cultural Adaptation Good Practice Principles guidelines. The process consisted of 1) forward and back translation, 2) input from an expert panel, and 3) cognitive debriefing interviews. We assessed the content validity with experts in guideline development who rated instrument items for relevance to determine the item content validity index and scale content validity index (I-CVI and S-CVI). We tested the reliability with health guidelines panels from Spanish-speaking countries and measured internal consistency (Cronbach's alpha). We examined acceptability through the number of missing responses for each item., Results: The content comparison between the back-translation and the original version showed that most items (24/34) were conceptually equivalent but with grammatical differences. Through the cognitive interviews, we identified six items with wording issues, ten with clarity issues, and two with applicability issues. I-CVI ranged from 0.77 to 1.00, with two items needing revision. S-CVI was 0.92, showing excellent content validity. The PANELVIEW Spanish version demonstrated very good reliability (Cronbach's alpha coefficient of 0.96). Panel members responded to all items, showing good acceptability., Conclusion: The PANELVIEW Spanish version was conceptually equivalent to the original version and provided satisfactory evidence of acceptability, validity and reliability., Competing Interests: Declaration of competing interest W.W., Elie Akl and H.S. will retain the copyright of the PANELVIEW tool through McMaster University (Technology ID 20-031), and the tool may be licensed in the future. There are no competing interests for any other author., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Background and Objective: Grading of Recommendations, Assessment, Development, and Evaluation (GRADE)-ADOLOPMENT is widely applied to efficiently use existing credible guidelines and contextualize them to a target setting. To highlight the experiences of a Latin American Guideline Development Group (GDG) applying GRADE-ADOLOPMENT to adapt the American Society of Hematology's clinical practice guideline on managing venous thromboembolisms (VTEs) in Latin America., Study Design and Setting: We employed a mixed-method postevaluation using self-administered surveys and semistructured interviews. We assessed the Latin American GDG (1) general satisfaction and confidence using the approach, (2) their ratings on the usefulness, appropriateness, and importance of GRADE-ADOLOPMENT and its tools to inform their judgements, and (3) any additional facilitators and barriers to refine the process., Results: Eleven of the 14 GDG members, including nine panelists and two methodologists, provided survey responses and eight participated in the interview. Respondents felt "mostly" or "completely" satisfied with the adapted guideline. Eight panelists who were surveyed agree that GRADE-ADOLOPMENT is useful in countries with limited resources. Although panelists expressed initial apprehensions in their understanding of the process, they demonstrated enhanced confidence in their capacity to apply GRADE after completing workshop training and by acquiring experience. Panelists reiterated the importance of considering evidence-to-decision (EtD) criteria (ie, resources, feasibility, and cost-effectiveness) when adapting recommendations. The GDG encountered challenges with collecting local and regional data, prioritizing recommendations while considering intraregional diversity, and the lengthy publication period, although the latter stemmed from procedures not related to GRADE-ADOLOPMENT., Conclusions: GRADE-ADOLOPMENT is an important tool to facilitate the adaptation and uptake of clinical practice guidelines in novel settings. The GDG felt satisfied with their overall experience using the GRADE-ADOLOPMENT approach. However, their experience could have been optimized if they had access to robust regional evidence, more recommendations to adapt from, and worked with more efficient guideline production timelines., Competing Interests: Declaration of competing interest The authors have no conflict of interest to declare., (Copyright © 2024. Published by Elsevier Inc.)

Background and Objective: The Grading of Recommendations, Assessment, Development and Evaluations (GRADE)-ADOLOPMENT methodology has been widely used to adopt, adapt, or de novo develop recommendations from existing or new guideline and evidence synthesis efforts. The objective of this guidance is to refine the operationalization for applying GRADE-ADOLOPMENT., Methods: Through iterative discussions, online meetings, and email communications, the GRADE-ADOLOPMENT project group drafted the updated guidance. We then conducted a review of handbooks of guideline-producing organizations, and a scoping review of published and planned adolopment guideline projects. The lead authors refined the existing approach based on the scoping review findings and feedback from members of the GRADE working group. We presented the revised approach to the group in November 2022 (approximately 115 people), in May 2023 (approximately 100 people), and twice in September 2023 (approximately 60 and 90 people) for approval., Results: This GRADE guidance shows how to effectively and efficiently contextualize recommendations using the GRADE-ADOLOPMENT approach by doing the following: (1) showcasing alternative pathways for starting an adolopment effort; (2) elaborating on the different essential steps of this approach, such as building on existing evidence-to-decision (EtDs), when available or developing new EtDs, if necessary; and (3) providing examples from adolopment case studies to facilitate the application of the approach. We demonstrate how to use contextual evidence to make judgments about EtD criteria, and highlight the importance of making the resulting EtDs available to facilitate adolopment efforts by others., Conclusion: This updated GRADE guidance further operationalizes the application of GRADE-ADOLOPMENT based on over 6 years of experience. It serves to support uptake and application by end users interested in contextualizing recommendations to a local setting or specific reality in a short period of time or with limited resources., Competing Interests: Declaration of competing interest The authors are members or contributors to the GRADE working group. Zachary Munn is supported by an NHMRC investigator grant APP 1195676. Alexander G. Mathioudakis was supported by the National Institute for Health and Care Research Manchester Biomedical Research Centre (NIHR Manchester BRC, NIHR203308) and by an NIHR Clinical Lectureship in Respiratory Medicine. The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care., (Copyright © 2024. Published by Elsevier Inc.)

Objectives: To define the minimum knowledge required for guideline panel members (healthcare professionals and consumers) involved in developing recommendations about healthcare related testing., Study Design and Setting: A developmental study with a multistaged approach. We derived a first set of knowledge components from literature and subsequently performed semistructured interviews with 9 experts. We refined the set of knowledge components and checked it with the interviewees for final approval., Results: Understanding the test-management pathway, for example, how test results should be used in context of decisions about interventions, is the key knowledge component. The final list includes 26 items on the following topics: health question, test-management pathway, target population, test, test result, interpretation of test results and subsequent management, and impact on people important outcomes. For each item, the required level of knowledge is defined., Conclusion: We developed a list of knowledge components required for guideline panels to formulate recommendations on healthcare related testing. The list could be used to design specific training programs for guideline panel members when developing recommendations about tests and testing strategies in healthcare., Competing Interests: Declaration of competing interest M.K.T is GIN member and member of the GRADE Working Group. J.J.A. de Beer is GIN member and member of the GRADE Working Group. R.A.M. is a member of the GRADE working Group and ex-member of the GRADE Guidance Group and the GRADE test and diagnosis project group. C.S. is GIN member and past chair of the GIN PUBLIC working group (2020), member of the Guideline Commission of the German Association of the scientific medical societies, and chair of the German Health Literacy Network. H.J.S. is co-chair of the GRADE Working Group, chair of the GIN board, and head of the INGUIDE steering committee. M.W.L. is member of the GRADE Guidance Group, member of the GIN board, and INGUIDE instructor., (Copyright © 2024 The Author(s). Published by Elsevier Inc. All rights reserved.)

Objectives: Guidelines and essential medicine lists (EMLs) bear similarities and differences in the process that lead to decisions. Access to essential medicines is central to achieve universal health coverage. The World Health Organization (WHO) EML has guided prioritization of essential medicines globally for nearly 50 years, and national EMLs (NEMLs) exist in over 130 countries. Guideline and EML decisions, at WHO or national levels, are not always coordinated and aligned. We sought to explore challenges, and potential solutions, for decision-making to support trustworthy medicine selection for EMLs from a Grading of Recommendations, Assessment, Development and Evaluations (GRADE) Working Group perspective. We primarily focus on the WHO EML; however, our findings may be applicable to NEML decisions as well., Study Design and Setting: We identified key challenges in connecting the EML to health guidelines by involving a broad group of stakeholders and assessing case studies including real applications to the WHO EML, South Africa NEML, and a multiple sclerosis guideline connected to a WHO EML application for multiple sclerosis treatments. To address challenges, we utilized the results of a survey and feedback from the stakeholders, and iteratively met as a project group. We drafted a conceptual framework of challenges and potential solutions. We presented a summary of the results for feedback to all attendees of the GRADE Working Group meetings in November 2022 (approximately 120 people) and in May 2023 (approximately 100 people) before finalizing the framework., Results: We prioritized issues and insights/solutions that addressed the connections between the EML and health guidelines. Our suggested solutions include early planning alignment of guideline groups and EMLs, considering shared participation to strengthen linkage, further clarity on price/cost considerations, and using explicit shared criteria to make guideline and EML decisions. We also provide recommendations to strengthen the connection between WHO EML and NEMLs including through contextualization methods., Conclusion: This GRADE concept article, jointly developed by key stakeholders from the guidelines and EMLs field, identified key conceptual issues and potential solutions to support the continued advancement of trustworthy EMLs. Adopting structured decision criteria that can be linked to guideline recommendations bears the potential to advance health equity and gaps in availability of essential medicines within and between countries., Competing Interests: Declaration of competing interest Tamara Kredo is a member of the South African National Essential Medicines List Committee and Standard Treatment Guidelines; Co-lead of South African GRADE Network; Cochrane Board member; Member of the National Advisory Group on Immunisation. Trudy Leong is a member of the SA GRADE Network. Michael McCaul is a Member of the South African Primary Health Care and Adult Hospital Level Expert Review Committee; Founding member of the SA GRADE Network. The remaining authors are all members of the GRADE Working Group, however, have no other interests to declare., (Copyright © 2024. Published by Elsevier Inc.)

Objectives: Incorporating health equity considerations into guideline development often requires information beyond that gathered through traditional evidence synthesis methodology. This article outlines an operationalization plan for the Grading of Recommendations Assessment, Development, and Evaluation (GRADE)-equity criterion to gather and assess evidence from primary studies within systematic reviews, enhancing guideline recommendations to promote equity. We demonstrate its use in a clinical guideline on medical cannabis for chronic pain., Study Design and Setting: We reviewed GRADE guidance and resources recommended by team members regarding the use of evidence for equity considerations, drafted an operationalization plan, and iteratively refined it through team discussion and feedback and piloted it on a medicinal cannabis guideline., Results: We propose a seven-step approach: 1) identify disadvantaged populations, 2) examine available data for specific populations, 3) evaluate population baseline risk for primary outcomes, 4) assess representation of these populations in primary studies, 5) appraise analyses, 6) note barriers to implementation of effective interventions for these populations, and 7) suggest supportive strategies to facilitate implementation of effective interventions., Conclusion: Our approach assists guideline developers in recognizing equity considerations, particularly in resource-constrained settings. Its application across various guideline topics can verify its feasibility and necessary adjustments., Competing Interests: Declaration of competing interest The authors have nothing to disclose., (Copyright © 2023 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: To provide guidance on rating imprecision in a body of evidence assessing the accuracy of a single test. This guide will clarify when Grading of Recommendations Assessment, Development and Evaluation (GRADE) users should consider rating down the certainty of evidence by one or more levels for imprecision in test accuracy., Study Design and Setting: A project group within the GRADE working group conducted iterative discussions and presentations at GRADE working group meetings to produce this guidance., Results: Before rating the certainty of evidence, GRADE users should define the target of their certainty rating. GRADE recommends setting judgment thresholds defining what they consider a very accurate, accurate, inaccurate, and very inaccurate test. These thresholds should be set after considering consequences of testing and effects on people-important outcomes. GRADE's primary criterion for judging imprecision in test accuracy evidence is considering confidence intervals (i.e., CI approach) of absolute test accuracy results (true and false, positive, and negative results in a cohort of people). Based on the CI approach, when a CI appreciably crosses the predefined judgment threshold(s), one should consider rating down certainty of evidence by one or more levels, depending on the number of thresholds crossed. When the CI does not cross judgment threshold(s), GRADE suggests considering the sample size for an adequately powered test accuracy review (optimal or review information size [optimal information size (OIS)/review information size (RIS)]) in rating imprecision. If the combined sample size of the included studies in the review is smaller than the required OIS/RIS, one should consider rating down by one or more levels for imprecision., Conclusion: This paper extends previous GRADE guidance for rating imprecision in single test accuracy systematic reviews and guidelines, with a focus on the circumstances in which one should consider rating down one or more levels for imprecision., Competing Interests: Declaration of competing interest The authors declare no direct financial conflict of interest. They are members or contributors to the GRADE working group. Multiple coauthors have served as methodologists on a number of systematic reviews or guidelines that provided examples for this manuscript., (Copyright © 2023. Published by Elsevier Inc.)

Objectives: To make informed decisions, the general population should have access to accessible and understandable health recommendations. To compare understanding, accessibility, usability, satisfaction, intention to implement, and preference of adults provided with a digital "Plain Language Recommendation" (PLR) format vs. the original "Standard Language Version" (SLV)., Study Design and Setting: An allocation-concealed, blinded, controlled superiority trial and a qualitative study to understand participant preferences. An international on-line survey. 488 adults with some English proficiency. 67.8% of participants identified as female, 62.3% were from the Americas, 70.1% identified as white, 32.2% had a bachelor's degree as their highest completed education, and 42% said they were very comfortable reading health information. In collaboration with patient partners, advisors, and the Cochrane Consumer Network, we developed a plain language format of guideline recommendations (PLRs) to compare their effectiveness vs. the original standard language versions (SLVs) as published in the source guideline. We selected two recommendations about COVID-19 vaccine, similar in their content, to compare our versions, one from the World Health Organization (WHO) and one from Centers for Disease Control and Prevention (CDC). The primary outcome was understanding, measured as the proportion of correct responses to seven comprehension questions. Secondary outcomes were accessibility, usability, satisfaction, preference, and intended behavior, measured on a 1-7 scale., Results: Participants randomized to the PLR group had a higher proportion of correct responses to the understanding questions for the WHO recommendation (mean difference [MD] of 19.8%, 95% confidence interval [CI] 14.7-24.9%; P < 0.001) but this difference was smaller and not statistically significant for the CDC recommendation (MD of 3.9%, 95% CI -0.7% to 8.3%; P = 0.096). However, regardless of the recommendation, participants found the PLRs more accessible, (MD of 1.2 on the seven-point scale, 95% CI 0.9-1.4%; P < 0.001) and more satisfying (MD of 1.2, 95% CI 0.9-1.4%; P < 0.001). They were also more likely to follow the recommendation if they had not already followed it (MD of 1.2, 95% CI 0.7-1.8%; P < 0.001) and share it with other people they know (MD of 1.9, 95% CI 0.5-1.2%; P < 0.001). There was no significant difference in the preference between the two formats (MD of -0.3, 95% CI -0.5% to 0.03%; P = 0.078). The qualitative interviews supported and contextualized these findings., Conclusion: Health information provided in a PLR format improved understanding, accessibility, usability, and satisfaction and thereby has the potential to shape public decision-making behavior., Competing Interests: Declaration of competing interest Dr. Pottie is supported by the Ian McWhinney Research Chair in Family Medicine, Western University. Dr. Butcher declares consulting fees from Nobias Therapeutics, Inc. Dr. Hartling is supported by a Canada Research Chair in Knowledge Synthesis and Translation. Dr. Scott is supported by a Canada Research Chair in Knowledge Translation in Child Health. Dr. Hartling and Dr. Scott are Distinguished Researchers with the Stollery Science Lab supported by the Stollery Children's Hospital Foundation. Dr. Richards is a volunteer vice president of the Canadian Arthritis Patient Alliance, a volunteer-run organization that receives independent grants from pharmaceutical companies., (Copyright © 2023 The Authors. Published by Elsevier Inc. All rights reserved.)

Introduction: This updated guidance from the Grading of Recommendations Assessment, Development, and Evaluation addresses rating up certainty of evidence due to a dose-response gradient (DRG) observed in synthesis of intervention and exposure studies., Study Design and Setting: This guidance was developed using iterative discussions and consensus in multiple meetings and was presented to attendees of the Grading of Recommendations Assessment, Development, and Evaluation Working Group meeting for feedback in November 2022 and for final approval in May 2023., Results: The guidance consists of two steps. The first is to determine whether the DRG is credible. We describe five items for assessing credibility: a) is DRG identified using a proper analytical approach; b) is confounding the cause of the DRG; c) is there serious concern about ecological bias; d) is the DRG consistent across studies; and e) is there indirect evidence supporting the DRG. The first two of these items are the most critical. If the DRG was judged to be credible, then the second step is to apply the DRG domain and consider rating up, but only by one level due to the concern about residual confounding., Conclusion: Systematic review authors should only rate up certainty in evidence when a DRG is deemed credible., Competing Interests: Declaration of competing interest Authors of this paper are GRADE Working Group members. Gordon H. Guyatt and Holger J. Schünemann are the co-founders and co-chairs of GRADE Working Group. Gordon H. Guyatt is a member of the Journal of Clinical Epidemiology Editorial Board., (Copyright © 2023 Elsevier Inc. All rights reserved.)

Objectives: We describe how consideration of external evidence may play an important role in judging certainty in the process of establishing the certainty of the evidence. Our example is a network meta-analysis (NMA) addressing treatment for Ebola virus disease, which informed a World Health Organization guideline., Study Design and Setting: Through Grading of Recommendations Assessment, Development, and Evaluations (GRADE) project group iterative online, in-person and email discussions, we developed this GRADE concept and obtained approval from the GRADE working group. Using the null as a threshold, we rated our certainty for network estimates in mortality, including consideration of evidence external to the NMA (i.e., did not meet eligibility criteria) and formal logical construction., Results: Based on the existing GRADE guidance, we rated the network estimate for one indirect comparison as low certainty. The formal logical construction that lead us reevaluate the certainty of the evidence is as follows: if A is superior to B, and B is not inferior to C, then A must be superior to C. After considering the logic and the external indirect evidence, we concluded at least moderate certainty for the comparison., Conclusion: Systematic review authors and guideline developers should apply the fundamental logical construction for indirect comparisons and consider compelling external evidence in NMA certainty ratings., Competing Interests: Declaration of competing interest Several authors (Qiukui Hao, Ya Gao, M. Hassan Murad, Reem Mustafa, Mohammed T. Ansari, Holger J Schünemann, David M Rind, Romina Brignardello-Petersen, Gordon Guyatt) in this paper are members of the GRADE working group. Gordon Guyatt also serves on the Editorial Board for the Journal of Clinical Epidemiology., (Copyright © 2023 Elsevier Inc. All rights reserved.)

Background: This is the 23rd in a series of articles describing the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to grading the certainty of evidence and strength of recommendations for systematic reviews, health technology assessments, and clinical guideline development., Objectives: We outline how resource utilization and cost-effectiveness analyses are integrated into health-related recommendations, using the GRADE Evidence to Decision (EtD) frameworks., Study Design and Setting: Through iterative discussions and refinement, in-person, and online meetings, and through e-mail communication, we developed draft guidance to incorporate economic evidence in the formulation of health-related recommendations. We developed scenarios to operationalize the guidance. We presented a summary of the results to members of the GRADE Economic Evaluation Project Group., Results: We describe how to estimate the cost of preventing (or achieving) an event to inform assessments of cost-effectiveness of alternative treatments, when there are no published economic evaluations. Evidence profiles and Summary of Findings tables based on systematic reviews of cost-effectiveness analyses can be created to provide top-level summaries of results and quality of multiple published economic evaluations. We also describe how this information could be integrated in GRADE's EtD frameworks to inform health-related recommendations. Three scenarios representing various levels of available cost-effectiveness evidence were used to illustrate the integration process., Conclusion: This GRADE guidance provides practical information for presenting cost-effectiveness data and its integration in the development of health-related recommendations, using the EtD frameworks., Competing Interests: Declaration of competing interest The GRADE system has been developed by the GRADE Working Group. The named authors drafted and revised this article. F.X. is a member of the EuroQol Group and elected deputy chair of its Executive Committee. Seven authors of this article (I.S., L.V., F.R., M.F.D., J.L., K.H., and M.X.R.) are all convenors or members of the Campbell and Cochrane Economics Methods Group, a Cochrane Methods Group. F.X., I.S., L.V., Y.Z., M.X.R., C.C.A., P.A.C., and H.J.S. are members of the GRADE working group. The views expressed in this article are those of the authors and not necessarily those of any organization., (Copyright © 2023 Elsevier Inc. All rights reserved.)

Competing Interests: Declaration of competing interest EAA, RAM, RN, AS, HJS declare no relationships of interest to disclose.

Objectives: To assess the effectiveness of plain language compared with standard language versions of COVID-19 recommendations specific to child health., Study Design and Setting: Pragmatic, allocation-concealed, blinded, superiority randomized controlled trial with nested qualitative component. Trial was conducted online, internationally. Parents or legal guardians (≥18 years) of a child (<18 years) were eligible. Participants were randomized to receive a plain language recommendation (PLR) or standard (SLV) verison of a COVID-19 recommendation specific to child health. Primary outcome was understanding. Secondary outcomes included: preference, accessibility, usability, satisfaction, and intended behavior. Interviews explored perceptions and preferences for each format., Results: Between July and August 2022, 295 parents were randomized; 241 (81.7%) completed the study (intervention n = 121, control n = 120). Mean understanding scores were significantly different between groups (PLR 3.96 [standard deviation (SD) 2.02], SLV 3.33 [SD 1.88], P = 0.014). Overall participants preferred the PLR version: mean rating 5.05/7.00 (95% CI 4.81, 5.29). Interviews (n = 12 parents) highlighted their preference for the PLR and provided insight on elements to enhance future knowledge mobilization of health recommendations., Conclusion: Compared to SLVs, parents preferred PLRs and better understood the recommendation. Guideline developers should strive to use plain language to increase understanding, uptake, and implementation of evidence by the public., Competing Interests: Declaration of Competing Interest The authors have no conflicts of interest to declare., (Copyright © 2023 The Authors. Published by Elsevier Inc. All rights reserved.)

Competing Interests: Declaration of Competing Interest TWC declares contracts with CDC NIOSH to evaluate a research program with a stakeholder engagement component, with NY AIDS Bureau and HRSA to develop measures for engagement in HIV care quality, with PCORI to identify measures of engagement, with Innovation and Value Initiative to develop an approach for patient involvement in HTA, and with PhRMA to evaluate equity and inclusion in HTA. He has received an award and subsequent contract by PhRMA foundation for proposing a new approach for engaging patient in HTA. He serves as a director of community and stakeholder engagement for the Tufts CTSI (NIH/NCATS). He receives royalties by Springer International Publishing for serving as editor of a volume: “Broadly Engaged Team Science in Clinical and Translational Research.” He received a payment by Virginia Commonwealth University Wright Center for reviewing a grant proposal and anticipates payments by Henry Stewart Talks in 2022-23 for delivering four to six online lectures. PhRMA supported his attendance at ISPOR 22. All other authors have no competing interests to declare.

Competing Interests: Declaration of Competing Interest All authors are member of the GRADE Guidance Group. HJS and GG are cochairs of GRADE.

Objectives: This Grading of Recommendations Assessment, Development and Evaluation (GRADE) concept article offers systematic reviewers, guideline authors, and other users of evidence assistance in addressing randomized trial situations in which interventions or comparators differ from those in the target people, interventions, comparators, and outcomes. To clarify what GRADE considers under indirectness of interventions and comparators, we focus on a particular example: when comparator arm participants receive some or all aspects of the intervention management strategy (treatment switching)., Study Design and Setting: An interdisciplinary panel of the GRADE working group members developed this concept article through an iterative review of examples in multiple teleconferences, small group sessions, and e-mail correspondence. After presentation at a GRADE working group meeting in November 2022, attendees approved the final concept paper, which we support with examples from systematic reviews and individual trials., Results: In the presence of safeguards against risk of bias, trials provide unbiased estimates of the effect of an intervention on the people as enrolled, the interventions as implemented, the comparators as implemented, and the outcomes as measured. Within the GRADE framework, differences in the people, interventions, comparators, and outcomes elements between the review or guideline recommendation targets and the trials as implemented constitute issues of indirectness. The intervention or comparator group management strategy as implemented, when it differs from the target comparator, constitutes one potential source of indirectness: Indirectness of interventions and comparators-comparator group receipt of the intervention constitutes a specific subcategory of said indirectness. The proportion of comparator arm participants that received the intervention and the apparent magnitude of effect bear on whether one should rate down, and if one does, to what extent., Conclusion: Treatment switching and other differences between review or guideline recommendation target interventions and comparators vs. interventions and comparators as implemented in otherwise relevant trials are best considered issues of indirectness., (Copyright © 2023 Elsevier Inc. All rights reserved.)

Objectives: Clinical practice guidelines (CPGs) are often created through collaboration among organizations. The use of inconsistent terminology may cause poor communication and delays. This study aimed to develop a glossary of terms related to collaboration in guideline development., Study Design and Setting: A literature review of collaborative guidelines was performed to develop an initial list of terms related to guideline collaboration. The list of terms was presented to the members of the Guideline International Network Guidelines Collaboration Working Group, who provided presumptive definitions for each term and proposed additional terms to be included. The revised list was subsequently reviewed by an international, multidisciplinary panel of expert stakeholders. Recommendations received during this pre-Delphi review were implemented to augment an initial draft glossary. The glossary was then critically evaluated and refined through two rounds of Delphi surveys and a virtual consensus meeting with all panel members as Delphi participants., Results: Forty-nine experts participated in the pre-Delphi survey, and 44 participated in the two-round Delphi process. Consensus was reached for 37 terms and definitions., Conclusion: Uptake and utilization of this guideline collaboration glossary by key organizations and stakeholder groups may facilitate collaboration among guideline-producing organizations by improving communication, minimizing conflicts, and increasing guideline development efficiency., (Copyright © 2023 Elsevier Inc. All rights reserved.)

Objectives: To highlight how using the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) approach to understand the certainty in the evidence about the impact of climate change in health outcomes increases transparency. Also, how GRADE can enhance communication and decisions about adaptation and mitigation strategies., Study Design and Setting: We developed a narrative review based on an assessment of exiting systematic reviews addressing the effect of climate change on health outcomes and the impact of mitigation and adaptation strategies., Results: Adopting structured approaches such as GRADE to tackle the impact of climate change on health may help to (1) define the specific question to be addressed; (2) summarize the evidence in a structured way and assess uncertainty; (3) provide a systematic framework to move from evidence to action and to offer recommendations of different strength; (4) provide a systematic way to adapt recommendations to specific settings; and (5) provide a framework to assess the certainty of modeled evidence., Conclusion: In this article, we describe epidemiologic principles that could be used to move decision-making in climate change forward., (Copyright © 2023 Elsevier Inc. All rights reserved.)

Background and Objectives: Our objective was to develop an extension of the widely used GIN-McMaster Guideline Development Checklist and Tool for the integration of quality assurance and improvement (QAI) schemes with guideline development., Methods: We used a mixed-methods approach incorporating evidence from a systematic review, an expert workshop and a survey of experts to iteratively create an extension of the checklist for QAI through three rounds of feedback. As a part of this process, we also refined criteria of a good guideline-based quality indicator., Results: We developed a 40-item checklist extension addressing steps for the integration of QAI into guideline development across the existing 18 topics and created one new topic specific to QAI. The steps span from 'organization, budget, planning and training', to updating of QAI and guideline implementation., Conclusion: The tool supports integration of QAI schemes with guideline development initiatives and it will be used in the forthcoming integrated European Commission Initiative on Colorectal Cancer. Future work should evaluate this extension and QAI items requiring additional support for guideline developers and links to QAI schemes., (Copyright © 2022 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: To evaluate alternative formats of summary of findings (SoF) tables for single comparison with multiple outcomes., Study Design and Setting: We conducted a three-arm randomized controlled noninferiority trial (RCT) in the following systematic review (SR) users: researchers, clinical practice guideline developers, health care providers, policymakers, and knowledge transfer organizations to measure understanding, accessibility, satisfaction, and preference across the current grading of recommendations assessment, development, and evaluation (GRADE) SoF, an alternative GRADE SoF, or an adapted evidence-based practice center (EPC) program SoF table., Results: One Hundred Seventy-Nine participants were randomized, and 129 participants completed the RCT (n = 47 current GRADE, n = 41 alternative GRADE, n = 41 adapted EPC). Understanding the certainty of evidence and treatment effect was comparable across groups. The adapted EPC SoF table was inferior for quantifying risk and RD compared to the alternatives (<35% correct vs. >85% correct). Participants reported increased satisfaction when SoF tables presented number needed to treat (NNT), anticipated absolute effect differences, and narrative syntheses for evidence that could not be meta-analyzed. Participants reported accessibility to information as significantly better in both GRADE SoF tables, when compared with the adapted EPC SoF table. Participants preferred the alternative GRADE SoF table format., Conclusion: The alternative GRADE SoF table is a promising format for SR users preferring a comprehensive presentation of SR results for single comparisons., (Copyright © 2022 Elsevier Inc. All rights reserved.)

▪., (Copyright © 2022 Elsevier Inc. All rights reserved.)

Objectives: Randomized controlled trials (RCTs) are the preferred source of evidence for the relative effect of healthcare interventions summarized in knowledge syntheses. Nonrandomized studies of interventions (NRSI) may provide replacement, sequential, or complementary evidence to RCTs. The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach can provide different options for properly using RCTs and NRSI integrated in health syntheses. In this article, we discuss different implications on the certainty of evidence when authors consider the use of NRSI and RCTs in systematic reviews using GRADE. Although this is a GRADE-related article, it is not an official GRADE guidance or concept article., Study Design and Setting: We present case studies used during GRADE working group meetings for discussion of the effects of using NRSI and RCTs on GRADE domains and on the certainty of evidence. Several concepts were discussed through iterative feedback with experts in GRADE methods and Cochrane authors. We compared suggested solutions for possible scenarios that can be met in evidence syntheses informing decisions and future guidance., Results: Different scenarios for the use of RCTs and NRSI in evidence syntheses are presented, focusing on how different GRADE ratings between RCTs and NRSI affect the overall assessment of the evidence and possible health recommendations., Conclusions: Considering differences and similarities grounded in the GRADE approach between NRSI and RCTs may help complement one another and maximize the value of knowledge syntheses and health recommendations., (Copyright © 2022 Elsevier Inc. All rights reserved.)

Objectives: The aim of this study is to provide updated guidance on when The Grading of Recommendations Assessment, Development and Evaluation (GRADE) users should consider rating down more than one level for imprecision using a minimally contextualized approach., Study Design and Setting: Based on the first GRADE guidance addressing imprecision rating in 2011, a project group within the GRADE Working Group conducted iterative discussions and presentations at GRADE Working Group meetings to produce this guidance., Results: GRADE suggests aligning imprecision criterion for systematic reviews and guidelines using the approach that relies on thresholds and confidence intervals (CI) of absolute effects as a primary criterion for imprecision rating (i.e., CI approach). Based on the CI approach, when a CI appreciably crosses the threshold(s) of interest, one should consider rating down two or three levels. When the CI does not cross the threshold(s) and the relative effect is large, one should implement the optimal information size (OIS) approach. If the sample size of the meta-analysis is far less than the OIS, one should consider rating down more than one level for imprecision., Conclusion: GRADE provides updated guidance for imprecision rating in a minimally contextualized approach, with a focus on the circumstances in which one should seriously consider rating down two or three levels for imprecision., (Copyright © 2022 The Author(s). Published by Elsevier Inc. All rights reserved.)

Objectives: Grading of Recommendations Assessment, Development and Evaluation (GRADE) guidance to rate the certainty domain of imprecision is presently not fully operationalized for rating down by two levels and when different baseline risk or uncertainty in these risks are considered. In addition, there are scenarios in which lowering the certainty of evidence by three levels for imprecision is more appropriate than lowering it by two levels. In this article, we conceptualize and operationalize rating down for imprecision by one, two and three levels for imprecision using the contextualized GRADE approaches and making decisions., Methods: Through iterative discussions and refinement in online meetings and through email communication, we developed draft guidance to rating the certainty of evidence down by up to three levels based on examples. The lead authors revised the approach according to the feedback and the comments received during these meetings and developed GRADE guidance for how to apply it. We presented a summary of the results to all attendees of the GRADE Working Group meeting for feedback in October 2021 (approximately 80 people) where the approach was formally approved., Results: This guidance provides GRADE's novel approach for the considerations about rating down for imprecision by one, two and three levels based on serious, very serious and extremely serious concerns. The approach includes identifying or defining thresholds for health outcomes that correspond to trivial or none, small, moderate or large effects and using them to rate imprecision. It facilitates the use of evidence to decision frameworks and also provides guidance for how to address imprecision about implausible large effects and trivial or no effects using the concept of the 'review information size' and for varying baseline risks. The approach is illustrated using practical examples, an online calculator and graphical displays and can be applied to dichotomous and continuous outcomes., Conclusion: In this GRADE guidance article, we provide updated guidance for how to rate imprecision using the partially and fully contextualized GRADE approaches for making recommendations or decisions, considering alternate baseline risks and for both dichotomous and continuous outcomes., (Copyright © 2022 Elsevier Inc. All rights reserved.)

Objectives: Analytical frameworks are graphical representation of the key questions answered by a systematic review and can support the development of guideline recommendations. Our objectives were to a) conduct a systematic review to identify, describe and compare all analytical frameworks published as part of a systematic and guideline development process related to colorectal cancer (CRC), and b) to use this case study to develop guidance on how to conduct systematic reviews of analytical frameworks., Methods: We developed a search strategy to identify eligible studies in Medline and Embase from 1996 until December 2020. We also manually searched guideline databases and websites to identify all guidelines and systematic reviews in CRC that used an analytical framework. We assessed the quality of the guidelines using the Appraisal of Guidelines for Research and Evaluation II tool. The systematic review was registered in International Prospective Register of Systematic Reviews, registration CRD42020172117., Results: We screened 34,505 records and identified 1,166 guidelines and 3,127 systematic reviews on CRC of which five met our inclusion criteria. These five publications included four analytical frameworks in colorectal cancer (one update). We also describe our methodological approach to systematic reviews for analytical frameworks and underlying concepts for developing analytical framework using a bottom-up or top-down approach., Conclusion: Few guidelines and systematic reviews are utilizing analytical frameworks in the development of recommendations. Development of analytical frameworks should begin with a systematic search for existing analytical frameworks and follow a structured conceptual approach for their development to support guideline recommendations. Our methods may be helpful in achieving these objectives., (Copyright © 2022 Elsevier Inc. All rights reserved.)

Background and Objectives: To develop a digital communication tool to improve the implementation of up-to-date COVID-19 recommendations. Specifically, to improve patient, caregiver and public understanding of healthcare recommendations on prevention, diagnoses and treatment., Methods: Multi-stakeholder engagement design. In conjunction with the COVID-19 Recommendations and Gateway to Contextualization RecMap, we co-developed a stakeholder prioritization, drafting and editing process to enhance guideline communication and understanding., Results: This paper presents the multi-stakeholder development process with three distinct plain language recommendation formats: formal recommendation, good practice statement, and additional guidance. Our case study of COVID-19 plain language recommendations PLRs addresses both public health interventions (e.g., vaccination, face masks) and clinical interventions (e.g., home pulse oximetry)., Conclusion: This paper presents a novel approach to engaging stakeholders in improving the communication and understanding of published guidelines during the COVID-19 pandemic., (Copyright © 2022 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: Grading of Recommendations Assessment, Development and Evaluation (GRADE) practice guideline developers often perform systematic reviews of potential economic evaluations to inform recommendation decision-making. We aimed to identify indirectness characteristics of economic evaluations, related to GRADE evidence-to-decision (EtD) theoretical frameworks, that influence selection of these articles., Study Design and Setting: MEDLINE, EMBASE, CINAHL, and EconLit were systematically searched to May 2020 to identify indirectness characteristics relevant for economic evaluation transferability to GRADE EtD theoretical frameworks. Four reviewers screened citations to identify articles of any type that explored study characteristics most important or relevant to economic evaluation transferability, restricted to English language we generated frequencies of article features, used thematic analysis to summarize study characteristics, and assessed certainty in the evidence using GRADE-CERQual., Results: We included 57 articles, with a dearth of empirical literature-some may have been missed. We identified eight general themes and 28 subthemes most important to transferability from 41% of articles. Moderate-to-high confidence evidence suggested that GRADE EtD domains of population, intervention and comparison research question elements, resource use estimation and methodology, and provider and decision maker acceptability are most important indirectness study characteristics that economists consider when choosing economic evaluation outcomes for use in recommendation decision-making., Conclusion: We have identified factors important for guideline developers to consider when selecting economic evaluations as research evidence. An economic competency on the development team facilitates these endeavors. This supports the GRADE Working Group's tenant of transparent reporting or availability of sufficient information elsewhere to assess indirectness., (Copyright © 2022 Elsevier Inc. All rights reserved.)

Background: There is a little empirical evidence of the impact of pooling randomized controlled trials (RCTs) and cohort studies (CSs) on the certainty-of-evidence. To evaluate the hypothetical-scenario of pooling bodies-of-evidence from RCTs with matched bodies-of-evidence from CSs on the certainty-of-evidence., Methods: We extracted GRADE ratings of bodies-of-evidence from RCTs in Cochrane reviews, and rated the certainty-of-evidence from matched bodies-of-evidence from CSs. We then evaluated the impact of pooling both bodies-of-evidence on the overall certainty-of-evidence, and on individual GRADE domains., Results: Fourty-two pooled bodies-of-evidence were rated, ranging from very-low (bodies-of-evidence RCTs : 9.5%; bodies-of-evidence CSs : 40.5%; pooled-bodies-of-evidence: 0%) to low (bodies-of-evidence RCTs : 38.1%; bodies-of-evidence CSs : 45.2%; pooled-bodies-of-evidence: 19.1%), moderate (bodies-of-evidence RCTs : 33.4%; bodies-of-evidence CSs : 14.3%; pooled-bodies-of-evidence: 57.1%), and high (bodies-of-evidence RCTs : 19%; bodies-of-evidence CSs : 0%; pooled-bodies-of-evidence: 23.8%). Certainty-of-evidence was downgraded mostly for imprecision and risk of bias for bodies-of-evidence from RCTs, and for risk of bias and inconsistency for bodies-of-evidence from CSs. Pooling both bodies-of-evidence mitigates rating down for imprecision compared to bodies-of-evidence from RCTs and inconsistency compared to bodies-of-evidence from CSs., Conclusion: Our hypothetical study suggests that pooling both bodies-of-evidence would reduce the amount of very-low and low certainty-of-evidence ratings, but how to integrate RCTs and CSs and whether or not to pool these bodies-of-evidence requires proper guidance before systematic review authors or guideline developers should consider this approach., (Copyright © 2022 Elsevier Inc. All rights reserved.)

Objectives: We conceptualize patient values and preferences as the relative importance of health outcomes (RIO) which are often obtained through utility elicitation research. A transparent and structured approach to present synthesized RIO evidence and the certainty of this evidence is needed. This study aims to adapt the summary of findings (SoF) table to describe the RIO., Study Design and Setting: We performed three interactive workshops with a protype version of the SoF table for RIO, evidence adapted from the SoF table for intervention effects. We then tested the new format through semi-structured interviews with professionals who interpret RIO evidence (e.g., systematic review authors and guideline developers)., Results: We adapted the SoF table for the presentation of RIO evidence. This SoF table may be easy to use, but bears one risk: some participants misunderstood the utility information and the variability around the RIO. We added a visual analogue scale to clarify the concept of utilities., Conclusion: Through a multi-stage process including brainstorming sessions and interviews, we adapted the SoF table to present RIO evidence. This table may enhance understanding of evidence synthesis of values and preferences, facilitating the incorporation of this type of evidence in decision-making., (Copyright © 2022. Published by Elsevier Inc.)

Objectives: Guideline panels must assess the magnitude of health benefits and harms to develop sensible recommendations. However, they rarely use explicit thresholds. In this paper we report on the piloting and the use thresholds for benefits and harms., Study Design and Setting: We piloted the use of thresholds in a Chilean COVID-19 living guideline. For each of the critical outcomes, we asked panelists to suggest values of the thresholds for large, moderate, small, or trivial or no effect. We collected this information through a survey and an on-line discussion., Results: Twelve panelists decided on thresholds for three critical outcomes (mortality, need for mechanical ventilation and serious adverse events). For all outcomes, an absolute risk reduction was considered larger with more than 50 events, moderate with less than 50 events, small with less than 25 events, and trivial with less than 10 events. Having these a priori thresholds in place significantly impacted on the development of recommendations., Conclusion: Explicit thresholds were a valuable addition to the judgment of the certainty in the evidence, to decide the direction and strength of the recommendation and to evaluate the need for update. We believe this is a line of research worth perusing., (Copyright © 2022 Elsevier Inc. All rights reserved.)

Objectives: To describe divergence between actionable statements issued by coronavirus disease 2019 (COVID-19) guideline developers cataloged on the "COVID-19 Recommendations and Gateway to Contextualization" platform., Study Design and Setting: We defined divergence as at least two comparable actionable statements with different explicit judgments of strength, direction, or subgroup consideration of the population or intervention. We applied a content analysis to compare guideline development methods for a sample of diverging statements and to evaluate factors associated with divergence., Results: Of the 138 guidelines evaluated, 85 (62%) contained at least one statement that diverged from another guideline. We identified 223 diverging statements in these 85 guidelines. We grouped statements into 66 clusters. Each cluster addressed the same population, intervention, and comparator group or just similar interventions. Clinical practice statements were more likely to diverge in an explicit judgment of strength or direction compared to public health statements. Statements were more likely to diverge in strength than direction. The date of publication, used evidence, interpretation of evidence, and contextualization considerations were associated with divergence., Conclusion: More than half of the assessed guidelines issued at least one diverging statement. This study helps in understanding the types of differences between guidelines issuing comparable statements and factors associated with their divergence., (Copyright © 2022. Published by Elsevier Inc.)

Background: Grading of Recommendations Assessment, Development and Evaluation (GRADE) Evidence-to-Decision (EtDs) frameworks are increasingly applied by health guideline developers to improve the use of evidence and transparency of health recommendations. Typically, EtDs include 12 criteria but these are flexible and EtDs have been adapted to different types of health decisions. However, developers of health recommendations struggle with the content that they should include in the EtD. The goal of this work was to provide a standardized template that facilitate the development of GRADE EtDs in health guidelines and examples for practical training., Methods: We began by establishing the need for standardized wording templates in 10 American Society of Hematology guidelines with over 250 recommendations. We drafted template wording, and examples, and sought iterative feedback from methodologists and guideline panels in this guideline and two additional guidelines., Results: We generated templates for all EtD criteria describing the type of research evidence considered, ideally based on systematic reviews, using standardized reporting of effect size, integrating the certainty of evidence and additional considerations. We also produced templates to inform the completion of the EtD conclusions section that includes recommendations, justification, implementation considerations, monitoring and evaluation and research priorities., Conclusion: We developed a pragmatic and useful tool to support clarity, transparency, and efficiency of the guideline EtD process., (Copyright © 2022. Published by Elsevier Inc.)

Objectives: This study aimed to identify and describe the use of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) dose-response gradient domain to upgrade the certainty of evidence (CoE) in nutrition systematic reviews (SRs)., Study Design and Setting: We searched for SRs of observational studies of nutrition topics that used GRADE and upgraded the CoE of at least one outcome for a dose-response gradient or reported reasons for not upgrading., Results: Within eligible SRs (21/281), 123 of 371 outcomes were upgraded for a dose-response gradient. For 118 outcomes, the authors conducted linear dose-response analyses, and for 106 outcomes, the authors conducted nonlinear dose-response analyses. From these, 107 outcomes showed a statistically significant (P < 0.05) association in the linear dose-response model, and for 28 outcomes, the test for nonlinearity was statistically significant. The CoE for 0.8% of all outcomes was rated as high, 47.2% as moderate, 43.9% as low, and 8.1% as very low. Fifty-five percent of outcomes that were upgraded for a dose-response gradient were already downgraded for at least one domain. This is contrary to GRADE guidance., Conclusion: The approach for rating up the CoE for dose-response relationship is not consistent in nutrition reviews, likely because of a lack of clear guidance for when and how to do it. Therefore, more comprehensive GRADE guidance is necessary to enhance the correct use and comparability of dose-response upgrading., (Copyright © 2022 Elsevier Inc. All rights reserved.)

Background: "Biological plausibility" is a concept frequently referred to in environmental and public health when researchers are evaluating how confident they are in the results and inferences of a study or evidence review. Biological plausibility is not, however, a domain of one of the most widely used approaches for assessing the certainty of evidence (CoE) which underpins the findings of a systematic review, the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) CoE Framework., Objectives: Whether the omission of biological plausibility is a potential limitation of the GRADE CoE Framework is a topic that is regularly discussed, especially in the context of environmental health systematic reviews., Study Design and Setting: We analyze how the concept of "biological plausibility," as applied in the context of assessing certainty of the evidence that supports the findings of a systematic review, is accommodated under the processes of systematic review and the existing GRADE domains., Results: We argue that "biological plausibility" is a concept which primarily comes into play when direct evidence about the effects of an exposure on a population of concern (usually humans) is absent, at high risk of bias, inconsistent, or limited in other ways. In such circumstances, researchers look toward evidence from other study designs to draw conclusions. In this respect, we can consider experimental animal and in vitro evidence as "surrogates" for the target populations, exposures, comparators, and outcomes of actual interest. Through discussion of 10 examples of experimental surrogates, we propose that the concept of biological plausibility consists of two principal aspects: a "generalizability aspect" and a "mechanistic aspect.", Conclusions: The "generalizability aspect" concerns the validity of inferences from experimental models to human scenarios, and asks the same question as does the assessment of external validity or indirectness in systematic reviews. The "mechanistic aspect" concerns certainty in knowledge of biological mechanisms and would inform judgments of indirectness under GRADE, and thus the overall CoE. Although both aspects are accommodated under the indirectness domain of the GRADE CoE Framework, further research is needed to determine how to use knowledge of biological mechanisms in the assessment of indirectness of the evidence in systematic reviews., (Copyright © 2021 Paul Whaley, Thomas Piggott, Rebecca L. Morgan, Sebastian Hoffmann, Katya Tsaioun, Lukas Schwingshackl, Mohammed T. Ansari, Kristina A. Thayer, Holger Schünemann. Published by Elsevier Inc. All rights reserved.)

Objective: The objective of this paper is to propose an approach to visual unification of adapted guidelines and transformation of classifications of certainty of evidence (CoE) and strength of recommendations (SoR) into the approach suggested by the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group., Study Design and Setting: We carried out a literature search in MEDLINE and Epistemonikos, an analysis of selected guidelines, and an iterative discussion to decide on a consistent visual presentation and CoE and SoR depictions., Results: The results of the literature search suggested this issue had not been addressed yet. The analysis of the chosen eight guidelines showed significant heterogeneity in the visual presentation of recommendations. Recommendations were often worded similarly to whether or not they were strong or conditional. Many guidelines contained "statements," almost all of which did not fulfill the good practice statement (GPS) criteria. We proposed an approach for transforming recommendations that are being adapted and which use various classification systems for CoE and SoR into GRADE and a consistent visual style., Conclusion: Guideline developers should aim for unification in the formulation of recommendations to improve transferability., (Copyright © 2021 The Authors. Published by Elsevier Inc. All rights reserved.)

Background: Health guideline development requires sequential prioritization of the guideline topic, questions, and health outcomes. In this paper we report on new approaches for prioritizing questions and outcomes in guidelines., Methods: Ten guideline panels on venous thromboembolism rated potential guideline questions on a 9-point scale according to their overall importance and 6 criteria: common in practice, uncertainty in practice, variation in practice, new evidence available, cost consequences, not previously addressed. We randomized panelists to rate one potential question with and without the 6 criteria. Panelists rated importance of outcomes, defined with health outcome descriptors (HODs), using a 9-point scale, and health utility of outcomes on a visual analogue scale., Results: Of 469 potential questions identified, 72.5% were rated as important but not of high priority, and 25.4% as high priority. Each criterion was significantly associated with the overall importance rating. The overall importance rating means were 5.96 (SD 2.38) and 6.53 (SD 2.45) (P = 0.25) for those randomized to rate questions with and without the criteria, respectively. The mean importance rating for 121 outcomes was 6.01 (SD 1.25), with 35.5% rated as critical for decision-making. Panelists provided health utility ratings for 127 outcomes, with a minimum mean rating of 0.12 (SD 0.10) and maximum of 0.91 (SD 0.15)., Conclusion: Our structured process provided information to help explain perspectives of question importance, to facilitate panels' outcome prioritization, and to facilitate decision-making in guideline development., (Copyright © 2021. Published by Elsevier Inc.)

Background: Prognostic models combine several prognostic factors to provide an estimate of the likelihood (or risk) of future events in individual patients, conditional on their prognostic factor values. A fundamental part of evaluating prognostic models is undertaking studies to determine whether their predictive performance, such as calibration and discrimination, is reproduced across settings. Systematic reviews and meta-analyses of studies evaluating prognostic models' performance are a necessary step for selection of models for clinical practice and for testing the underlying assumption that their use will improve outcomes, including patient's reassurance and optimal future planning., Methods: In this paper, we highlight key concepts in evaluating the certainty of evidence regarding the calibration of prognostic models., Results and Conclusion: Four concepts are key to evaluating the certainty of evidence on prognostic models' performance regarding calibration. The first concept is that the inference regarding calibration may take one of two forms: deciding whether one is rating certainty that a model's performance is satisfactory or, instead, unsatisfactory, in either case defining the threshold for satisfactory (or unsatisfactory) model performance. Second, inconsistency is the critical GRADE domain to deciding whether we are rating certainty in the model performance being satisfactory or unsatisfactory. Third, depending on whether one is rating certainty in satisfactory or unsatisfactory performance, different patterns of inconsistency of results across studies will inform ratings of certainty of evidence. Fourth, exploring the distribution of point estimates of observed to expected ratio across individual studies, and its determinants, will bear on the need for and direction of future research., (Copyright © 2021. Published by Elsevier Inc.)