Search

Introduction: GRADE and other Evidence to Decision (EtD) frameworks are widely used by guideline development groups (GDG) and other decision-makers. When GDGs judge the magnitude of desirable and undesirable health outcomes on EtDs, they typically categorize them as trivial, small, moderate or large. However, generic judgment or decision thresholds (DTs) that could guide the user about such estimates of effect size or serve as references for interpretation of findings are not yet available. The objective of this study was to empirically derive DTs for EtD judgments about the magnitude of dichotomously assessed health benefits and harms., Methods: We conducted a methodological randomised controlled trial to derive empirical DTs across conditions and health outcomes. We invited stakeholders including clinicians, epidemiologists, decision scientists, health research methodologists, experts in Health Technology Assessment (HTA), members of GDGs, patient representatives, and the public to participate in the trial. We employed randomly assigned case scenarios to elicit ranges of absolute risk differences (ARD) judged as small and moderate effects from study participants. We then used the collected data to derive empirical decision thresholds. We also investigated the validity of our DTs by measuring the agreement between judgments that were made by GDGs in the past and the judgments that our DTs approach would suggest if applied to the same guideline data., Results: A total of 445 stakeholders accessed the survey of which 409 were randomised and 288 rated at least one case scenario. Based on these participants, the study findings support our a priori hypothesis of a difference in the DTs for trivial, small, moderate, and large effects and are suggestive of a relation between raters' judgments and the joint measure of absolute effects and outcome values. The results permit the use and calculation of DTs for a variety of scenarios and we present three ways of how to use the results practically., Conclusions: In this trial we confirmed that empirically derived DTs discriminate between judgments on the EtDs. These DTs can be used for judgments about desirable and undesirable health effects in systematic reviews or to initiate and inform a discussion with a GDG. This ensures consistency in judgments across different guideline questions and promotes transparency in judgments., (Copyright © 2024. Published by Elsevier Inc.)

Background and Objectives: An emerging body of randomized controlled trials (RCTs) on COVID-19 vaccines has served as the evidence base for public health decision-making. While it is recommended that RCTs report results by health equity stratifiers to reduce bias in health care and gaps in research, it is unknown whether this was done in COVID-19 vaccine trials. To critically examine the use of health equity stratifiers in COVID-19 vaccine trials., Study Design and Setting: We conducted a methodological review of published COVID-19 vaccine trials available in the COVID-19 living Network Meta-Analysis systematic review database through February 8, 2023. Based on the PROGRESS-Plus framework, we examined the following health equity stratifiers: place of residence, race/ethnicity, occupation, gender/sex, religion, education, socio-economic status, social capital, age, disability, features of relationships, and temporary situations. We assessed each study in duplicate according to three criteria for comprehensive health-equity reporting: 1) describing participants, 2) reporting equity-relevant results, and 3) discussing equity-relevant implications of trial findings., Results: We reviewed 144 trial manuscripts. The most frequently used PROGRESS-Plus stratifiers to describe participants were age (100%), place of residence (100%), gender/sex (99%), and race/ethnicity (64%). Age was most often used to disaggregate or adjust results (67%), followed by gender or sex (35%). Discussions of equity-relevant implications often indicated limited generalizability of results concerning age (40% of studies). Half (47%) of the studies considered at least one health equity stratifier for all three criteria. No trials included stratifiers related to religion, socioeconomic status, sexual orientation, or features of relationships., Conclusion: COVID-19 vaccine trials provided a limited description of health equity stratifiers as defined by PROGRESS-Plus and infrequently disaggregated results or discussed the study implications as they related to health equity. Considering the health disparities exacerbated during the pandemic, increased uptake of PROGRESS-Plus in RCTs would support a more nuanced understanding of health disparities and better inform actions to improve health equity., Competing Interests: Declaration of competing interest All authors have no conflicts of interest to declare. This study receieved no funding., (Copyright © 2024 The Author(s). Published by Elsevier Inc. All rights reserved.)

Objectives: Evidence-based research (EBR) is the systematic and transparent use of prior research to inform a new study so that it answers questions that matter in a valid, efficient, and accessible manner. This study surveyed experts about existing (e.g., citation analysis) and new methods for monitoring EBR and collected ideas about implementing these methods., Study Design and Setting: We conducted a cross-sectional study via an online survey between November 2022 and March 2023. Participants were experts from the fields of evidence synthesis and research methodology in health research. Open-ended questions were coded by recurring themes; descriptive statistics were used for quantitative questions., Results: Twenty-eight expert participants suggested that citation analysis should be supplemented with content evaluation (not just what is cited but also in which context), content expert involvement, and assessment of the quality of cited systematic reviews. They also suggested that citation analysis could be facilitated with automation tools. They emphasized that EBR monitoring should be conducted by ethics committees and funding bodies before the research starts. Challenges identified for EBR implementation monitoring were resource constraints and clarity on responsibility for EBR monitoring., Conclusion: Ideas proposed in this study for monitoring the implementation of EBR can be used to refine methods and define responsibility but should be further explored in terms of feasibility and acceptability. Different methods may be needed to determine if the use of EBR is improving over time., Competing Interests: Declaration of competing interest The authors have no competing interests to declare., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Objectives: To enhance equity in clinical and epidemiological research, it is crucial to understand researcher motivations for conducting equity-relevant studies. Therefore, we evaluated author motivations in a randomly selected sample of equity-relevant observational studies published during the COVID-19 pandemic., Study Design and Setting: We searched MEDLINE for studies from 2020 to 2022, resulting in 16,828 references. We randomly selected 320 studies purposefully sampled across income setting (high vs low-middle-income), COVID-19 topic (vs non-COVID-19), and focus on populations experiencing inequities. Of those, 206 explicitly mentioned motivations which we analyzed thematically. We used discourse analysis to investigate the reasons behind emerging motivations., Results: We identified the following motivations: (1) examining health disparities, (2) tackling social determinants to improve access, and (3) addressing knowledge gaps in health equity. Discourse analysis showed motivations stem from commitments to social justice and recognizing the importance of highlighting it in research. Other discourses included aspiring to improve health-care efficiency, wanting to understand cause-effect relationships, and seeking to contribute to an equitable evidence base., Conclusion: Understanding researchers' motivations for assessing health equity can aid in developing guidance that tailors to their needs. We will consider these motivations in developing and sharing equity guidance to better meet researchers' needs., Competing Interests: Declaration of competing interest L.G.C. is employed by the Pan American Health Organization (PAHO/WHO). The ideas expressed in this manuscript are the authors' own and do not necessarily reflect the decisions and policies of PAHO/WHO. There are no competing interests for any other author., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Objectives: To make informed decisions, the general population should have access to accessible and understandable health recommendations. To compare understanding, accessibility, usability, satisfaction, intention to implement, and preference of adults provided with a digital "Plain Language Recommendation" (PLR) format vs. the original "Standard Language Version" (SLV)., Study Design and Setting: An allocation-concealed, blinded, controlled superiority trial and a qualitative study to understand participant preferences. An international on-line survey. 488 adults with some English proficiency. 67.8% of participants identified as female, 62.3% were from the Americas, 70.1% identified as white, 32.2% had a bachelor's degree as their highest completed education, and 42% said they were very comfortable reading health information. In collaboration with patient partners, advisors, and the Cochrane Consumer Network, we developed a plain language format of guideline recommendations (PLRs) to compare their effectiveness vs. the original standard language versions (SLVs) as published in the source guideline. We selected two recommendations about COVID-19 vaccine, similar in their content, to compare our versions, one from the World Health Organization (WHO) and one from Centers for Disease Control and Prevention (CDC). The primary outcome was understanding, measured as the proportion of correct responses to seven comprehension questions. Secondary outcomes were accessibility, usability, satisfaction, preference, and intended behavior, measured on a 1-7 scale., Results: Participants randomized to the PLR group had a higher proportion of correct responses to the understanding questions for the WHO recommendation (mean difference [MD] of 19.8%, 95% confidence interval [CI] 14.7-24.9%; P < 0.001) but this difference was smaller and not statistically significant for the CDC recommendation (MD of 3.9%, 95% CI -0.7% to 8.3%; P = 0.096). However, regardless of the recommendation, participants found the PLRs more accessible, (MD of 1.2 on the seven-point scale, 95% CI 0.9-1.4%; P < 0.001) and more satisfying (MD of 1.2, 95% CI 0.9-1.4%; P < 0.001). They were also more likely to follow the recommendation if they had not already followed it (MD of 1.2, 95% CI 0.7-1.8%; P < 0.001) and share it with other people they know (MD of 1.9, 95% CI 0.5-1.2%; P < 0.001). There was no significant difference in the preference between the two formats (MD of -0.3, 95% CI -0.5% to 0.03%; P = 0.078). The qualitative interviews supported and contextualized these findings., Conclusion: Health information provided in a PLR format improved understanding, accessibility, usability, and satisfaction and thereby has the potential to shape public decision-making behavior., Competing Interests: Declaration of competing interest Dr. Pottie is supported by the Ian McWhinney Research Chair in Family Medicine, Western University. Dr. Butcher declares consulting fees from Nobias Therapeutics, Inc. Dr. Hartling is supported by a Canada Research Chair in Knowledge Synthesis and Translation. Dr. Scott is supported by a Canada Research Chair in Knowledge Translation in Child Health. Dr. Hartling and Dr. Scott are Distinguished Researchers with the Stollery Science Lab supported by the Stollery Children's Hospital Foundation. Dr. Richards is a volunteer vice president of the Canadian Arthritis Patient Alliance, a volunteer-run organization that receives independent grants from pharmaceutical companies., (Copyright © 2023 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: To determine, in a sample of noninferiority trials, the noninferiority margins (NIMs) and their relation to effect estimates in superiority trials, the rationale being that in general NIMs should be no larger than effects declared important in superiority trials., Study Design and Setting: To identify cardiovascular trials published in high-impact journals with a statistically significant primary outcome involving mortality, we systematically searched PubMed, Embase, and MEDLINE databases (January 2015-July 2020). We documented the NIMs and determined the proportion of trials with NIMs larger than the median effect estimates across superiority trials., Results: From 1,477 screened titles, 65 (39 noninferiority, 26 superiority) trials proved eligible. The NIMs ranged from risk difference of 0.54-10%. The effect estimate across superiority trials was a median risk difference of 2.1% (interquartile range 1.5-4.9); the NIM was larger than 2.1% in 28 (71.8%) noninferiority trials and larger than 1.5%, which was the lower bound of the interquartile range in 32 (82.1%) noninferiority trials., Conclusion: The very wide range of noninferiority margins and the proportion above a threshold that most would consider an important mortality reduction suggest that clinicians and guideline panels should focus on study results, paying little attention to authors' noninferiority margins., Competing Interests: Declaration of Competing Interest, (Copyright © 2023 Elsevier Inc. All rights reserved.)

Objectives: To evaluate the support from the available guidance on reporting of health equity in research for our candidate items and to identify additional items for the Strengthening Reporting of Observational studies in Epidemiology-Equity extension., Study Design and Setting: We conducted a scoping review by searching Embase, MEDLINE, CINAHL, Cochrane Methodology Register, LILACS, and Caribbean Center on Health Sciences Information up to January 2022. We also searched reference lists and gray literature for additional resources. We included guidance and assessments (hereafter termed "resources") related to conduct and/or reporting for any type of health research with or about people experiencing health inequity., Results: We included 34 resources, which supported one or more candidate items or contributed to new items about health equity reporting in observational research. Each candidate item was supported by a median of six (range: 1-15) resources. In addition, 12 resources suggested 13 new items, such as "report the background of investigators"., Conclusion: Existing resources for reporting health equity in observational studies aligned with our interim checklist of candidate items. We also identified additional items that will be considered in the development of a consensus-based and evidence-based guideline for reporting health equity in observational studies., Competing Interests: Declaration of Competing Interest Although Luis Gabriel Cuervo is an official of the Pan American Health Organization, the views expressed in this publication are his sole responsibility and do not necessarily represent the decisions or policies of the Pan American Health Organization (PAHO/WHO)., (Copyright © 2023 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: To evaluate alternative formats of summary of findings (SoF) tables for single comparison with multiple outcomes., Study Design and Setting: We conducted a three-arm randomized controlled noninferiority trial (RCT) in the following systematic review (SR) users: researchers, clinical practice guideline developers, health care providers, policymakers, and knowledge transfer organizations to measure understanding, accessibility, satisfaction, and preference across the current grading of recommendations assessment, development, and evaluation (GRADE) SoF, an alternative GRADE SoF, or an adapted evidence-based practice center (EPC) program SoF table., Results: One Hundred Seventy-Nine participants were randomized, and 129 participants completed the RCT (n = 47 current GRADE, n = 41 alternative GRADE, n = 41 adapted EPC). Understanding the certainty of evidence and treatment effect was comparable across groups. The adapted EPC SoF table was inferior for quantifying risk and RD compared to the alternatives (<35% correct vs. >85% correct). Participants reported increased satisfaction when SoF tables presented number needed to treat (NNT), anticipated absolute effect differences, and narrative syntheses for evidence that could not be meta-analyzed. Participants reported accessibility to information as significantly better in both GRADE SoF tables, when compared with the adapted EPC SoF table. Participants preferred the alternative GRADE SoF table format., Conclusion: The alternative GRADE SoF table is a promising format for SR users preferring a comprehensive presentation of SR results for single comparisons., (Copyright © 2022 Elsevier Inc. All rights reserved.)

Objectives: Grading of Recommendations Assessment, Development and Evaluation (GRADE) guidance to rate the certainty domain of imprecision is presently not fully operationalized for rating down by two levels and when different baseline risk or uncertainty in these risks are considered. In addition, there are scenarios in which lowering the certainty of evidence by three levels for imprecision is more appropriate than lowering it by two levels. In this article, we conceptualize and operationalize rating down for imprecision by one, two and three levels for imprecision using the contextualized GRADE approaches and making decisions., Methods: Through iterative discussions and refinement in online meetings and through email communication, we developed draft guidance to rating the certainty of evidence down by up to three levels based on examples. The lead authors revised the approach according to the feedback and the comments received during these meetings and developed GRADE guidance for how to apply it. We presented a summary of the results to all attendees of the GRADE Working Group meeting for feedback in October 2021 (approximately 80 people) where the approach was formally approved., Results: This guidance provides GRADE's novel approach for the considerations about rating down for imprecision by one, two and three levels based on serious, very serious and extremely serious concerns. The approach includes identifying or defining thresholds for health outcomes that correspond to trivial or none, small, moderate or large effects and using them to rate imprecision. It facilitates the use of evidence to decision frameworks and also provides guidance for how to address imprecision about implausible large effects and trivial or no effects using the concept of the 'review information size' and for varying baseline risks. The approach is illustrated using practical examples, an online calculator and graphical displays and can be applied to dichotomous and continuous outcomes., Conclusion: In this GRADE guidance article, we provide updated guidance for how to rate imprecision using the partially and fully contextualized GRADE approaches for making recommendations or decisions, considering alternate baseline risks and for both dichotomous and continuous outcomes., (Copyright © 2022 Elsevier Inc. All rights reserved.)

Background and Objectives: To develop a digital communication tool to improve the implementation of up-to-date COVID-19 recommendations. Specifically, to improve patient, caregiver and public understanding of healthcare recommendations on prevention, diagnoses and treatment., Methods: Multi-stakeholder engagement design. In conjunction with the COVID-19 Recommendations and Gateway to Contextualization RecMap, we co-developed a stakeholder prioritization, drafting and editing process to enhance guideline communication and understanding., Results: This paper presents the multi-stakeholder development process with three distinct plain language recommendation formats: formal recommendation, good practice statement, and additional guidance. Our case study of COVID-19 plain language recommendations PLRs addresses both public health interventions (e.g., vaccination, face masks) and clinical interventions (e.g., home pulse oximetry)., Conclusion: This paper presents a novel approach to engaging stakeholders in improving the communication and understanding of published guidelines during the COVID-19 pandemic., (Copyright © 2022 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: To describe divergence between actionable statements issued by coronavirus disease 2019 (COVID-19) guideline developers cataloged on the "COVID-19 Recommendations and Gateway to Contextualization" platform., Study Design and Setting: We defined divergence as at least two comparable actionable statements with different explicit judgments of strength, direction, or subgroup consideration of the population or intervention. We applied a content analysis to compare guideline development methods for a sample of diverging statements and to evaluate factors associated with divergence., Results: Of the 138 guidelines evaluated, 85 (62%) contained at least one statement that diverged from another guideline. We identified 223 diverging statements in these 85 guidelines. We grouped statements into 66 clusters. Each cluster addressed the same population, intervention, and comparator group or just similar interventions. Clinical practice statements were more likely to diverge in an explicit judgment of strength or direction compared to public health statements. Statements were more likely to diverge in strength than direction. The date of publication, used evidence, interpretation of evidence, and contextualization considerations were associated with divergence., Conclusion: More than half of the assessed guidelines issued at least one diverging statement. This study helps in understanding the types of differences between guidelines issuing comparable statements and factors associated with their divergence., (Copyright © 2022. Published by Elsevier Inc.)

Objective: To propose a taxonomy and framework that identifies and presents actionable statements in guidelines., Study Design and Setting: We took an iterative approach reviewing case studies of guidelines produced by the World Health Organization and the American Society of Hematology to develop an initial conceptual framework. We then tested it using randomly selected recommendations from published guidelines addressing COVID-19 from different organizations, evaluated its results, and refined it before retesting. The urgency and availability of evidence for development of these recommendations varied. We consulted with experts in research methodology and guideline developers to improve the final framework., Results: The resulting taxonomy and framework distinguishes five types of actional statements: formal recommendations; research recommendations; good practice statements; implementation considerations, tools and tips; and informal recommendations. These statements should respond to a priori established criteria and require a clear structure and recognizable presentation in a guideline. Most importantly, this framework identifies informal recommendations that differ from formal recommendations by how they consider evidence and in their development process., Conclusion: The identification, standardization and explicit labelling of actionable statements according to the framework may support guideline developers to create actionable statements with clear intent, avoid informal recommendations and improve their understanding and implementation by users., (Copyright © 2021. Published by Elsevier Inc.)

Objective: To systematically review the epidemiology of prerandomized run-in periods in randomized controlled trials (RCTs) of chronic diseases., Study Design and Setting: Meta-epidemiologic study of all RCTs from the four highest impact medical journals from 2011 to 2016. Eligible trials included parallel RCTs that evaluated pharmacologic therapies in adults with chronic diseases with a minimum follow-up of 24 weeks., Results: Of 262 eligible manuscripts, 48 (18.3%), representing 42 unique RCTs, included run-in periods. Run-in periods were most common in cardiovascular disease and diabetes trials. Of the 42 RCTs, in 22 patients received the experimental therapy, 15 placebo, 4 both (either sequentially or in combination), and one did not report the run-in period drug. The median run-in period duration was 28 days (Q1: Q3 14: 66 days). Reasons for including a run-in period included ensuring eligibility criteria were met (18, 42.9%), excluding participants with nonadherence (18, 42.9%) and intolerances to therapy (15, 35.7%), and to standardize therapy prior to randomization (8, 19.0%). The median run-in completion rate was 77.4% (Q1: Q3 62.2:87.8%)., Conclusions: Run-in periods are uncommon in RCTs of chronic drug treatments and when used, their reporting is heterogeneous. Further research to improve the design, use, and reporting of run-in periods is necessary., (Copyright © 2020 Elsevier Inc. All rights reserved.)

Objectives: The aim of the study was to develop a Grading of Recommendations, Assessment, Development and Evaluation (GRADE) summary of findings (SoF) table format that displays the critical information from a network meta-analysis (NMA)., Study Design and Setting: We applied a user experience model for data analysis based on four rounds of semistructured interviews., Results: We interviewed 32 stakeholders who conduct or use MA. Four rounds of interviews produced six candidate NMA-SoF tables. Users found a final NMA-SoF table that included the following components highly acceptable: (1) details of the clinical question (PICO), (2) a plot depicting network geometry, (3) relative and absolute effect estimates, (4) certainty of evidence, (5) ranking of treatments, and (6) interpretation of findings., Conclusion: Using stakeholder feedback, we developed a new GRADE NMA-SoF table that includes the relevant components that facilitate understanding NMA findings and health decision-making., (Copyright © 2019. Published by Elsevier Inc.)

Objective: To describe and summarize equity reporting in human immunodeficiency virus (HIV) systematic reviews and explore the extent to which equity issues are addressed and reported in HIV reviews using the PROGRESS Plus framework., Study Design and Setting: Application of the PROGRESS Plus framework to a bibliometric analysis of HIV reviews in the Cochrane Database of Systematic Reviews., Results: The analysis included 103 reviews published as of March 2014, with a median of five studies per review (first quartile; Q1 = 2; third quartile; Q3 = 11). Reporting of PROGRESS Plus factors was as follows: Place of residence (low, middle, and high income; 55.3%), place of residence (urban or rural; 24.3%), race or ethnicity (20.4%), occupation (10.7%), gender (65.0%), religion (1.9%), education (7.8%), socioeconomic position (10.7%), social networks and capital (1.0%), age (1.9%), and sexual orientation (3.8%)., Conclusion: Gaps in the reporting of relevant equity indicators were identified within Cochrane HIV systematic review indicating that research is not consistently conducted through an equity lens. There is a need to incorporate PROGRESS Plus factors into both primary and secondary studies., (Copyright © 2016 Elsevier Inc. All rights reserved.)

Objectives: The current format of summary of findings (SoFs) tables for presenting effect estimates and associated quality of evidence improve understanding and assist users finding key information in systematic reviews. Users of SoF tables have demanded alternative formats to express findings from systematic reviews., Study Design and Setting: We conducted a randomized controlled trial among systematic review users to compare the relative merits of a new format with the current formats of SoF tables regarding understanding, accessibility of information, satisfaction, and preference. Our primary goal was to show that the new format is not inferior to the current format., Results: Of 390 potentially eligible subjects, 290 were randomized. Of seven items testing understanding, three showed similar results, two showed small differences favoring the new format, and two (understanding risk difference and quality of the evidence associated with a treatment effect) showed large differences favoring the new format [63% (95% confidence interval {CI}: 55, 71) and 62% (95% CI: 52, 71) more correct answers, respectively]. Respondents rated information in the alternative format as more accessible overall and preferred the new format over the current format., Conclusions: While providing at least similar levels of understanding for some items and increased understanding for others, users prefer the new format of SoF tables., (Copyright © 2016 Elsevier Inc. All rights reserved.)

Objectives: To summarize the current gaps in human immunodeficiency virus (HIV) research evidence, describe the adequacy of reporting "implications for research," and map the number of studies that inform reviews with the prevalence of HIV for each country., Study Design and Setting: A bibliometric analyses of HIV reviews in the Cochrane Database of Systematic Reviews with content analysis of the "implications for research" section., Results: We analyzed 103 high-quality reviews published as of March 2014. They included a median of five studies (min 0 and max 44). Almost all the reviews recommended more trials (89.3%). Limitations in trial design, duration, setting, sample size, and choice of participants were also noted. Reporting of EPICOT+ items was as follows: evidence (35.9%), population (57.3%), intervention (71.8%), comparison (20.4%), outcomes (57.3%), time stamp (34.0%), and disease burden (13.6%). The primary studies were conducted in 67 countries. Six of the top 10 countries in which primary studies were conducted had a high HIV prevalence., Conclusion: Knowledge gaps were identified for research in younger participants, over longer periods, using more pragmatic interventions, conducted in resource-limited settings and incorporating economic evaluations. Implications for research are not always reported according to the EPICOT+ format. Not all countries with a high prevalence of HIV are contributing sufficiently to HIV research., (Copyright © 2015 Elsevier Inc. All rights reserved.)