Search

Importance: Mendelian randomization (MR) studies use genetic variation associated with modifiable exposures to assess their possible causal relationship with outcomes and aim to reduce potential bias from confounding and reverse causation. Objective: To develop the STROBE-MR Statement as a stand-alone extension to the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guideline for the reporting of MR studies. Design, Setting, and Participants: The development of the STROBE-MR Statement followed the Enhancing the Quality and Transparency of Health Research (EQUATOR) framework guidance and used the STROBE Statement as a starting point to draft a checklist tailored to MR studies. The project was initiated in 2018 by reviewing the literature on the reporting of instrumental variable and MR studies. A group of 17 experts, including MR methodologists, MR study design users, developers of previous reporting guidelines, and journal editors, participated in a workshop in May 2019 to define the scope of the Statement and draft the checklist. The draft checklist was published as a preprint in July 2019 and discussed on the preprint platform, in social media, and at the 4th Mendelian Randomization Conference. The checklist was then revised based on comments, further refined through 2020, and finalized in July 2021. Findings: The STROBE-MR checklist is organized into 6 sections (Title and Abstract, Introduction, Methods, Results, Discussion, and Other Information) and includes 20 main items and 30 subitems. It covers both 1-sample and 2-sample MR studies that assess 1 or multiple exposures and outcomes, and addresses MR studies that follow a genome-wide association study and are reported in the same article. The checklist asks authors to justify why MR is a helpful method to address the study question and state prespecified causal hypotheses. The measurement, quality, and selection of genetic variants must be described and attempts to assess validi

Importance: Mendelian randomization (MR) studies use genetic variation associated with modifiable exposures to assess their possible causal relationship with outcomes and aim to reduce potential bias from confounding and reverse causation. Objective: To develop the STROBE-MR Statement as a stand-alone extension to the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guideline for the reporting of MR studies. Design, Setting, and Participants: The development of the STROBE-MR Statement followed the Enhancing the Quality and Transparency of Health Research (EQUATOR) framework guidance and used the STROBE Statement as a starting point to draft a checklist tailored to MR studies. The project was initiated in 2018 by reviewing the literature on the reporting of instrumental variable and MR studies. A group of 17 experts, including MR methodologists, MR study design users, developers of previous reporting guidelines, and journal editors, participated in a workshop in May 2019 to define the scope of the Statement and draft the checklist. The draft checklist was published as a preprint in July 2019 and discussed on the preprint platform, in social media, and at the 4th Mendelian Randomization Conference. The checklist was then revised based on comments, further refined through 2020, and finalized in July 2021. Findings: The STROBE-MR checklist is organized into 6 sections (Title and Abstract, Introduction, Methods, Results, Discussion, and Other Information) and includes 20 main items and 30 subitems. It covers both 1-sample and 2-sample MR studies that assess 1 or multiple exposures and outcomes, and addresses MR studies that follow a genome-wide association study and are reported in the same article. The checklist asks authors to justify why MR is a helpful method to address the study question and state prespecified causal hypotheses. The measurement, quality, and selection of genetic variants must be described and attempts to assess validi

Importance: Mendelian randomization (MR) studies use genetic variation associated with modifiable exposures to assess their possible causal relationship with outcomes and aim to reduce potential bias from confounding and reverse causation. Objective: To develop the STROBE-MR Statement as a stand-alone extension to the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guideline for the reporting of MR studies. Design, Setting, and Participants: The development of the STROBE-MR Statement followed the Enhancing the Quality and Transparency of Health Research (EQUATOR) framework guidance and used the STROBE Statement as a starting point to draft a checklist tailored to MR studies. The project was initiated in 2018 by reviewing the literature on the reporting of instrumental variable and MR studies. A group of 17 experts, including MR methodologists, MR study design users, developers of previous reporting guidelines, and journal editors, participated in a workshop in May 2019 to define the scope of the Statement and draft the checklist. The draft checklist was published as a preprint in July 2019 and discussed on the preprint platform, in social media, and at the 4th Mendelian Randomization Conference. The checklist was then revised based on comments, further refined through 2020, and finalized in July 2021. Findings: The STROBE-MR checklist is organized into 6 sections (Title and Abstract, Introduction, Methods, Results, Discussion, and Other Information) and includes 20 main items and 30 subitems. It covers both 1-sample and 2-sample MR studies that assess 1 or multiple exposures and outcomes, and addresses MR studies that follow a genome-wide association study and are reported in the same article. The checklist asks authors to justify why MR is a helpful method to address the study question and state prespecified causal hypotheses. The measurement, quality, and selection of genetic variants must be described and attempts to assess validi

Mendelian randomisation (MR) studies allow a better understanding of the causal effects of modifiable exposures on health outcomes, but the published evidence is often hampered by inadequate reporting. Reporting guidelines help authors effectively communicate all critical information about what was done and what was found. STROBE-MR (strengthening the reporting of observational studies in epidemiology using mendelian randomisation) assists authors in reporting their MR research clearly and transparently. Adopting STROBE-MR should help readers, reviewers, and journal editors evaluate the quality of published MR studies. This article explains the 20 items of the STROBE-MR checklist, along with their meaning and rationale, using terms defined in a glossary. Examples of transparent reporting are used for each item to illustrate best practices.

This research on data extraction methods applies recent advances in natural language processing to evidence synthesis based on medical texts. Texts of interest include abstracts of clinical trials in English and in multilingual contexts. The main focus is on information characterized via the Population, Intervention, Comparator, and Outcome (PICO) framework, but data extraction is not limited to these fields. Recent neural network architectures based on transformers show capacities for transfer learning and increased performance on downstream natural language processing tasks such as universal reading comprehension, brought forward by this architecture's use of contextualized word embeddings and self-attention mechanisms. This paper contributes to solving problems related to ambiguity in PICO sentence prediction tasks, as well as highlighting how annotations for training named entity recognition systems are used to train a high-performing, but nevertheless flexible architecture for question answering in systematic review automation. Additionally, it demonstrates how the problem of insufficient amounts of training annotations for PICO entity extraction is tackled by augmentation. All models in this paper were created with the aim to support systematic review (semi)automation. They achieve high F1 scores, and demonstrate the feasibility of applying transformer-based classification methods to support data mining in the biomedical literature.

OBJECTIVES: Primary objective: To estimate the effect of corticosteroids compared with usual care or placebo on mortality up to 28 days after randomization. Secondary objectives: To examine whether the effect of corticosteroids compared with usual care or placebo on mortality up to 28 days after randomization varies between subgroups related to treatment characteristics, disease severity at the time of randomization, patient characteristics, or risk of bias. To examine the effect of corticosteroids compared with usual care or placebo on serious adverse events.STUDY DESIGN: Prospective meta-analysis of randomized controlled trials. Both placebo-controlled and open-label trials are eligible.PARTICIPANTS: Hospitalised, critically ill patients with suspected or confirmed COVID-19.INTERVENTION AND COMPARATOR: Intervention groups will have received therapeutic doses of a steroid (dexamethasone, hydrocortisone or methylprednisolone) with IV or oral administration immediately after randomization. The comparator groups will have received standard of care or usual care or placebo.MAIN OUTCOME: All-cause mortality up to 28 days after randomization.SEARCH METHODS: Systematic searching of clinicaltrials.gov , EudraCT, the WHO ISRCTN registry, and the Chinese clinical trials registry. Additionally, research and WHO networks will be asked for relevant trials.RISK OF BIAS ASSESSMENTS: These will be based on the Cochrane RoB 2 tool, and will use structured information provided by the trial investigators on a form designed for this prospective meta-analysis. We will use GRADE to assess the certainty of the evidence.STATISTICAL ANALYSES: Trial investigators will provide data on the numbers of participants who did and did not experience each outcome according to intervention group, overall and in specified subgroups. We will conduct fixed-effect (primary analysis) and random-effects (Paule-Mandel estimate of heterogeneity and Hartung

Importance: Effective therapies for patients with coronavirus disease 2019 (COVID-19) are needed, and clinical trial data have demonstrated that low-dose dexamethasone reduced mortality in hospitalized patients with COVID-19 who required respiratory support. Objective: To estimate the association between administration of corticosteroids compared with usual care or placebo and 28-day all-cause mortality. Design, Setting, and Participants: Prospective meta-analysis that pooled data from 7 randomized clinical trials that evaluated the efficacy of corticosteroids in 1703 critically ill patients with COVID-19. The trials were conducted in 12 countries from February 26, 2020, to June 9, 2020, and the date of final follow-up was July 6, 2020. Pooled data were aggregated from the individual trials, overall, and in predefined subgroups. Risk of bias was assessed using the Cochrane Risk of Bias Assessment Tool. Inconsistency among trial results was assessed using the I2statistic. The primary analysis was an inverse variance-weighted fixed-effect meta-analysis of overall mortality, with the association between the intervention and mortality quantified using odds ratios (ORs). Random-effects meta-analyses also were conducted (with the Paule-Mandel estimate of heterogeneity and the Hartung-Knapp adjustment) and an inverse variance-weighted fixed-effect analysis using risk ratios. Exposures: Patients had been randomized to receive systemic dexamethasone, hydrocortisone, or methylprednisolone (678 patients) or to receive usual care or placebo (1025 patients). Main Outcomes and Measures: The primary outcome measure was all-cause mortality at 28 days after randomization. A secondary outcome was investigator-defined serious adverse events. Results: A total of 1703 patients (median age, 60 years [interquartile range, 52-68 years]; 488 [29%] women) were included in the analysis. Risk of bias was assessed as "low" for 6 of the 7 mortality results and as "some concerns" in

OBJECTIVES: Primary objective: To estimate the effect of corticosteroids compared with usual care or placebo on mortality up to 28 days after randomization. Secondary objectives: To examine whether the effect of corticosteroids compared with usual care or placebo on mortality up to 28 days after randomization varies between subgroups related to treatment characteristics, disease severity at the time of randomization, patient characteristics, or risk of bias. To examine the effect of corticosteroids compared with usual care or placebo on serious adverse events.STUDY DESIGN: Prospective meta-analysis of randomized controlled trials. Both placebo-controlled and open-label trials are eligible.PARTICIPANTS: Hospitalised, critically ill patients with suspected or confirmed COVID-19.INTERVENTION AND COMPARATOR: Intervention groups will have received therapeutic doses of a steroid (dexamethasone, hydrocortisone or methylprednisolone) with IV or oral administration immediately after randomization. The comparator groups will have received standard of care or usual care or placebo.MAIN OUTCOME: All-cause mortality up to 28 days after randomization.SEARCH METHODS: Systematic searching of clinicaltrials.gov , EudraCT, the WHO ISRCTN registry, and the Chinese clinical trials registry. Additionally, research and WHO networks will be asked for relevant trials.RISK OF BIAS ASSESSMENTS: These will be based on the Cochrane RoB 2 tool, and will use structured information provided by the trial investigators on a form designed for this prospective meta-analysis. We will use GRADE to assess the certainty of the evidence.STATISTICAL ANALYSES: Trial investigators will provide data on the numbers of participants who did and did not experience each outcome according to intervention group, overall and in specified subgroups. We will conduct fixed-effect (primary analysis) and random-effects (Paule-Mandel estimate of heterogeneity and Hartung

Studies combined in a meta-analysis often have differences in their design and conduct that can lead to heterogeneous results. A random-effects model accounts for these differences in the underlying study effects, which includes a heterogeneity variance parameter. The DerSimonian-Laird method is often used to estimate the heterogeneity variance, but simulation studies have found the method can be biased and other methods are available. This paper compares the properties of nine different heterogeneity variance estimators using simulated meta-analysis data. Simulated scenarios include studies of equal size and of moderate and large differences in size. Results confirm that the DerSimonian-Laird estimator is negatively biased in scenarios with small studies and in scenarios with a rare binary outcome. Results also show the Paule-Mandel method has considerable positive bias in meta-analyses with large differences in study size. We recommend the method of restricted maximum likelihood (REML) to estimate the heterogeneity variance over other methods. However, considering that meta-analyses of health studies typically contain few studies, the heterogeneity variance estimate should not be used as a reliable gauge for the extent of heterogeneity in a meta-analysis. The estimated summary effect of the meta-analysis and its confidence interval derived from the Hartung-Knapp-Sidik-Jonkman method are more robust to changes in the heterogeneity variance estimate and show minimal deviation from the nominal coverage of 95% under most of our simulated scenarios.

Studies combined in a meta-analysis often have differences in their design and conduct that can lead to heterogeneous results. A random-effects model accounts for these differences in the underlying study effects, which includes a heterogeneity variance parameter. The DerSimonian-Laird method is often used to estimate the heterogeneity variance, but simulation studies have found the method can be biased and other methods are available. This paper compares the properties of nine different heterogeneity variance estimators using simulated meta-analysis data. Simulated scenarios include studies of equal size and of moderate and large differences in size. Results confirm that the DerSimonian-Laird estimator is negatively biased in scenarios with small studies and in scenarios with a rare binary outcome. Results also show the Paule-Mandel method has considerable positive bias in meta-analyses with large differences in study size. We recommend the method of restricted maximum likelihood (REML) to estimate the heterogeneity variance over other methods. However, considering that meta-analyses of health studies typically contain few studies, the heterogeneity variance estimate should not be used as a reliable gauge for the extent of heterogeneity in a meta-analysis. The estimated summary effect of the meta-analysis and its confidence interval derived from the Hartung-Knapp-Sidik-Jonkman method are more robust to changes in the heterogeneity variance estimate and show minimal deviation from the nominal coverage of 95% under most of our simulated scenarios.

Major depressive disorder is one of the most common, burdensome, and costly psychiatric disorders worldwide in adults. Pharmacological and non-pharmacological treatments are available; however, because of inadequate resources, antidepressants are used more frequently than psychological interventions. Prescription of these agents should be informed by the best available evidence. Therefore, we aimed to update and expand our previous work to compare and rank antidepressants for the acute treatment of adults with unipolar major depressive disorder.We did a systematic review and network meta-analysis. We searched Cochrane Central Register of Controlled Trials, CINAHL, Embase, LILACS database, MEDLINE, MEDLINE In-Process, PsycINFO, the websites of regulatory agencies, and international registers for published and unpublished, double-blind, randomised controlled trials from their inception to Jan 8, 2016. We included placebo-controlled and head-to-head trials of 21 antidepressants used for the acute treatment of adults (≥18 years old and of both sexes) with major depressive disorder diagnosed according to standard operationalised criteria. We excluded quasi-randomised trials and trials that were incomplete or included 20% or more of participants with bipolar disorder, psychotic depression, or treatment-resistant depression; or patients with a serious concomitant medical illness. We extracted data following a predefined hierarchy. In network meta-analysis, we used group-level data. We assessed the studies' risk of bias in accordance to the Cochrane Handbook for Systematic Reviews of Interventions, and certainty of evidence using the Grading of Recommendations Assessment, Development and Evaluation framework. Primary outcomes were efficacy (response rate) and acceptability (treatment discontinuations due to any cause). We estimated summary odds ratios (ORs) using pairwise and network meta-analysis with random effects. This study is registered with PROSP

BACKGROUND: We estimated the effectiveness of serial B-type natriuretic peptide (BNP) blood testing to guide up-titration of medication compared with symptom-guided up-titration of medication in patients with heart failure (HF).METHODS: Systematic review and meta-analysis of randomised controlled trials (RCTs). We searched: MEDLINE (Ovid) 1950 to 9/06/2016; Embase (Ovid), 1980 to 2016 week 23; the Cochrane Library; ISI Web of Science (Citations Index and Conference Proceedings). The primary outcome was all-cause mortality; secondary outcomes were death related to HF, cardiovascular death, all-cause hospital admission, hospital admission for HF, adverse events, and quality of life. IPD were sought from all RCTs identified. Random-effects meta-analyses (two-stage) were used to estimate hazard ratios (HR) and confidence intervals (CIs) across RCTs, including HR estimates from published reports of studies that did not provide IPD. We estimated treatment-by-covariate interactions for age, gender, New York Heart Association (NYHA) class, HF type; diabetes status and baseline BNP subgroups. Dichotomous outcomes were analysed using random-effects odds ratio (OR) with 95% CI.RESULTS: We identified 14 eligible RCTs, five providing IPD. BNP-guided therapy reduced the hazard of hospital admission for HF by 19% (13 RCTs, HR 0.81, 95% CI 0.68 to 0.98) but not all-cause mortality (13 RCTs; HR 0.87, 95% CI 0.75 to 1.01) or cardiovascular mortality (5 RCTs; OR 0.88, 95% CI 0.67 to 1.16). For all-cause mortality, there was a significant interaction between treatment strategy and age (p = 0.034, 11 RCTs; HR 0.70, 95% CI 0.53-0.92, patients < 75 years old and HR 1.07, 95% CI 0.84-1.37, patients ≥ 75 years old); ejection fraction (p = 0.026, 11 RCTs; HR 0.84, 95% CI 0.71-0.99, patients with heart failure with reduced ejection fraction (HFrEF); and HR 1.33, 95% CI 0.83-2.11, patients with heart failure with preserved ejection fraction (HFpEF)). Adverse events were sig

BACKGROUND: We estimated the effectiveness of serial B-type natriuretic peptide (BNP) blood testing to guide up-titration of medication compared with symptom-guided up-titration of medication in patients with heart failure (HF).METHODS: Systematic review and meta-analysis of randomised controlled trials (RCTs). We searched: MEDLINE (Ovid) 1950 to 9/06/2016; Embase (Ovid), 1980 to 2016 week 23; the Cochrane Library; ISI Web of Science (Citations Index and Conference Proceedings). The primary outcome was all-cause mortality; secondary outcomes were death related to HF, cardiovascular death, all-cause hospital admission, hospital admission for HF, adverse events, and quality of life. IPD were sought from all RCTs identified. Random-effects meta-analyses (two-stage) were used to estimate hazard ratios (HR) and confidence intervals (CIs) across RCTs, including HR estimates from published reports of studies that did not provide IPD. We estimated treatment-by-covariate interactions for age, gender, New York Heart Association (NYHA) class, HF type; diabetes status and baseline BNP subgroups. Dichotomous outcomes were analysed using random-effects odds ratio (OR) with 95% CI.RESULTS: We identified 14 eligible RCTs, five providing IPD. BNP-guided therapy reduced the hazard of hospital admission for HF by 19% (13 RCTs, HR 0.81, 95% CI 0.68 to 0.98) but not all-cause mortality (13 RCTs; HR 0.87, 95% CI 0.75 to 1.01) or cardiovascular mortality (5 RCTs; OR 0.88, 95% CI 0.67 to 1.16). For all-cause mortality, there was a significant interaction between treatment strategy and age (p = 0.034, 11 RCTs; HR 0.70, 95% CI 0.53-0.92, patients < 75 years old and HR 1.07, 95% CI 0.84-1.37, patients ≥ 75 years old); ejection fraction (p = 0.026, 11 RCTs; HR 0.84, 95% CI 0.71-0.99, patients with heart failure with reduced ejection fraction (HFrEF); and HR 1.33, 95% CI 0.83-2.11, patients with heart failure with preserved ejection fraction (HFpEF)). Adverse events were sig

Non-randomised studies of the effects of interventions are critical to many areas of healthcare evaluation, but their results may be biased. It is therefore important to understand and appraise their strengths and weaknesses. We developed ROBINS-I (“Risk Of Bias In Non-randomised Studies - of Interventions”), a new tool for evaluating risk of bias in estimates of the comparative effectiveness (harm or benefit) of interventions from studies that did not use randomisation to allocate units (individuals or clusters of individuals) to comparison groups. The tool will be particularly useful to those undertaking systematic reviews that include non-randomised studies.

ObjectiveTo assess the impact of inadequate water and sanitation on diarrhoeal disease in low- and middle-income settings.MethodsThe search strategy used Cochrane Library, MEDLINE & PubMed, Global Health, Embase and BIOSIS supplemented by screening of reference lists from previously published systematic reviews, to identify studies reporting on interventions examining the effect of drinking water and sanitation improvements in low- and middle-income settings published between 1970 and May 2013. Studies including randomised controlled trials, quasi-randomised trials with control group, observational studies using matching techniques and observational studies with a control group where the intervention was well defined were eligible. Risk of bias was assessed using a modified Ottawa-Newcastle scale. Study results were combined using meta-analysis and meta-regression to derive overall and intervention-specific risk estimates.ResultsOf 6819 records identified for drinking water, 61 studies met the inclusion criteria, and of 12,515 records identified for sanitation, 11 studies were included. Overall, improvements in drinking water and sanitation were associated with decreased risks of diarrhoea. Specific improvements, such as the use of water filters, provision of high-quality piped water and sewer connections, were associated with greater reductions in diarrhoea compared with other interventions.ConclusionsThe results show that inadequate water and sanitation are associated with considerable risks of diarrhoeal disease and that there are notable differences in illness reduction according to the type of improved water and sanitation implemented.

ObjectiveTo assess the impact of inadequate water and sanitation on diarrhoeal disease in low- and middle-income settings.MethodsThe search strategy used Cochrane Library, MEDLINE & PubMed, Global Health, Embase and BIOSIS supplemented by screening of reference lists from previously published systematic reviews, to identify studies reporting on interventions examining the effect of drinking water and sanitation improvements in low- and middle-income settings published between 1970 and May 2013. Studies including randomised controlled trials, quasi-randomised trials with control group, observational studies using matching techniques and observational studies with a control group where the intervention was well defined were eligible. Risk of bias was assessed using a modified Ottawa-Newcastle scale. Study results were combined using meta-analysis and meta-regression to derive overall and intervention-specific risk estimates.ResultsOf 6819 records identified for drinking water, 61 studies met the inclusion criteria, and of 12,515 records identified for sanitation, 11 studies were included. Overall, improvements in drinking water and sanitation were associated with decreased risks of diarrhoea. Specific improvements, such as the use of water filters, provision of high-quality piped water and sewer connections, were associated with greater reductions in diarrhoea compared with other interventions.ConclusionsThe results show that inadequate water and sanitation are associated with considerable risks of diarrhoeal disease and that there are notable differences in illness reduction according to the type of improved water and sanitation implemented.

Context: Meta-analyses are regularly used to inform healthcare decisions. Concerns have been expressed about the quality of meta-analyses and, in particular, about those supported by the pharmaceutical industry. Objective: The objective of this study is to compare the quality of pharmaceutical-industry-supported meta-analyses with academic meta-analyses and of meta-analyses published before and after companies started to disclose their data. Data Sources: We identified industry-supported meta-analyses by searching the Scopus bibliographic database, using author affiliations. We matched each industry-supported meta-analysis with an academic meta-analysis using high-level MeSH terms in PubMed. Study Selection: We included meta-analyses of randomized trials assessing the efficacy or safety of any pharmaceutical intervention in humans, published in 2002–2004 or 2008–2009. Cochrane reviews were excluded. Two individuals independently selected papers, with discrepancies resolved by two further individuals. Assessment: We developed and piloted a quality-assessment tool, consisting of 43 questions in four domains, with a key summary question covering each domain. Two individuals independently assessed each meta-analysis. Results: We examined 126 meta-analysis publications in 63 matched pairs. The average quality was low, with fewer than 50% adequate in three of the four domains. Industry-supported meta-analyses less often

Context: Meta-analyses are regularly used to inform healthcare decisions. Concerns have been expressed about the quality of meta-analyses and, in particular, about those supported by the pharmaceutical industry. Objective: The objective of this study is to compare the quality of pharmaceutical-industry-supported meta-analyses with academic meta-analyses and of meta-analyses published before and after companies started to disclose their data. Data Sources: We identified industry-supported meta-analyses by searching the Scopus bibliographic database, using author affiliations. We matched each industry-supported meta-analysis with an academic meta-analysis using high-level MeSH terms in PubMed. Study Selection: We included meta-analyses of randomized trials assessing the efficacy or safety of any pharmaceutical intervention in humans, published in 2002–2004 or 2008–2009. Cochrane reviews were excluded. Two individuals independently selected papers, with discrepancies resolved by two further individuals. Assessment: We developed and piloted a quality-assessment tool, consisting of 43 questions in four domains, with a key summary question covering each domain. Two individuals independently assessed each meta-analysis. Results: We examined 126 meta-analysis publications in 63 matched pairs. The average quality was low, with fewer than 50% adequate in three of the four domains. Industry-supported meta-analyses less often

Background: Because meta-analyses are increasingly prevalent and cited in the medical literature, it is important that tools are available to assess their methodological quality. When performing an empirical study of the quality of published meta-analyses, we found that existing tools did not place a strong emphasis on statistical and interpretational issues. Methods: We developed a quality-assessment tool using existing materials and expert judgment as a starting point, followed by multiple iterations of input from our working group, piloting, and discussion. After having used the tool for our empirical study, agreement for four key items in the tool was measured using weighted kappa coefficients. Results: Our tool contained 43 items divided into four key areas (data sources, analysis of individual studies, meta-analysis methods, and interpretation), and each area ended with a summary question. We also produced guidance for completing the tool. Agreement between raters was fair to moderate. Conclusions: The tool should usefully inform subsequent initiatives to develop quality-assessment tools for meta-analysis. We advocate use of consensus between independent raters when assessing statistical appropriateness and adequacy of interpretation in meta-analyses.

Background: Because meta-analyses are increasingly prevalent and cited in the medical literature, it is important that tools are available to assess their methodological quality. When performing an empirical study of the quality of published meta-analyses, we found that existing tools did not place a strong emphasis on statistical and interpretational issues. Methods: We developed a quality-assessment tool using existing materials and expert judgment as a starting point, followed by multiple iterations of input from our working group, piloting, and discussion. After having used the tool for our empirical study, agreement for four key items in the tool was measured using weighted kappa coefficients. Results: Our tool contained 43 items divided into four key areas (data sources, analysis of individual studies, meta-analysis methods, and interpretation), and each area ended with a summary question. We also produced guidance for completing the tool. Agreement between raters was fair to moderate. Conclusions: The tool should usefully inform subsequent initiatives to develop quality-assessment tools for meta-analysis. We advocate use of consensus between independent raters when assessing statistical appropriateness and adequacy of interpretation in meta-analyses.

Although meta-analyses are typically viewed as retrospective activities, they are increasingly being applied prospectively to provide up-to-date evidence on specific research questions. When meta-analyses are updated account should be taken of the possibility of false-positive findings due to repeated significance tests. We discuss the use of sequential methods for meta-analyses that incorporate random effects to allow for heterogeneity across studies. We propose a method that uses an approximate semi-Bayes procedure to update evidence on the among-study variance, starting with an informative prior distribution that might be based on findings from previous meta-analyses. We compare our methods with other approaches, including the traditional method of cumulative meta-analysis, in a simulation study and observe that it has Type I and Type II error rates close to the nominal level. We illustrate the method using an example in the treatment of bleeding peptic ulcers.

Although meta-analyses are typically viewed as retrospective activities, they are increasingly being applied prospectively to provide up-to-date evidence on specific research questions. When meta-analyses are updated account should be taken of the possibility of false-positive findings due to repeated significance tests. We discuss the use of sequential methods for meta-analyses that incorporate random effects to allow for heterogeneity across studies. We propose a method that uses an approximate semi-Bayes procedure to update evidence on the among-study variance, starting with an informative prior distribution that might be based on findings from previous meta-analyses. We compare our methods with other approaches, including the traditional method of cumulative meta-analysis, in a simulation study and observe that it has Type I and Type II error rates close to the nominal level. We illustrate the method using an example in the treatment of bleeding peptic ulcers.

To examine the extent of multiplicity of data in trial reports and to assess the impact of multiplicity on meta-analysis results.

To examine the extent of multiplicity of data in trial reports and to assess the impact of multiplicity on meta-analysis results.

To examine the extent of multiplicity of data in trial reports and to assess the impact of multiplicity on meta-analysis results.

Udgivelsesdato: 2009-null, OBJECTIVE: To study the inter-observer variation related to extraction of continuous and numerical rating scale data from trial reports for use in meta-analyses. DESIGN: Observer agreement study. DATA SOURCES: A random sample of 10 Cochrane reviews that presented a result as a standardised mean difference (SMD), the protocols for the reviews and the trial reports (n=45) were retrieved. DATA EXTRACTION: Five experienced methodologists and five PhD students independently extracted data from the trial reports for calculation of the first SMD result in each review. The observers did not have access to the reviews but to the protocols, where the relevant outcome was highlighted. The agreement was analysed at both trial and meta-analysis level, pairing the observers in all possible ways (45 pairs, yielding 2025 pairs of trials and 450 pairs of meta-analyses). Agreement was defined as SMDs that differed less than 0.1 in their point estimates or confidence intervals. RESULTS: The agreement was 53% at trial level and 31% at meta-analysis level. Including all pairs, the median disagreement was SMD=0.22 (interquartile range 0.07-0.61). The experts agreed somewhat more than the PhD students at trial level (61% v 46%), but not at meta-analysis level. Important reasons for disagreement were differences in selection of time points, scales, control groups, and type of calculations; whether to include a trial in the meta-analysis; and data extraction errors made by the observers. In 14 out of the 100 SMDs calculated at the meta-analysis level, individual observers reached different conclusions than the originally published review. CONCLUSIONS: Disagreements were common and often larger than the effect of commonly used treatments. Meta-analyses using SMDs are prone to observer variation and should be interpreted with caution. The reliability of meta-analyses might be improved by having more detailed review protocols, more than one observer, and statistical expertise.

Udgivelsesdato: 2009-null, OBJECTIVE: To study the inter-observer variation related to extraction of continuous and numerical rating scale data from trial reports for use in meta-analyses. DESIGN: Observer agreement study. DATA SOURCES: A random sample of 10 Cochrane reviews that presented a result as a standardised mean difference (SMD), the protocols for the reviews and the trial reports (n=45) were retrieved. DATA EXTRACTION: Five experienced methodologists and five PhD students independently extracted data from the trial reports for calculation of the first SMD result in each review. The observers did not have access to the reviews but to the protocols, where the relevant outcome was highlighted. The agreement was analysed at both trial and meta-analysis level, pairing the observers in all possible ways (45 pairs, yielding 2025 pairs of trials and 450 pairs of meta-analyses). Agreement was defined as SMDs that differed less than 0.1 in their point estimates or confidence intervals. RESULTS: The agreement was 53% at trial level and 31% at meta-analysis level. Including all pairs, the median disagreement was SMD=0.22 (interquartile range 0.07-0.61). The experts agreed somewhat more than the PhD students at trial level (61% v 46%), but not at meta-analysis level. Important reasons for disagreement were differences in selection of time points, scales, control groups, and type of calculations; whether to include a trial in the meta-analysis; and data extraction errors made by the observers. In 14 out of the 100 SMDs calculated at the meta-analysis level, individual observers reached different conclusions than the originally published review. CONCLUSIONS: Disagreements were common and often larger than the effect of commonly used treatments. Meta-analyses using SMDs are prone to observer variation and should be interpreted with caution. The reliability of meta-analyses might be improved by having more detailed review protocols, more than one observer, and statistical expertise.