Search

A considerable proportion of quantitative research remains unpublished once completed. Little research has documented non-dissemination and dissemination bias in qualitative research. This study aimed to generate evidence on the extent of non-dissemination in qualitative research. We followed a cohort of qualitative studies presented as conference abstracts to ascertain their subsequent publication status. We searched for subsequent full publication in MEDLINE, in the Cumulative Index to Nursing & Allied Health Literature and in Google Scholar. We matched abstracts to subsequent publications according to authors, method of data collection and phenomenon of interest. Fisher's exact test was calculated to examine associations between study characteristics and publication. Factors potentially associated with time to publication were evaluated with Cox regression analysis. For 91 of 270 included abstracts (33.70%; 95% CI 28.09%-39.68%), no full publication was identified. Factors that were found to be associated with subsequent full publication were oral presentation (OR 4.62; 95% CI 2.43-8.94) and university affiliation (OR 1.96; 95% CI 1.05-3.66). Compared to oral presentations, studies presented as posters took longer time to reach full publication (hazard ratio 0.35, 95% CI 0.21-0.58). This study shows that it was not possible to retrieve a full publication for over one-third of abstracts. Our findings suggest that where this non-dissemination is systematic, it may lead to distortions of the qualitative evidence-base for decision-making through dissemination bias. Our findings are congruent with those of other studies. Further research might investigate non-dissemination of qualitative studies in other disciplines to consolidate our findings.

Abstract Background Methodological research has found that non-published studies often have different results than those that are published, a phenomenon known as publication bias. When results are not published, or are published selectively based on the direction or the strength of the findings, healthcare professionals and consumers of healthcare cannot base their decision-making on the full body of current evidence. Methods As part of the OPEN project (http://www.open-project.eu) we will conduct a systematic review with the following objectives: 1. To determine the proportion and/or rate of non-publication of studies by systematically reviewing methodological research projects that followed up a cohort of studies that a. received research ethics committee (REC) approval, b. were registered in trial registries, or c. were presented as abstracts at conferences. 2. To assess the association of study characteristics (for example, direction and/or strength of findings) with likelihood of full publication. To identify reports of relevant methodological research projects we will conduct electronic database searches, check reference lists, and contact experts. Published and unpublished projects will be included. The inclusion criteria are as follows: a. RECs: methodological research projects that examined the subsequent proportion and/or rate of publication of studies that received approval from RECs; b. Trial registries: methodological research projects that examine the subsequent proportion and/or rate of publication of studies registered in trial registries; c. Conference abstracts: methodological research projects that examine the subsequent proportion and/or rate of full publication of studies which were initially presented at conferences as abstracts. Primary outcomes: Proportion/rate of published studies; time to full publication (mean/median; cumulative publication rate by time). Secondary outcomes: Association of study characteristics with full publication. The different questions (a, b, and c) will be investigated separately. Data synthesis will involve a combination of descriptive and statistical summaries of the included methodological research projects. Discussion Results are expected to be publicly available in mid 2013.

Using the Hartung-Knapp method and 95% prediction intervals (PIs) in random-effects meta-analyses is recommended by experts but rarely applied. Therefore, we aimed to reevaluate statistically significant meta-analyses using the Hartung-Knapp method and 95% PIs. In this methodological study, three databases were searched from January 2010 to July 2019. We included systematic reviews reporting a statistically significant meta-analysis of at least four randomized controlled trials in advanced cancer patients using either a fixed-effect or random-effects model. We investigated the impact of switching from fixed-effect to random-effects meta-analysis and of using the recommended Hartung-Knapp method in random-effects meta-analyses. Furthermore, we calculated 95% PIs for all included meta-analyses. We identified 6234 hits, of which 261 statistically significant meta-analyses were included. Our recalculations of these 261 meta-analyses produced statistically significant results in 132 of 138 fixed-effect and 114 of 123 random-effects meta-analyses. When switching to a random-effects model, 19 of 132 fixed-effect meta-analyses (14.4%) were no longer statistically significant. Using the Hartung-Knapp method in random-effects meta-analyses resulted in 34 of 114 nonsignificant meta-analyses (29.8%). In the full sample (N = 261), the "null effect" was included by the 95% PI in 195 (74.7%) and the "opposite effect" (e.g., hazard ratio 0.5, opposite effect 2) in 98 meta-analyses (37.5%). Using the Hartung-Knapp method and PIs substantially influenced the interpretation of many published, statistically significant meta-analyses. We strongly encourage researchers to check if using the Hartung-Knapp method and reporting 95% PIs is appropriate in random-effects meta-analyses.

Abstract Background The heterogeneity statistic I2, interpreted as the percentage of variability due to heterogeneity between studies rather than sampling error, depends on precision, that is, the size of the studies included. Methods Based on a real meta-analysis, we simulate artificially 'inflating' the sample size under the random effects model. For a given inflation factor M = 1, 2, 3,... and for each trial i, we create a M-inflated trial by drawing a treatment effect estimate from the random effects model, using si2 MathType@MTEF@5@5@+=feaagaart1ev2aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xH8viVGI8Gi=hEeeu0xXdbba9frFj0xb9qqpG0dXdb9aspeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaGaem4Cam3aa0baaSqaaiabdMgaPbqaaiabikdaYaaaaaa@2FBE@/M as within-trial sampling variance. Results As precision increases, while estimates of the heterogeneity variance τ2 remain unchanged on average, estimates of I2 increase rapidly to nearly 100%. A similar phenomenon is apparent in a sample of 157 meta-analyses. Conclusion When deciding whether or not to pool treatment estimates in a meta-analysis, the yard-stick should be the clinical relevance of any heterogeneity present. τ2, rather than I2, is the appropriate measure for this purpose.

Abstract Background In order to reduce systematic errors (such as language bias) and increase the precision of the summary treatment effect estimate, a comprehensive identification of randomised controlled trials (RCT), irrespective of publication language, is crucial in systematic reviews and meta-analyses. We identified trials in the German general health care literature. Methods Eight German language general health care journals were searched for randomised controlled trials and analysed with respect to the number of published RCTs each year and the size of trials. Results A total of 1618 trials were identified with a median total number of 43 patients per trial. Between 1970 and 2004 a small but constant rise in sample size from a median number of 30 to 60 patients per trial can be observed. The number of published trials was very low between 1948 and 1970, but increased between 1970 and 1986 to a maximum of 11.2 RCTs per journal and year. In the following time period a striking decline of the number of RCTs was observed. Between 1999 and 2001 only 0.8 RCTs per journal and year were published, in the next three years, the number of published trials increased to 1.7 RCTs per journal and year. Conclusion German language general health care journals no longer have a role in the dissemination of trial results. The slight rise in the number of published RCTs in the last three years can be explained by a change of publication language from German to English of three of the analysed journals.

The Peto odds ratio is a well-known effect measure in meta-analysis of binary outcomes. For pairwise comparisons, the Peto odds ratio estimator can be severely biased in the situation of unbalanced sample sizes in the two treatment groups or large treatment effects. In this publication, we evaluate Peto odds ratio estimators in the setting of multi-arm studies and in network meta-analysis using illustrative examples. We observe that Peto odds ratio estimators in a multi-arm study are inconsistent if the observed event probabilities are different or the sample sizes of treatment groups are unbalanced. The same problem emerges in network meta-analysis including only two-arm studies and translates to indirect comparisons of pairwise meta-analyses. We conclude that the Peto odds ratio should not be used as effect measure in network meta-analysis or indirect comparisons of pairwise meta-analyses.

In the era of the "reproducibility crisis" and the "P-value controversy" new ways of presentation and interpretation of the results of a meta-analysis are desirable. One suggestion that has been made for single studies almost six decades ago and taken up now and then is the P-value function. For a given outcome, this function assigns a P-value to each possible hypothetical value, given the data. Moreover, the P-value function simultaneously provides two-sided confidence intervals for all possible alpha levels. An application to meta-analysis, while suggested early, has not been widely established. We introduce the drapery plot that presents the P-value function for all individual studies and pooled estimates in a meta-analysis as curves and the prediction range for a single future study. We also present a scaled variant with the test statistic on the y-axis. Both plots visualize the full information of a pairwise meta-analysis. We see a drapery plot as a complementary figure to a forest plot. It may be even an alternative in meta-analyses with many studies where forest plots tend to become very large and complex.

Standard generic inverse variance methods for the combination of single proportions are based on transformed proportions using the logit, arcsine, and Freeman-Tukey double arcsine transformations. Generalized linear mixed models are another more elaborate approach. Irrespective of the approach, meta-analysis results are typically back-transformed to the original scale in order to ease interpretation. Whereas the back-transformation of meta-analysis results is straightforward for most transformations, this is not the case for the Freeman-Tukey double arcsine transformation, albeit possible. In this case study with five studies, we demonstrate how seriously misleading the back-transformation of the Freeman-Tukey double arcsine transformation can be. We conclude that this transformation should only be used with special caution for the meta-analysis of single proportions due to potential problems with the back-transformation. Generalized linear mixed models seem to be a promising alternative.

Quantifying the contributions, or weights, of comparisons or single studies to the estimates in a network meta‐analysis (NMA) is an active area of research. We extend this work to include the contributions of paths of evidence. We present a general framework, based on the path‐design matrix, that describes the problem of finding path contributions as a linear equation. The resulting solutions may have negative coefficients. We show that two known approaches, called shortestpath and randomwalk, are special solutions of this equation, and both meet an optimization criterion, as they minimize the sum of absolute path contributions. In general, there is an infinite set of solutions, which can be identified using the generalized inverse (Moore‐Penrose pseudoinverse). We consider two further special approaches. For large networks we find that shortestpath is superior with respect to run time and variability, compared to the other approaches, and is thus recommended in practice. The path‐weights framework also has the potential to answer more general research questions in NMA. [ABSTRACT FROM AUTHOR]

Key Points: Question: Which psychiatric disorder exhibits the strongest improvement associated with placebo treatment in randomized clinical trials (RCTs)? Findings: This systematic review and meta-analysis of 90 high-quality RCTs with 9985 participants found significant improvement under placebo treatment for all 9 disorders, but the degree of improvement varied significantly among diagnoses. Patients with major depressive disorder experienced the greatest improvement, followed by those with generalized anxiety disorder, panic disorder, attention-deficit/hyperactivity disorder, posttraumatic stress disorder, social phobia, mania, and OCD, while patients with schizophrenia benefited the least. Meaning: These findings may inform planning of RCTs, interpreting of uncontrolled studies, and advising patients for or against a specific treatment. This systematic review and meta-analysis compares outcomes in placebo groups in 90 randomized clinical trials across 9 psychiatric disorders. Importance: Placebo is the only substance systematically evaluated across common psychiatric diagnoses, but comprehensive cross-diagnostic comparisons are lacking. Objective: To compare changes in placebo groups in recent high-quality randomized clinical trials (RCTs) across a broad spectrum of psychiatric disorders in adult patients. Data Sources: MEDLINE and the Cochrane Database of Systematic Reviews were systematically searched in March 2022 for the latest systematic reviews meeting predetermined high-quality criteria for 9 major psychiatric diagnoses. Study Selection: Using these reviews, the top 10 highest-quality (ie, lowest risk of bias, according to the Cochrane Risk of Bias tool) and most recent placebo-controlled RCTs per diagnosis (totaling 90 RCTs) were selected, adhering to predetermined inclusion and exclusion criteria. Data Extraction and Synthesis: Following the Cochrane Handbook, 2 authors independently carried out the study search, selection, and data extraction. Cross-diagnosis comparisons were based on standardized pre-post effect sizes (mean change divided by its SD) for each placebo group. This study is reported following the Meta-analysis of Observational Studies in Epidemiology (MOOSE) reporting guideline. Main Outcome and Measure: The primary outcome, pooled pre-post placebo effect sizes (dav) with 95% CIs per diagnosis, was determined using random-effects meta-analyses. A Q test assessed statistical significance of differences across diagnoses. Heterogeneity and small-study effects were evaluated as appropriate. Results: A total of 90 RCTs with 9985 placebo-treated participants were included. Symptom severity improved with placebo in all diagnoses. Pooled pre-post placebo effect sizes differed across diagnoses (Q = 88.5; df = 8; P <.001), with major depressive disorder (dav = 1.40; 95% CI, 1.24-1.56) and generalized anxiety disorder (dav = 1.23; 95% CI, 1.06-1.41) exhibiting the largest dav. Panic disorder, attention-deficit/hyperactivity disorder, posttraumatic stress disorder, social phobia, and mania showed dav between 0.68 and 0.92, followed by OCD (dav = 0.65; 95% CI, 0.51-0.78) and schizophrenia (dav = 0.59; 95% CI, 0.41-0.76). Conclusion and Relevance: This systematic review and meta-analysis found that symptom improvement with placebo treatment was substantial in all conditions but varied across the 9 included diagnoses. These findings may help in assessing the necessity and ethical justification of placebo controls, in evaluating treatment effects in uncontrolled studies, and in guiding patients in treatment decisions. These findings likely encompass the true placebo effect, natural disease course, and nonspecific effects. [ABSTRACT FROM AUTHOR]

The LFK index has been promoted as an improved method to detect bias in meta‐analysis. Putatively, its performance does not depend on the number of studies in the meta‐analysis. We conducted a simulation study, comparing the LFK index test to three standard tests for funnel plot asymmetry in settings with smaller or larger group sample sizes. In general, false positive rates of the LFK index test markedly depended on the number and size of studies as well as the between‐study heterogeneity with values between 0% and almost 30%. Egger's test adhered well to the pre‐specified significance level of 5% under homogeneity, but was too liberal (smaller groups) or conservative (larger groups) under heterogeneity. The rank test was too conservative for most simulation scenarios. The Thompson–Sharp test was too conservative under homogeneity, but adhered well to the significance level in case of heterogeneity. The true positive rate of the LFK index test was only larger compared with classic tests if the false positive rate was inflated. The power of classic tests was similar or larger than the LFK index test if the false positive rate of the LFK index test was used as significance level for the classic tests. Under ideal conditions, the false positive rate of the LFK index test markedly and unpredictably depends on the number and sample size of studies as well as the extent of between‐study heterogeneity. The LFK index test in its current implementation should not be used to assess funnel plot asymmetry in meta‐analysis. [ABSTRACT FROM AUTHOR]

In systematic reviews, meta-analyses are routinely applied to summarize the results of the relevant studies for a specific research question. If one can assume that in all studies the same true effect is estimated, the application of a meta-analysis with common effect (commonly referred to as fixed-effect meta-analysis) is adequate. If between-study heterogeneity is expected to be present, the method of choice is a meta-analysis with random effects. The widely used DerSimonian and Laird method for meta-analyses with random effects has been criticized due to its unfavorable statistical properties, especially in the case of very few studies. A working group of the Cochrane Collaboration recommended the use of the Knapp-Hartung method for meta-analyses with random effects. However, as heterogeneity cannot be reliably estimated if only very few studies are available, the Knapp-Hartung method, while correctly accounting for the corresponding uncertainty, has very low power. Our aim is to summarize possible methods to perform meaningful evidence syntheses in the situation with only very few (ie, 2-4) studies. Some general recommendations are provided on which method should be used when. Our recommendations are based on the existing literature on methods for meta-analysis with very few studies and consensus of the authors. The recommendations are illustrated by 2 examples coming from dossier assessments of the Institute for Quality and Efficiency in Health Care.

Network meta-analysis has evolved into a core method for evidence synthesis in health care. In network meta-analysis, 3 or more treatments for a given medical condition are compared, based on a number of clinical studies, usually randomized controlled trials. Often, many different endpoints are investigated, related to different aspects of the patient's outcome, such as efficacy, safety, acceptability, or costs of a treatment. Different outcomes may lead to different rankings of the treatments. We use the existing theory of partially ordered sets and show how the relations between the treatments in a network meta-analysis can be illustrated by Hasse diagrams, that is, directed graphs showing the partial order relations, and by structured scatter plots and biplots.

Systematic reviewers conducting pairwise meta-analyses sometimes encounter multi-arm studies. To include these studies, and to avoid a unit-of-analysis error, often two or more arms are combined or the control arm is split. In this tutorial, we present 5 different approaches that can be used. Particularly, we present a novel approach (method 4) that to the best of our knowledge has not been presented before. We demonstrate their application on 3 selected data sets, discuss their scope of application and their advantages and limitations, and give recommendations.