Examining How Meta-Analytic Methods Perform in the Presence of Bias: A Simulation Study

Standard meta-analysis methods are vulnerable to bias from incomplete reporting of results (both publication and outcome reporting bias) and poor study quality. Several alternative methods have been proposed as being less vulnerable to such biases. To evaluate these claims independently we simulated study results under a broad range of conditions first with no bias, then introducing simulated publication bias, outcome reporting bias, and bias from poor study quality. We then implemented common and the proposed bias robust meta-analysis methods and compared the mean bias and mean squared error (MSE) for four estimates of effect and the coverage probability of seven confidence intervals. We found that no methods perform well in the presence of any substantial bias. A regression based extension to Egger's test gave an estimate of effect with lower mean bias than standard methods in the presence of publication bias or poor study quality, but had a substantially worse MSE except in very specific conditions. Coverage of all 95% confidence intervals was very poor with increasing numbers of studies in biased conditions, often falling below 50%. The Knapp-Hartung interval performed closest to nominal coverage with fewer than 10 studies in most conditions, and the Henmi-Copas interval generally performed best with more than 10 studies. There was no evidence that a multiplicative term for heterogeneity improved coverage. Multiple forms of bias remain problematic for all meta-analysis methods, with very poor performance under conceivable conditions.