Quasi-experimental study designs series—paper 6: risk of bias assessment

Rigorous and transparent critical appraisal is a core component of high quality systematic reviews. Well-conducted quasi-experiments have been empirically shown to estimate credible, unbiased treatment quantities. Conversely, when inappropriately designed or executed, these estimates are likely to be biased. This paper draws on recent advances in risk of bias assessment. It presents an approach to evaluating the internal validity of credible quasi-experiments. These are non-randomised studies using design-based approaches to control for unobservable sources of confounding such as difference studies, instrumental variables, interrupted time series, natural experiments and regression discontinuity designs. Our review suggests that existing risk of bias tools provide, to different degrees, incomplete transparent criteria to assess the validity of credible quasi-experiments. We argue that a tool is needed to assess risk of bias consistently across credible quasi-experiments. Drawing on existing tools, in particular Cochrane’s new tool for non-randomized studies of interventions (Sterne et al., 2014), we discuss domains of bias and suggest directions for evaluation questions.