External factors may influence Cochrane reviewers when classifying the risk of bias of original reports

Objective: The objective of the study was to explore contextual factors associated with high or low risk-of-bias judgment in case of incomplete or unclear information in study reports. Study Design and Setting: Research-on-research study, using matched case–control design, with a sample of 304 randomized controlled trials (RCTs) included in two Cochrane reviews for which there was disagreement on the risk-of-bias judgment related to incomplete or unclear information in the study report. A case was defined as an RCT judged at high or low risk of bias; a control was the same RCT judged at unclear risk. We used a conditional logistic regression model for analysis. Results: Review authors being also authors of the RCT were more likely to assess an item at low risk of bias than unclear (OR: 11.71; 95% CI: 1.39–98.76). Earlier trials in a review were more often assigned a low risk (OR: 0.37; [0.15–0.96]). Review groups and authors that had completed a lower number of reviews slightly more often assigned a low risk, whereas others reported “unclear” (OR: 0.97, [95% CI: 0.95–0.99] for groups) and 0.97 (95% CI: 0.95–0.998) for authors). Conclusions: Risk-of-bias assessment of RCTs in case of incomplete or unclear information may be affected by contextual factors.