Search

Objectives: We aimed to provide a systematic overview of freely available tools which may help plan or monitor costs for randomized clinical trials (RCTs). Study Design and Setting: We systematically searched MEDLINE, EMBASE, and EconLit and conducted internet searches via Google (last search, October 2018). We included all freely available tools and determined their specific purpose, which parts of clinical trial projects and which types of costs they covered, and if they were user-tested or validated in any form. Results: We identified 25 available tools. Most tools were downloadable on websites from institutions related to clinical research. Seven tools were developed to plan the budget for an entire RCT, 17 tools for calculating budgets of an individual trial center, and one tool for monitoring costs of ongoing RCTs. Eighteen tools considered fixed, variable, and indirect costs. Only two tools were clearly user-tested or validated. Conclusion: Several freely available tools aim to support investigators in planning costs of an entire trial or in planning the budget for a clinical trial site. How valid and useful they are remains to be shown for most of them. Future tools should be openly shared, user-tested, and validated.

BACKGROUND AND OBJECTIVES: Assessing changes in coverage, recall, review, conclusions and references not found when searching fewer databases. METHODS: In randomly selected 60 Cochrane reviews, we checked included study publications' coverage (indexation) and recall (findability) using different search approaches with MEDLINE, Embase, and CENTRAL and related them to authors' conclusions and certainty. We assessed characteristics of unfound references. RESULTS: Overall 1989/2080 included references, were indexed in ≥1 database (coverage = 96%). In reviews where using one of our search approaches would not change conclusions and certainty (n = 44-54), median coverage and recall were highest (range 87.9%-100.0% and 78.2%-93.3%, respectively). Here, searching ≥2 databases reached >95% coverage and ≥87.9% recall. In reviews with unchanged conclusions but less certainty (n = 2-8): 63.3%-79.3% coverage and 45.0%-75.0% recall. In reviews with opposite conclusions (n = 1-3): 63.3%-96.6% and 52.1%-78.7%. In reviews where a conclusion was no longer possible (n = 3-7): 60.6%-86.0% and 20.0%-53.8%. The 265 references that were indexed but unfound were more often abstractless (30% vs. 11%) and older (28% vs. 17% published before 1991) than found references. CONCLUSION: Searching ≥2 databases improves coverage and recall and decreases the risk of missing eligible studies. If researchers suspect that relevant articles are difficult to find, supplementary search methods should be used.

Objectives Off-label drug use is highly prevalent but controversial and often discouraged assuming generally inferior medical effects associated with off-label use. Study Design and Setting We searched PubMed, MEDLINE, PubMed Health, and the Cochrane Library up to May 2015 for systematic reviews including meta-analyses of randomized clinical trials (RCTs) comparing off-label and approved drugs head-to-head in any population and on any medical outcome. We combined the comparative effects in meta-analyses providing summary odds ratios (sOR) for each treatment comparison and outcome, and then calculated an overall summary of the sOR across all comparisons (ssOR). Results We included 25 treatment comparisons with 153 RCTs and 24,592 patients. In six of 25 comparisons (24%), off-label drugs were significantly superior (five of 25) or inferior (one of 25) to approved treatments. There was substantial statistical heterogeneity across comparisons (I2 = 43%). Overall, off-label drugs were more favorable than approved treatments (ssOR 0.72; 95% CI = 0.54–0.95). Analyses of patient-relevant outcomes were similar (statistical significant differences in 24% (six of 25); ssOR 0.74; 95% CI = 0.56–0.98; I2 = 60%). Analyses of primary outcomes of the systematic reviews (n = 22 comparisons) indicated less heterogeneity and no statistically significant difference overall (ssOR 0.85; 95% CI = 0.67–1.06; I2 = 0%). Conclusion Approval status does not reliably indicate which drugs are more favorable in situations with clinical trial evidence comparing off-label with approved use. Drug effectiveness assessments without considering off-label use may provide incomplete information. To ensure that patients receive the best available care, funding, policy, reimbursement, and treatment decisions should be evidence based considering the entire spectrum of available therapeutic choices.

Objectives To evaluate how estimated treatment effects agree between nonrandomized studies using causal modeling with marginal structural models (MSM-studies) and randomized trials (RCTs). Study Design Meta-epidemiological study. Setting MSM-studies providing effect estimates on any healthcare outcome of any treatment were eligible. We systematically sought RCTs on the same clinical question and compared the direction of treatment effects, effect sizes, and confidence intervals. Results The main analysis included 19 MSM-studies (1,039,570 patients) and 141 RCTs (120,669 patients). MSM-studies indicated effect estimates in the opposite direction from RCTs for eight clinical questions (42%), and their 95% CI (confidence interval) did not include the RCT estimate in nine clinical questions (47%). The effect estimates deviated 1.58-fold between the study designs (median absolute deviation OR [odds ratio] 1.58; IQR [interquartile range] 1.37 to 2.16). Overall, we found no systematic disagreement regarding benefit or harm but confidence intervals were wide (summary ratio of odds ratios [sROR] 1.04; 95% CI 0.88 to 1.23). The subset of MSM-studies focusing on healthcare decision-making tended to overestimate experimental treatment benefits (sROR 1.44; 95% CI 0.99 to 2.09). Conclusion Nonrandomized studies using causal modeling with MSM may give different answers than RCTs. Caution is still required when nonrandomized “real world” evidence is used for healthcare decisions.

Objectives To determine how marginal structural models (MSMs), which are increasingly used to estimate causal effects, are used in randomized clinical trials (RCTs) and compare their results with those from intention-to-treat (ITT) or other analyses. Study Design and Setting We searched PubMed, Scopus, citations of key references, and Clinicaltrials.gov. Eligible RCTs reported clinical effects based on MSMs and at least one other analysis. Results We included 12 RCTs reporting 138 analyses for 24 clinical questions. In 19/24 (79%), MSM-based and other effect estimates were all in the same direction, 22/22 had overlapping 95% confidence intervals (CIs), and in 19/22 (86%), the MSM effect estimate lay within all 95% CIs of all other effects (in two cases no CIs were reported). For the same clinical question, the largest effect estimate from any analysis was 1.19-fold (median; interquartile range 1.13-1.34) larger than the smallest. All MSM and ITT effect estimates were in the same direction and had overlapping 95% CIs. In 71% (12/17), they also agreed on the presence of statistical significance. MSM-based effect estimates deviated more from the null than those based on ITT (P = 0.18). The effect estimates of both approaches differed 1.12-fold (median; interquartile range 1.02-1.22). Conclusions MSMs provided largely similar effect estimates as other available analyses. Nevertheless, some of the differences in effect estimates or statistical significance may become important in clinical decision-making, and the multiple estimates require utmost attention of possible selective reporting bias.