Search

Many real-word evidence (RWE) studies that utilize existing healthcare data to evaluate treatment effects incur substantial but avoidable bias from methodologically flawed study design; however, the extent of preventable methodological pitfalls in current RWE is unknown. To characterize the prevalence of avoidable methodological pitfalls with potential for bias in published claims-based studies of medication safety or effectiveness, we conducted an English-language search of PubMed for articles published from January 1, 2010 to May 20, 2019 and randomly selected 75 studies (10 case-control and 65 cohort studies) that evaluated safety or effectiveness of cardiovascular, diabetes, or osteoporosis medications using US health insurance claims. General and methodological study characteristics were extracted independently by two reviewers, and potential for bias was assessed across nine bias domains. Nearly all studies (95%) had at least one avoidable methodological issue known to incur bias, and 81% had potentially at least one of the four issues considered major due to their potential to undermine study validity: time-related bias (57%), potential for depletion of outcome-susceptible individuals (44%), inappropriate adjustment for postbaseline variables (41%), or potential for reverse causation (39%). The median number of major issues per study was 2 (interquartile range (IQR), 1-3) and was lower in cohort studies with a new-user, active-comparator design (median 1, IQR 0-1) than in cohort studies of prevalent users with a nonuser comparator (median 3, IQR 3-4). Recognizing and avoiding known methodological study design pitfalls could substantially improve the utility of RWE and confidence in its validity.

Actionable real-world evidence (RWE) requires accurate estimation of causal treatment effects. Calibration of RWE against randomized controlled trials (RCTs) is sometimes done to demonstrate that RWE can support the same causal conclusion as RCTs. Disagreements can occur when studies in each pair asked different questions in different populations or due to the presence of residual bias. Distinguishing among reasons for differences will impact the level of confidence in RWE. Several projects, such as the RCT DUPLICATE initiative funded by US FDA, NIH, and others, have launched in the last few years with the aim of assessing whether nonrandomized database studies can in some circumstances produce conclusions on the effectiveness of medications that are similar to those provided by RCTs.1,2 While comparison of randomized and nonrandomized findings is not new, heightened current interest is in part spurred by new initiatives at several regulatory agencies focused on assessing the role real-world evidence (RWE) can play in regulatory decision-making.3,4 FDA defines RWE as evidence on the benefits and risks of medications derived from routinely collected healthcare data, although other data sources such as patient registries may also be used. A key challenge for any project attempting to calibrate RWE findings against RCT findings is that differences between treatment effect estimates from the two study types can be driven by bias due to lack of randomization in RWE and/or by other differences in the design, such as inclusion/exclusion criteria, outcome measurement, or motivations for patients to adhere to study medications. Even if RWE studies are designed to match the corresponding RCT as closely as possible, emulation of all study components is typically impossible. Emulating target trials In clinical epidemiology, it has been recommended for several decades to contemplate how a randomized trial would be designed to answer a specific question before designing the nonrandomized counterpart to study the same question. Hernan and Robins call the former a hypothetical “target trial,” which would then be emulated by a nonrandomized study.5 This process has proven very useful in clarifying design choices for nonrandomized research on medications. It is also a highly flexible process, as specification of the target trial is often iterative. Realities of data collection, patient access, and other practical considerations impose constraints to the nonrandomized emulation; the hypothetical target trial can thus be adjusted so that the design that is feasible in a given healthcare database can accord with the design of the target trial. In calibrating RWE studies against existing RCTs, the process of emulating a target RCT in nonrandomized data is similar, except that in this case the target trial is already underway or completed. Thus, adjusting the design of the target RCT to improve the feasibility of the design in existing data is not possible, making exact emulation more difficult. Instead, investigators must adapt the design elements that they can from the trial, given the constraints of the database, and note the trial specifications that cannot be completely emulated. This process will highlight unavoidable emulation differences between a completed or in-progress RCT and the RWE replication. When assembling a series of RWE replications of RCTs, as in the RCT DUPLICATE initiative, there will be some trials where RWE specifications can closely emulate the RCT and others that cannot be emulated as closely. Examples of the latter are run-in periods during which non-adherent or drug intolerant patients are excluded before randomization, or, to homogenize patients, run-in periods that place all patients on a common medication before randomization. Therefore, enumerating – and quantifying to the extent possible – such emulation differences can provide insight into whether and to what extent differences in treatment effect estimates between RWE and corresponding RCTs are due to bias related to a lack of randomization versus other differences in design between the two study types. Specifically, attempting to correlate measures of emulation difference, such as those suggested in Table 1, with the magnitude of differences in treatment effect estimates may provide understanding of which emulation differences are most important in contributing to the “efficacy-effectiveness gap” between RWE and RCT findings, as long as bias due to confounding is not also correlated with the emulation difference of interest. Once better understood, such metrics could possibly be transformed into a simple three-point emulation scale. Table 1. Challenges in calibrating RWE against RCTs and measures of differences between the study types.

The emergence and global spread of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has resulted in an urgent need for evidence on medical interventions and outcomes of the resulting disease, coronavirus disease 2019 (COVID-19). Although many randomized controlled trials (RCTs) evaluating treatments and vaccines for COVID-19 are already in progress, the number of clinical questions of interest greatly outpaces the available resources to conduct RCTs. Therefore, there is growing interest in whether nonrandomized real-world evidence (RWE) can be used to supplement RCT evidence and aid in clinical decision making, but concerns about nonrandomized RWE have been highlighted by a proliferation of RWE studies on medications and COVID-19 outcomes with widely varying conclusions. The objective of this paper is to review some clinical questions of interest, potential data types, challenges, and merits of RWE in COVID-19, resulting in recommendations for nonrandomized RWE designs and analyses based on established RWE principles.

Regulators consider randomized controlled trials (RCTs) as the gold standard for evaluating the safety and effectiveness of medications, but their costs, duration, and limited generalizability have caused some to look for alternatives. Real world evidence based on data collected outside of RCTs, such as registries and longitudinal healthcare databases, can sometimes substitute for RCTs, but concerns about validity have limited their impact. Greater reliance on such real world data (RWD) in regulatory decision making requires understanding why some studies fail while others succeed in producing results similar to RCTs. Key questions when considering whether RWD analyses can substitute for RCTs for regulatory decision making are WHEN one can study drug effects without randomization and HOW to implement a valid RWD analysis if one has decided to pursue that option. The WHEN is primarily driven by externalities not controlled by investigators, whereas the HOW is focused on avoiding known mistakes in RWD analyses.

In 2008, the US Food and Drug Administration (FDA) issued guidance on the need for cardiovascular outcome trials to assess the safety of new diabetes medications. Using two large commercial databases, we evaluated the effect of the FDA's cardiovascular safety guidance on drug development for type 2 diabetes as well as a comparison group of drugs intended to treat other alimentary and metabolic conditions. The FDA's guidance was associated with a 31% differential decrease in the rate of diabetes drugs entering phase II trials, but the remaining drugs were significantly more likely to target novel biological pathways (72% of drugs had novel mechanisms after the guidance vs. 49% before the guidance). No differential changes were observed for phase I and phase III trials. There was no measurable improvement during the study period in glycemic efficacy among investigational products entering phase III trials. This research highlights how regulatory actions can impact pharmaceutical innovation.