Search

Introduction: Observational studies estimating severe outcomes for paracetamol versus ibuprofen use have acknowledged the specific challenge of channeling bias. A previous study relying on negative controls suggested that using large-scale propensity score (LSPS) matching may mitigate bias better than models using limited lists of covariates., Objective: The aim was to assess whether using LSPS matching would enable the evaluation of paracetamol, compared to ibuprofen, and increased risk of myocardial infarction, stroke, gastrointestinal (GI) bleeding, or acute renal failure., Study Design and Setting: In a new-user cohort study, we used two propensity score model strategies for confounder controls. One replicated the approach of controlling for a hand-picked list. The second used LSPSs based on all available covariates for matching. Positive and negative controls assessed residual confounding and calibrated confidence intervals. The data source was the Clinical Practices Research Datalink (CPRD)., Results: A substantial proportion of negative controls were statistically significant after propensity score matching on the publication covariates, indicating considerable systematic error. LSPS adjustment was less biased, but residual error remained. The calibrated estimates resulted in very wide confidence intervals, indicating large uncertainty in effect estimates once residual error was incorporated., Conclusions: For paracetamol versus ibuprofen, when using LSPS methods in the CPRD, it is only possible to distinguish true effects if those effects are large (hazard ratio > 2). Due to their smaller hazard ratios, the outcomes under study cannot be differentiated from null effects (represented by negative controls) even if there were a true effect. Based on these data, we conclude that we are unable to determine whether paracetamol is associated with an increased risk of myocardial infarction, stroke, GI bleeding, and acute renal failure compared to ibuprofen, due to residual confounding.

Introduction: Drug safety researchers seek to know the degree of certainty with which a particular drug is associated with an adverse drug reaction. There are different sources of information used in pharmacovigilance to identify, evaluate, and disseminate medical product safety evidence including spontaneous reports, published peer-reviewed literature, and product labels. Automated data processing and classification using these evidence sources can greatly reduce the manual curation currently required to develop reference sets of positive and negative controls (i.e. drugs that cause adverse drug events and those that do not) to be used in drug safety research., Methods: In this paper we explore a method for automatically aggregating disparate sources of information together into a single repository, developing a predictive model to classify drug-adverse event relationships, and applying those predictions to a real world problem of identifying negative controls for statistical method calibration., Results: Our results showed high predictive accuracy for the models combining all available evidence, with an area under the receiver-operator curve of ⩾0.92 when tested on three manually generated lists of drugs and conditions that are known to either have or not have an association with an adverse drug event., Conclusions: Results from a pilot implementation of the method suggests that it is feasible to develop a scalable alternative to the time-and-resource-intensive, manual curation exercise previously applied to develop reference sets of positive and negative controls to be used in drug safety research., (Copyright © 2016 Elsevier Inc. All rights reserved.)

Most approaches used in postmarketing drug safety monitoring, including spontaneous reporting and statistical risk identification using electronic health care records, are primarily suited to pick up only acute adverse drug effects. With the availability of increasingly larger electronic health record and administrative claims databases comes the opportunity to monitor for potential adverse effects that occur only after prolonged exposure to a drug, but analysis methods are lacking. We propose an adaptation of the self-controlled case series design that uses the notion of accumulated exposure to capture long-term effects of drugs and evaluate extensions to correct for age and recurrent events. Several variations of the approach are tested on simulated data and two large insurance claims databases. To evaluate performance a set of positive and negative control drug-event pairs was created by medical experts based on drug product labels and review of the literature. Performance on the real data was measured using the area under the receiver operator characteristics curve. The best performing method achieved an area under the receiver operator characteristics curve of 0.86 in the largest database using a spline model, adjustment for age, and ignoring recurrent events, but it appears this performance can only be achieved with very large data sets., (© The Author(s) 2014.)

Identification of adverse drug reactions (ADRs) during the post-marketing phase is one of the most important goals of drug safety surveillance. Spontaneous reporting systems (SRS) data, which are the mainstay of traditional drug safety surveillance, are used for hypothesis generation and to validate the newer approaches. The publicly available US Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS) data requires substantial curation before they can be used appropriately, and applying different strategies for data cleaning and normalization can have material impact on analysis results. We provide a curated and standardized version of FAERS removing duplicate case records, applying standardized vocabularies with drug names mapped to RxNorm concepts and outcomes mapped to SNOMED-CT concepts, and pre-computed summary statistics about drug-outcome relationships for general consumption. This publicly available resource, along with the source code, will accelerate drug safety research by reducing the amount of time spent performing data management on the source FAERS reports, improving the quality of the underlying data, and enabling standardized analyses using common vocabularies.

Introduction: Observational healthcare data contain information useful for hastening detection of adverse drug reactions (ADRs) that may be missed by using data in spontaneous reporting systems (SRSs) alone. There are only several papers describing methods that integrate evidence from healthcare databases and SRSs. We propose a methodology that combines ADR signals from these two sources., Objectives: The aim of this study was to investigate whether the proposed method would result in more accurate ADR detection than methods using SRSs or healthcare data alone., Research Design: We applied the method to four clinically serious ADRs, and evaluated it using three experiments that involve combining an SRS with a single facility small-scale electronic health record (EHR), a larger scale network-based EHR, and a much larger scale healthcare claims database. The evaluation used a reference standard comprising 165 positive and 234 negative drug-ADR pairs., Measures: Area under the receiver operator characteristics curve (AUC) was computed to measure performance., Results: There was no improvement in the AUC when the SRS and small-scale HER were combined. The AUC of the combined SRS and large-scale EHR was 0.82 whereas it was 0.76 for each of the individual systems. Similarly, the AUC of the combined SRS and claims system was 0.82 whereas it was 0.76 and 0.78, respectively, for the individual systems., Conclusions: The proposed method resulted in a significant improvement in the accuracy of ADR detection when the resources used for combining had sufficient amounts of data, demonstrating that the method could integrate evidence from multiple sources and serve as a tool in actual pharmacovigilance practice.

Characterization of relationships between time-varying drug exposures and adverse events (AEs) related to health outcomes represents the primary objective in postmarketing drug safety surveillance. Such surveillance increasingly utilizes large-scale longitudinal observational databases (LODs), containing time-stamped patient-level medical information including periods of drug exposure and dates of diagnoses for millions of patients. Statistical methods for LODs must confront computational challenges related to the scale of the data, and must also address confounding and other biases that can undermine efforts to estimate effect sizes. Methods that compare on-drug with off-drug periods within patient offer specific advantages over between patient analysis on both counts. To accomplish these aims, we extend the self-controlled case series (SCCS) for LODs. SCCS implicitly controls for fixed multiplicative baseline covariates since each individual acts as their own control. In addition, only exposed cases are required for the analysis, which is computationally advantageous. The standard SCCS approach is usually used to assess single drugs and therefore estimates marginal associations between individual drugs and particular AEs. Such analyses ignore confounding drugs and interactions and have the potential to give misleading results. In order to avoid these difficulties, we propose a regularized multiple SCCS approach that incorporates potentially thousands or more of time-varying confounders such as other drugs. The approach successfully handles the high dimensionality and can provide a sparse solution via an L₁ regularizer. We present details of the model and the associated optimization procedure, as well as results of empirical investigations., (© 2013, The International Biometric Society.)

Background: Observational healthcare data offer the potential to enable identification of risks of medical products, but appropriate methodology has not yet been defined. The new user cohort method, which compares the post-exposure rate among the target drug to a referent comparator group, is the prevailing approach for many pharmacoepidemiology evaluations and has been proposed as a promising approach for risk identification but its performance in this context has not been fully assessed., Objectives: To evaluate the performance of the new user cohort method as a tool for risk identification in observational healthcare data., Research Design: The method was applied to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record) and in 6 simulated datasets with no effect and injected relative risks of 1.25, 1.5, 2, 4, and 10, respectively., Measures: Method performance was evaluated through Area Under ROC Curve (AUC), bias, and coverage probability., Results: The new user cohort method achieved modest predictive accuracy across the outcomes and databases under study, with the top-performing analysis near AUC >0.70 in most scenarios. The performance of the method was particularly sensitive to the choice of comparator population. For almost all drug-outcome pairs there was a large difference, either positive or negative, between the true effect size and the estimate produced by the method, although this error was near zero on average. Simulation studies showed that in the majority of cases, the true effect estimate was not within the 95 % confidence interval produced by the method., Conclusion: The new user cohort method can contribute useful information toward a risk identification system, but should not be considered definitive evidence given the degree of error observed within the effect estimates. Careful consideration of the comparator selection and appropriate calibration of the effect estimates is required in order to properly interpret study findings.

Background: Methodological research to evaluate the performance of methods requires a benchmark to serve as a referent comparison. In drug safety, the performance of analyses of spontaneous adverse event reporting databases and observational healthcare data, such as administrative claims and electronic health records, has been limited by the lack of such standards., Objectives: To establish a reference set of test cases that contain both positive and negative controls, which can serve the basis for methodological research in evaluating methods performance in identifying drug safety issues., Research Design: Systematic literature review and natural language processing of structured product labeling was performed to identify evidence to support the classification of drugs as either positive controls or negative controls for four outcomes: acute liver injury, acute kidney injury, acute myocardial infarction, and upper gastrointestinal bleeding., Results: Three-hundred and ninety-nine test cases comprised of 165 positive controls and 234 negative controls were identified across the four outcomes. The majority of positive controls for acute kidney injury and upper gastrointestinal bleeding were supported by randomized clinical trial evidence, while the majority of positive controls for acute liver injury and acute myocardial infarction were only supported based on published case reports. Literature estimates for the positive controls shows substantial variability that limits the ability to establish a reference set with known effect sizes., Conclusions: A reference set of test cases can be established to facilitate methodological research in drug safety. Creating a sufficient sample of drug-outcome pairs with binary classification of having no effect (negative controls) or having an increased effect (positive controls) is possible and can enable estimation of predictive accuracy through discrimination. Since the magnitude of the positive effects cannot be reliably obtained and the quality of evidence may vary across outcomes, assumptions are required to use the test cases in real data for purposes of measuring bias, mean squared error, or coverage probability.

Background: Researchers using observational data to understand drug effects must make a number of analytic design choices that suit the characteristics of the data and the subject of the study. Review of the published literature suggests that there is a lack of consistency even when addressing the same research question in the same database., Objective: To characterize the degree of similarity or difference in the method and analysis choices made by observational database research experts when presented with research study scenarios., Research Design: On-line survey using research scenarios on drug-effect studies to capture method selection and analysis choices that follow a dependency branching based on response to key questions., Subjects: Voluntary participants experienced in epidemiological study design solicited for participation through registration on the Observational Medical Outcomes Partnership website, membership in particular professional organizations, or links in relevant newsletters., Measures: Description (proportion) of respondents selecting particular methods and making specific analysis choices based on individual drug-outcome scenario pairs. The number of questions/decisions differed based on stem questions of study design, time-at-risk, outcome definition, and comparator., Results: There is little consistency across scenarios, by drug or by outcome of interest, in the decisions made for design and analyses in scenarios using large healthcare databases. The most consistent choice was the cohort study design but variability in the other critical decisions was common., Conclusions: There is great variation among epidemiologists in the design and analytical choices that they make when implementing analyses in observational healthcare databases. These findings confirm that it will be important to generate empiric evidence to inform these decisions and to promote a better understanding of the impact of standardization on research implementation.

Objective: The objective of this study is to present a data quality assurance program for disparate data sources loaded into a Common Data Model, highlight data quality issues identified and resolutions implemented., Background: The Observational Medical Outcomes Partnership is conducting methodological research to develop a system to monitor drug safety. Standard processes and tools are needed to ensure continuous data quality across a network of disparate databases, and to ensure that procedures used to extract-transform-load (ETL) processes maintain data integrity. Currently, there is no consensus or standard approach to evaluate the quality of the source data, or ETL procedures., Methods: We propose a framework for a comprehensive process to ensure data quality throughout the steps used to process and analyze the data. The approach used to manage data anomalies includes: (1) characterization of data sources; (2) detection of data anomalies; (3) determining the cause of data anomalies; and (4) remediation., Findings: Data anomalies included incomplete raw dataset: no race or year of birth recorded. Implausible data: year of birth exceeding current year, observation period end date precedes start date, suspicious data frequencies and proportions outside normal range. Examples of errors found in the ETL process were zip codes incorrectly loaded, drug quantities rounded, drug exposure length incorrectly calculated, and condition length incorrectly programmed., Conclusions: Complete and reliable observational data are difficult to obtain, data quality assurance processes need to be continuous as data is regularly updated; consequently, processes to assess data quality should be ongoing and transparent.

Background: A systematic risk identification system has the potential to test marketed drugs for important Health Outcomes of Interest or HOI. For each HOI, multiple definitions are used in the literature, and some of them are validated for certain databases. However, little is known about the effect of different definitions on the ability of methods to estimate their association with medical products., Objectives: Alternative definitions of HOI were studied for their effect on the performance of analytical methods in observational outcome studies., Methods: A set of alternative definitions for three HOI were defined based on literature review and clinical diagnosis guidelines: acute kidney injury, acute liver injury and acute myocardial infarction. The definitions varied by the choice of diagnostic codes and the inclusion of procedure codes and lab values. They were then used to empirically study an array of analytical methods with various analytical choices in four observational healthcare databases. The methods were executed against predefined drug-HOI pairs to generate an effect estimate and standard error for each pair. These test cases included positive controls (active ingredients with evidence to suspect a positive association with the outcome) and negative controls (active ingredients with no evidence to expect an effect on the outcome). Three different performance metrics where used: (i) Area Under the Receiver Operator Characteristics (ROC) curve (AUC) as a measure of a method's ability to distinguish between positive and negative test cases, (ii) Measure of bias by estimation of distribution of observed effect estimates for the negative test pairs where the true effect can be assumed to be one (no relative risk), and (iii) Minimal Detectable Relative Risk (MDRR) as a measure of whether there is sufficient power to generate effect estimates., Results: In the three outcomes studied, different definitions of outcomes show comparable ability to differentiate true from false control cases (AUC) and a similar bias estimation. However, broader definitions generating larger outcome cohorts allowed more drugs to be studied with sufficient statistical power., Conclusions: Broader definitions are preferred since they allow studying drugs with lower prevalence than the more precise or narrow definitions while showing comparable performance characteristics in differentiation of signal vs. no signal as well as effect size estimation.

Background: Observational healthcare data offer the potential to enable identification of risks of medical products, and the medical literature is replete with analyses that aim to accomplish this objective. A number of established analytic methods dominate the literature but their operating characteristics in real-world settings remain unknown., Objectives: To compare the performance of seven methods (new user cohort, case control, self-controlled case series, self-controlled cohort, disproportionality analysis, temporal pattern discovery, and longitudinal gamma poisson shrinker) as tools for risk identification in observational healthcare data., Research Design: The experiment applied each method to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record)., Measures: Method performance was evaluated through Area Under the receiver operator characteristics Curve (AUC), bias, mean square error, and confidence interval coverage probability., Results: Multiple methods offer strong predictive accuracy, with AUC > 0.70 achievable for all outcomes and databases with more than one analytical approach. Self-controlled methods (self-controlled case series, temporal pattern discovery, self-controlled cohort) had higher predictive accuracy than cohort and case-control methods across all databases and outcomes. Methods differed in the expected value and variance of the error distribution. All methods had lower coverage probability than the expected nominal properties., Conclusions: Observational healthcare data can inform risk identification of medical product effects on acute liver injury, acute myocardial infarction, acute renal failure and gastrointestinal bleeding. However, effect estimates from all methods require calibration to address inconsistency in method operating characteristics. Further empirical evaluation is required to gauge the generalizability of these findings to other databases and outcomes.

Objective: To evaluate the performance of a disproportionality design, commonly used for analysis of spontaneous reports data such as the FDA Adverse Event Reporting System database, as a potential analytical method for an adverse drug reaction risk identification system using healthcare data., Research Design: We tested the disproportionality design in 5 real observational healthcare databases and 6 simulated datasets, retrospectively studying the predictive accuracy of the method when applied to a collection of 165 positive controls and 234 negative controls across 4 outcomes: acute liver injury, acute myocardial infarction, acute kidney injury, and upper gastrointestinal bleeding., Measures: We estimate how well the method can be expected to identify true effects and discriminate from false findings and explore the statistical properties of the estimates the design generates. The primary measure was the area under the curve (AUC) of the receiver operating characteristic (ROC) curve., Results: For each combination of 4 outcomes and 5 databases, 48 versions of disproportionality analysis (DPA) were carried out and the AUC computed. The majority of the AUC values were in the range of 0.35 < AUC < 0.6, which is considered to be poor predictive accuracy, since the value AUC = 0.5 would be expected from mere random assignment. Several DPA versions achieved AUC of about 0.7 for the outcome Acute Renal Failure within the GE database. The overall highest DPA version across all 20 outcome-database combinations was the Bayesian Information Component method with no stratification by age and gender, using first occurrence of outcome and with assumed time-at-risk equal to duration of exposure + 30 d, but none were uniformly optimal. The relative risk estimates for the negative control drug-event combinations were very often biased either upward or downward by a factor of 2 or more. Coverage probabilities of confidence intervals from all methods were far below nominal., Conclusions: The disproportionality methods that we evaluated did not discriminate true positives from true negatives using healthcare data as they seem to do using spontaneous report data.

Background: The availability of large-scale observational healthcare data allows for the active monitoring of safety of drugs, but research is needed to determine which statistical methods are best suited for this task. Recently, the Longitudinal Gamma Poisson Shrinker (LGPS) and Longitudinal Evaluation of Observational Profiles of Adverse events Related to Drugs (LEOPARD) methods were developed specifically for this task. LGPS applies Bayesian shrinkage to an estimated incidence rate ratio, and LEOPARD aims to detect and discard associations due to protopathic bias. The operating characteristics of these methods still need to be determined., Objective: Establish the operating characteristics of LGPS and LEOPARD for large scale observational analysis in drug safety., Research Design: We empirically evaluated LGPS and LEOPARD in five real observational healthcare databases and six simulated datasets. We retrospectively studied the predictive accuracy of the methods when applied to a collection of 165 positive control and 234 negative control drug-outcome pairs across four outcomes: acute liver injury, acute myocardial infarction, acute kidney injury, and upper gastrointestinal bleeding., Results: In contrast to earlier findings, we found that LGPS and LEOPARD provide weak discrimination between positive and negative controls, although the use of LEOPARD does lead to higher performance in this respect. Furthermore, the methods produce biased estimates and confidence intervals that have poor coverage properties., Conclusions: For the four outcomes we examined, LGPS and LEOPARD may not be the designs of choice for risk identification.

Background: Observational healthcare data offer the potential to enable identification of risks of medical products, but appropriate methodology has not yet been defined. The self-controlled cohort method, which compares the post-exposure outcome rate with the pre-exposure rate among an exposed cohort, has been proposed as a potential approach for risk identification but its performance has not been fully assessed., Objectives: To evaluate the performance of the self-controlled cohort method as a tool for risk identification in observational healthcare data., Research Design: The method was applied to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record) and in 6 simulated datasets with no effect and injected relative risks of 1.25, 1.5, 2, 4, and 10, respectively., Measures: Method performance was evaluated through area under ROC curve (AUC), bias, and coverage probability., Results: The self-controlled cohort design achieved strong predictive accuracy across the outcomes and databases under study, with the top-performing settings exceeding AUC >0.76 in all scenarios. However, the estimates generated were observed to be highly biased with low coverage probability., Conclusions: If the objective for a risk identification system is one of discrimination, the self-controlled cohort method shows promise as a potential tool for risk identification. However, if a system is intended to generate effect estimates to quantify the magnitude of potential risks, the self-controlled cohort method may not be suitable, and requires substantial calibration to be properly interpreted under nominal properties.

Background: There has been only limited evaluation of statistical methods for identifying safety risks of drug exposure in observational healthcare data. Simulations can support empirical evaluation, but have not been shown to adequately model the real-world phenomena that challenge observational analyses., Objectives: To design and evaluate a probabilistic framework (OSIM2) for generating simulated observational healthcare data, and to use this data for evaluating the performance of methods in identifying associations between drug exposure and health outcomes of interest., Research Design: Seven observational designs, including case-control, cohort, self-controlled case series, and self-controlled cohort design were applied to 399 drug-outcome scenarios in 6 simulated datasets with no effect and injected relative risks of 1.25, 1.5, 2, 4, and 10, respectively., Subjects: Longitudinal data for 10 million simulated patients were generated using a model derived from an administrative claims database, with associated demographics, periods of drug exposure derived from pharmacy dispensings, and medical conditions derived from diagnoses on medical claims., Measures: Simulation validation was performed through descriptive comparison with real source data. Method performance was evaluated using Area Under ROC Curve (AUC), bias, and mean squared error., Results: OSIM2 replicates prevalence and types of confounding observed in real claims data. When simulated data are injected with relative risks (RR) ≥ 2, all designs have good predictive accuracy (AUC > 0.90), but when RR < 2, no methods achieve 100 % predictions. Each method exhibits a different bias profile, which changes with the effect size., Conclusions: OSIM2 can support methodological research. Results from simulation suggest method operating characteristics are far from nominal properties.

Background: There is great variation in choices of method and specific analytical details in epidemiological studies, resulting in widely varying results even when studying the same drug and outcome in the same database. Not only does this variation undermine the credibility of the research but it limits our ability to improve the methods., Methods: In order to evaluate the performance of methods and analysis choices we used standard references and a literature review to identify 164 positive controls (drug-outcome pairs believed to represent true adverse drug reactions), and 234 negative controls (drug-outcome pairs for which we have confidence there is no direct causal relationship). We tested 3,748 unique analyses (methods in combination with specific analysis choices) that represent the full range of approaches to adjusting for confounding in five large observational datasets on these controls. We also evaluated the impact of increasingly specific outcome definitions, and performed a replication study in six additional datasets. We characterized the performance of each method using the area under the receiver operator curve (AUC), bias, and coverage probability. In addition, we developed simulated datasets that closely matched the characteristics of the observational datasets into which we inserted data consistent with known drug-outcome relationships in order to measure the accuracy of estimates generated by the analyses., Discussion: We expect the results of this systematic, empirical evaluation of the performance of these analyses across a moderate range of outcomes and databases to provide important insights into the methods used in epidemiological studies and to increase the consistency with which methods are applied, thereby increasing the confidence in results and our ability to systematically improve our approaches.

Background: Observational healthcare data offer the potential to identify adverse drug reactions that may be missed by spontaneous reporting. The self-controlled cohort analysis within the Temporal Pattern Discovery framework compares the observed-to-expected ratio of medical outcomes during post-exposure surveillance periods with those during a set of distinct pre-exposure control periods in the same patients. It utilizes an external control group to account for systematic differences between the different time periods, thus combining within- and between-patient confounder adjustment in a single measure., Objectives: To evaluate the performance of the calibrated self-controlled cohort analysis within Temporal Pattern Discovery as a tool for risk identification in observational healthcare data., Research Design: Different implementations of the calibrated self-controlled cohort analysis were applied to 399 drug-outcome pairs (165 positive and 234 negative test cases across 4 health outcomes of interest) in 5 real observational databases (four with administrative claims and one with electronic health records)., Measures: Performance was evaluated on real data through sensitivity/specificity, the area under receiver operator characteristics curve (AUC), and bias., Results: The calibrated self-controlled cohort analysis achieved good predictive accuracy across the outcomes and databases under study. The optimal design based on this reference set uses a 360 days surveillance period and a single control period 180 days prior to new prescriptions. It achieved an average AUC of 0.75 and AUC >0.70 in all but one scenario. A design with three separate control periods performed better for the electronic health records database and for acute renal failure across all data sets. The estimates for negative test cases were generally unbiased, but a minor negative bias of up to 0.2 on the RR-scale was observed with the configurations using multiple control periods, for acute liver injury and upper gastrointestinal bleeding., Conclusions: The calibrated self-controlled cohort analysis within Temporal Pattern Discovery shows promise as a tool for risk identification; it performs well at discriminating positive from negative test cases. The optimal parameter configuration may vary with the data set and medical outcome of interest.

Background: A systematic risk identification system has the potential to study all marketed drugs. However, the rates of drug exposure and outcome occurrences in observational databases, the database size and the desired risk detection threshold determine the power and therefore limit the feasibility of the application of appropriate analytical methods. Drugs vary dramatically for these parameters because of their prevalence of indication, cost, time on the market, payer formularies, market pressures and clinical guidelines., Objectives: Evaluate (i) the feasibility of a risk identification system based on commercially available observational databases, (ii) the range of drugs that can be studied for certain outcomes, (iii) the influence of underpowered drug-outcome pairs on the performance of analytical methods estimating the strength of their association and (iv) the time required from the introduction of a new drug to accumulate sufficient data for signal detection., Methods: As part of the Observational Medical Outcomes Partnership experiment, we used data from commercially available observational databases and calculated the minimal detectable relative risk of all pairs of marketed drugs and eight health outcomes of interest. We then studied an array of analytical methods for their ability to distinguish between pre-determined positive and negative drug-outcome test pairs. The positive controls contained active ingredients with evidence of a positive association with the outcome, and the negative controls had no such evidence. As a performance measure we used the area under the receiver operator characteristics curve (AUC). We compared the AUC of methods using all test pairs or only pairs sufficiently powered for detection of a relative risk of 1.25. Finally, we studied all drugs introduced to the market in 2003-2008 and determined the time required to achieve the same minimal detectable relative risk threshold., Results: The performance of methods improved after restricting them to fully powered drug-outcome pairs. The availability of drug-outcome pairs with sufficient power to detect a relative risk of 1.25 varies enormously among outcomes. Depending on the market uptake, drugs can generate relevant signals in the first month after approval, or never reach sufficient power., Conclusion: The incidence of drugs and important outcomes determines sample size and method performance in estimating drug-outcome associations. Careful consideration is therefore necessary to choose databases and outcome definitions, particularly for newly introduced drugs.

Background: Considerable attention now focuses on the use of large-scale observational healthcare data for understanding drug safety. In this context, analysts utilize a variety of statistical and epidemiological approaches such as case-control, cohort, and self-controlled methods. The operating characteristics of these methods are poorly understood., Objective: Establish the operating characteristics of the case-control method for large scale observational analysis in drug safety., Research Design: We empirically evaluated the case-control approach in 5 real observational healthcare databases and 6 simulated datasets. We retrospectively studied the predictive accuracy of the method when applied to a collection of 165 positive controls and 234 negative controls across 4 outcomes: acute liver injury, acute myocardial infarction, acute kidney injury, and upper gastrointestinal bleeding., Results: In our experiment, the case-control method provided weak discrimination between positive and negative controls. Furthermore, the method yielded positively biased estimates and confidence intervals that had poor coverage properties., Conclusions: For the four outcomes we examined, the case-control method may not be the method of choice for estimating potentially harmful effects of drugs.

Background: The self-controlled case series (SCCS) offers potential as an statistical method for risk identification involving medical products from large-scale observational healthcare data. However, analytic design choices remain in encoding the longitudinal health records into the SCCS framework and its risk identification performance across real-world databases is unknown., Objectives: To evaluate the performance of SCCS and its design choices as a tool for risk identification in observational healthcare data., Research Design: We examined the risk identification performance of SCCS across five design choices using 399 drug-health outcome pairs in five real observational databases (four administrative claims and one electronic health records). In these databases, the pairs involve 165 positive controls and 234 negative controls. We also consider several synthetic databases with known relative risks between drug-outcome pairs., Measures: We evaluate risk identification performance through estimating the area under the receiver-operator characteristics curve (AUC) and bias and coverage probability in the synthetic examples., Results: The SCCS achieves strong predictive performance. Twelve of the twenty health outcome-database scenarios return AUCs >0.75 across all drugs. Including all adverse events instead of just the first per patient and applying a multivariate adjustment for concomitant drug use are the most important design choices. However, the SCCS as applied here returns relative risk point-estimates biased towards the null value of 1 with low coverage probability., Conclusions: The SCCS recently extended to apply a multivariate adjustment for concomitant drug use offers promise as a statistical tool for risk identification in large-scale observational healthcare databases. Poor estimator calibration dampens enthusiasm, but on-going work should correct this short-coming.

Background: Determining the aetiology of acute liver injury (ALI) may be challenging to both clinicians and researchers. Observational research is particularly useful in studying rare medical outcomes such as ALI; however, case definitions for ALI in previous observational studies lack consistency and sensitivity. ALI is a clinically important condition with various aetiologies, including drug exposure., Objective: The aim of this study was to evaluate four distinct case definitions for ALI across a diverse set of large observational databases, providing a better understanding of ALI prevalence and natural history., Data Sources: Seven healthcare databases: GE Healthcare, MarketScan(®) Lab Database, Humana Inc., Partners HealthCare System, Regenstrief Institute, SDI Health (now IMS Health, Inc.), and the National Patient Care Database of the Veterans Health Administration., Methods: We evaluated prevalence of ALI through the application of four distinct case definitions across seven observational healthcare databases. We described how laboratory and clinical characteristics of identified case populations varied across definitions and examined the prevalence of other hepatobiliary disorders among identified ALI cases that may decrease suspicion of drug-induced liver injury (DILI) in particular., Results: This study demonstrated that increasing the restrictiveness of the case definition resulted in fewer cases, but greater prevalence of ALI clinical features. Considerable heterogeneity in the frequency of laboratory testing and results observed among cases meeting the most restrictive definition suggests that the clinical features, monitoring patterns and suspicion of ALI are highly variable among patients., Conclusions: Creation of four distinct case definitions and application across a disparate set of observational databases resulted in significant variation in the prevalence of ALI. A greater understanding of the natural history of ALI through examination of electronic healthcare data can facilitate development of reliable and valid ALI case definitions that may enhance the ability to accurately identify associations between ALI and drug exposures. Considerable heterogeneity in laboratory values and frequency of laboratory testing among individuals meeting the criteria for ALI suggests that the evaluation of ALI is highly variable.

Background: Expanded availability of observational healthcare data (both administrative claims and electronic health records) has prompted the development of statistical methods for identifying adverse events associated with medical products, but the operating characteristics of these methods when applied to the real-world data are unknown., Methods: We studied the performance of eight analytic methods for estimating of the strength of association-relative risk (RR) and associated standard error of 53 drug-adverse event outcome pairs, both positive and negative controls. The methods were applied to a network of ten observational healthcare databases, comprising over 130 million lives. Performance measures included sensitivity, specificity, and positive predictive value of methods at RR thresholds achieving statistical significance of p < 0.05 or p < 0.001 and with absolute threshold RR > 1.5, as well as threshold-free measures such as area under receiver operating characteristic curve (AUC)., Results: Although no specific method demonstrated superior performance, the aggregate results provide a benchmark and baseline expectation for risk identification method performance. At traditional levels of statistical significance (RR > 1, p < 0.05), all methods have a false positive rate >18%, with positive predictive value <38%. The best predictive model, high-dimensional propensity score, achieved an AUC  =  0.77. At 50% sensitivity, false positive rate ranged from 16% to 30%. At 10% false positive rate, sensitivity of the methods ranged from 9% to 33%., Conclusions: Systematic processes for risk identification can provide useful information to supplement an overall safety assessment, but assessment of methods performance suggests a substantial chance of identifying false positive associations., (Copyright © 2012 John Wiley & Sons, Ltd.)

Purpose: To explore the use of disproportionality analysis of medication error data as a novel method to identify relationships that might not be obvious through traditional analyses. This approach can supplement descriptive data and target quality improvement efforts., Methods: Data came from the Medication Error Quality Initiative (MEQI) individual event reporting system. Participants were North Carolina nursing homes who submitted incident reports to the Web-based MEQI data repository during the 2006 and 2007 reporting years. Data from 206 nursing homes were summarized descriptively and then disproportionality analysis was applied. Associations between medication type and possible causes at the state level were explored. A single nursing home was selected to illustrate how the method might inform quality improvement at the facility level. Disproportionality analysis of drug errors in this home was compared with benchmarking., Results: Statewide, 59 drug-cause pairs met the disproportionality signal and 11 occurred in 10 or more reports. Among these, warfarin was co-reported with communication errors; esomeprazole, risperidone, and nitrofurantoin were disproportionately associated with transcription error; and oxycodone and morphine were disproportionately reported with name confusion. Facility-level analyses illustrate how descriptive frequencies and disproportionality analysis are complementary, but also identify different safety targets., Conclusions: Exploratory analysis tools can help identify medication error types that occur at disproportionate rates. Candidate associations might be used to target patient safety work, although further evaluation is needed to determine the value of this information., (Copyright © 2010 John Wiley & Sons, Ltd.)