Search

Postmarket safety surveillance is an integral part of mass vaccination programs. Typically relying on sequential analysis of real-world health data as they accrue, safety surveillance is challenged by sequential multiple testing and by biases induced by residual confounding in observational data. The current standard approach based on the maximized sequential probability ratio test (MaxSPRT) fails to satisfactorily address these practical challenges and it remains a rigid framework that requires prespecification of the surveillance schedule. We develop an alternative Bayesian surveillance procedure that addresses both aforementioned challenges using a more flexible framework. To mitigate bias, we jointly analyze a large set of negative control outcomes that are adverse events with no known association with the vaccines in order to inform an empirical bias distribution, which we then incorporate into estimating the effect of vaccine exposure on the adverse event of interest through a Bayesian hierarchical model. To address multiple testing and improve on flexibility, at each analysis timepoint, we update a posterior probability in favor of the alternative hypothesis that vaccination induces higher risks of adverse events, and then use it for sequential detection of safety signals. Through an empirical evaluation using six US observational healthcare databases covering more than 360 million patients, we benchmark the proposed procedure against MaxSPRT on testing errors and estimation accuracy, under two epidemiological designs, the historical comparator and the self-controlled case series. We demonstrate that our procedure substantially reduces Type 1 error rates, maintains high statistical power and fast signal detection, and provides considerably more accurate estimation than MaxSPRT. Given the extensiveness of the empirical study which yields more than 7 million sets of results, we present all results in a public R ShinyApp. As an effort to promote open science, we provide full implementation of our method in the open-source R package EvidenceSynthesis., (© 2023 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.)

Post-approval safety surveillance of medical products using observational healthcare data can help identify safety issues beyond those found in pre-approval trials. When testing sequentially as data accrue, maximum sequential probability ratio testing (MaxSPRT) is a common approach to maintaining nominal type 1 error. However, the true type 1 error may still deviate from the specified one because of systematic error due to the observational nature of the analysis. This systematic error may persist even after controlling for known confounders. Here we propose to address this issue by combing MaxSPRT with empirical calibration. In empirical calibration, we assume uncertainty about the systematic error in our analysis, the source of uncertainty commonly overlooked in practice. We infer a probability distribution of systematic error by relying on a large set of negative controls: exposure-outcome pairs where no causal effect is believed to exist. Integrating this distribution into our test statistics has previously been shown to restore type 1 error to nominal. Here we show how we can calibrate the critical value central to MaxSPRT. We evaluate this novel approach using simulations and real electronic health records, using H1N1 vaccinations during the 2009-2010 season as an example. Results show that combining empirical calibration with MaxSPRT restores nominal type 1 error. In our real-world example, adjusting for systematic error using empirical calibration has a larger impact than, and hence is just as essential as, adjusting for sequential testing using MaxSPRT. We recommend performing both, using the method described here., (© 2023 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.)

Purpose: To compare the incidence of diabetic ketoacidosis (DKA) among patients with type 2 diabetes mellitus (T2DM) who were new users of sodium glucose co-transporter 2 inhibitors (SGLT2i) versus other classes of antihyperglycemic agents (AHAs)., Methods: Patients were identified from four large US claims databases using broad (all T2DM patients) and narrow (intended to exclude patients with type 1 diabetes or secondary diabetes misclassified as T2DM) definitions of T2DM. New users of SGLT2i and seven groups of comparator AHAs were matched (1:1) on exposure propensity scores to adjust for imbalances in baseline covariates. Cox proportional hazards regression models, conditioned on propensity score-matched pairs, were used to estimate hazard ratios (HRs) of DKA for new users of SGLT2i versus other AHAs. When I 2 <40%, a combined HR across the four databases was estimated., Results: Using the broad definition of T2DM, new users of SGLT2i had an increased risk of DKA versus sulfonylureas (HR [95% CI]: 1.53 [1.31-1.79]), DPP-4i (1.28 [1.11-1.47]), GLP-1 receptor agonists (1.34 [1.12-1.60]), metformin (1.31 [1.11-1.54]), and insulinotropic AHAs (1.38 [1.15-1.66]). Using the narrow definition of T2DM, new users of SGLT2i had an increased risk of DKA versus sulfonylureas (1.43 [1.01-2.01]). New users of SGLT2i had a lower risk of DKA versus insulin and a similar risk as thiazolidinediones, regardless of T2DM definition., Conclusions: Increased risk of DKA was observed for new users of SGLT2i versus several non-SGLT2i AHAs when T2DM was defined broadly. When T2DM was defined narrowly to exclude possible misclassified patients, an increased risk of DKA with SGLT2i was observed compared with sulfonylureas., (© 2019 The Authors. Pharmacoepidemiology & Drug Safety Published by John Wiley & Sons Ltd.)

The case-control design is widely used in retrospective database studies, often leading to spectacular findings. However, results of these studies often cannot be replicated, and the advantage of this design over others is questionable. To demonstrate the shortcomings of applications of this design, we replicate two published case-control studies. The first investigates isotretinoin and ulcerative colitis using a simple case-control design. The second focuses on dipeptidyl peptidase-4 inhibitors and acute pancreatitis, using a nested case-control design. We include large sets of negative control exposures (where the true odds ratio is believed to be 1) in both studies. Both replication studies produce effect size estimates consistent with the original studies, but also generate estimates for the negative control exposures showing substantial residual bias. In contrast, applying a self-controlled design to answer the same questions using the same data reveals far less bias. Although the case-control design in general is not at fault, its application in retrospective database studies, where all exposure and covariate data for the entire cohort are available, is unnecessary, as other alternatives such as cohort and self-controlled designs are available. Moreover, by focusing on cases and controls it opens the door to inappropriate comparisons between exposure groups, leading to confounding for which the design has few options to adjust for. We argue that this design should no longer be used in these types of data. At the very least, negative control exposures should be used to prove that the concerns raised here do not apply., (© 2019 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.)

Purpose: Distributed research networks (DRNs) afford statistical power by integrating observational data from multiple partners for retrospective studies. However, laboratory test results across care sites are derived using different assays from varying patient populations, making it difficult to simply combine data for analysis. Additionally, existing normalization methods are not suitable for retrospective studies. We normalized laboratory results from different data sources by adjusting for heterogeneous clinico-epidemiologic characteristics of the data and called this the subgroup-adjusted normalization (SAN) method., Methods: Subgroup-adjusted normalization renders the means and standard deviations of distributions identical under population structure-adjusted conditions. To evaluate its performance, we compared SAN with existing methods for simulated and real datasets consisting of blood urea nitrogen, serum creatinine, hematocrit, hemoglobin, serum potassium, and total bilirubin. Various clinico-epidemiologic characteristics can be applied together in SAN. For simplicity of comparison, age and gender were used to adjust population heterogeneity in this study., Results: In simulations, SAN had the lowest standardized difference in means (SDM) and Kolmogorov-Smirnov values for all tests (p < 0.05). In a real dataset, SAN had the lowest SDM and Kolmogorov-Smirnov values for blood urea nitrogen, hematocrit, hemoglobin, and serum potassium, and the lowest SDM for serum creatinine (p < 0.05)., Conclusion: Subgroup-adjusted normalization performed better than normalization using other methods. The SAN method is applicable in a DRN environment and should facilitate analysis of data integrated across DRN partners for retrospective observational studies., (Copyright © 2015 John Wiley & Sons, Ltd.)

Often the literature makes assertions of medical product effects on the basis of ' p < 0.05'. The underlying premise is that at this threshold, there is only a 5% probability that the observed effect would be seen by chance when in reality there is no effect. In observational studies, much more than in randomized trials, bias and confounding may undermine this premise. To test this premise, we selected three exemplar drug safety studies from literature, representing a case-control, a cohort, and a self-controlled case series design. We attempted to replicate these studies as best we could for the drugs studied in the original articles. Next, we applied the same three designs to sets of negative controls: drugs that are not believed to cause the outcome of interest. We observed how often p < 0.05 when the null hypothesis is true, and we fitted distributions to the effect estimates. Using these distributions, we compute calibrated p-values that reflect the probability of observing the effect estimate under the null hypothesis, taking both random and systematic error into account. An automated analysis of scientific literature was performed to evaluate the potential impact of such a calibration. Our experiment provides evidence that the majority of observational studies would declare statistical significance when no effect is present. Empirical calibration was found to reduce spurious results to the desired 5% level. Applying these adjustments to literature suggests that at least 54% of findings with p < 0.05 are not actually statistically significant and should be reevaluated., (© 2013 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.)

Purpose: Most electronic health record databases contain unstructured free-text narratives, which cannot be easily analyzed. Case-detection algorithms are usually created manually and often rely only on using coded information such as International Classification of Diseases version 9 codes. We applied a machine-learning approach to generate and evaluate an automated case-detection algorithm that uses both free-text and coded information to identify asthma cases., Methods: The Integrated Primary Care Information (IPCI) database was searched for potential asthma patients aged 5-18 years using a broad query on asthma-related codes, drugs, and free text. A training set of 5032 patients was created by manually annotating the potential patients as definite, probable, or doubtful asthma cases or non-asthma cases. The rule-learning program RIPPER was then used to generate algorithms to distinguish cases from non-cases. An over-sampling method was used to balance the performance of the automated algorithm to meet our study requirements. Performance of the automated algorithm was evaluated against the manually annotated set., Results: The selected algorithm yielded a positive predictive value (PPV) of 0.66, sensitivity of 0.98, and specificity of 0.95 when identifying only definite asthma cases; a PPV of 0.82, sensitivity of 0.96, and specificity of 0.90 when identifying both definite and probable asthma cases; and a PPV of 0.57, sensitivity of 0.95, and specificity of 0.67 for the scenario identifying definite, probable, and doubtful asthma cases., Conclusions: The automated algorithm shows good performance in detecting cases of asthma utilizing both free-text and coded data. This algorithm will facilitate large-scale studies of asthma in the IPCI database., (Copyright © 2013 John Wiley & Sons, Ltd.)

Purpose: Increasingly, patient information is stored in electronic medical records, which could be reused for research. Often these records comprise unstructured narrative data, which are cumbersome to analyze. The authors investigated whether text mining can make these data suitable for epidemiological studies and compared a concept recognition approach and a range of machine learning techniques that require a manually annotated training set. The authors show how this training set can be created with minimal effort by using a broad database query., Methods: The approaches were tested on two data sets: a publicly available set of English radiology reports for which International Classification of Diseases, Ninth Revision, Clinical Modification code needed to be assigned and a set of Dutch GP records that needed to be classified as either liver disorder cases or noncases. Performance was tested against a manually created gold standard., Results: The best overall performance was achieved by a combination of a manually created filter for removing negations and speculations and rule learning algorithms such as RIPPER, with high scores on both the radiology reports (positive predictive value = 0.88, sensitivity = 0.85, specificity = 1.00) and the GP records (positive predictive value = 0.89, sensitivity =0.91, specificity =0.76)., Conclusions: Although a training set still needs to be created manually, text mining can help reduce the amount of manual work needed to incorporate narrative data in an epidemiological study and will make the data extraction more reproducible. An advantage of machine learning is that it is able to pick up specific language use, such as abbreviations and synonyms used by physicians., (Copyright © 2012 John Wiley & Sons, Ltd.)

Purpose: To provide estimates of the number and types of drugs that can be monitored for safety surveillance using electronic healthcare databases., Methods: Using data from eight European databases (administrative claims, medical records) and in the context of a cohort study, we determined the amount of drug exposure required for signal detection across varying magnitudes of relative risk (RR). We provide estimates of the number and types of drugs that can be monitored as a function of actual use, minimal detectable RR, and empirically derived incidence rates for the following adverse events: (i) acute myocardial infarction; (ii) acute renal failure; (iii) anaphylactic shock; (iv) bullous eruptions; (v) rhabdomyolysis; and (vi) upper gastrointestinal bleeding. We performed data simulation to see how expansion of database size would influence the capabilities of such system., Results: Data from 1,947,452 individuals (59,594,132 person-years follow-up) who used 2,289 drugs in the EU-ADR network show that for a frequent event such as acute myocardial infarction, there are 531 drugs (23% of total) for which an association with RR = 2, if present, can be investigated. For a rare event such as rhabdomyolysis, there are 19 drugs (1%) for which an association of same magnitude can be investigated., Conclusion: Active surveillance using healthcare data-based networks for signal detection is feasible, although the leverage to do so may be low for infrequently used drugs and for rare outcomes. Extending database network size to include data from heterogeneous populations and increasing follow-up time are warranted to maximize leverage of these surveillance systems., (Copyright © 2012 John Wiley & Sons, Ltd.)

Purpose: There is a growing interest in using longitudinal observational databases for drug safety signal detection, but most of the existing statistical methods are tailored towards spontaneous reporting. Here a sequential set of methods for detecting and filtering drug safety signals in longitudinal databases is presented., Method: Longitudinal GPS (LGPS) is a modification of the Gamma Poisson Shrinker (GPS) that uses person time rather than case counts for the estimation of the expected number of events. Longitudinal Evaluation of Observational Profiles of Adverse events Related to Drugs (LEOPARD) is a method that can be used to automatically discard false drug-event associations caused by protopathic bias or misclassification of the dates of the adverse events by comparing prior event prescription rates to post event prescription rates. LEOPARD can generate a single test statistic, or a visualization that can be used for more qualitative information on the relationship between drug and event. Both methods were evaluated using data simulated using the Observational medical dataset SIMulator (OSIM), including the dataset used in the Observational Medical Outcomes Partnership (OMOP) cup, a recent public competition for signal detection methods. The Mean Average Precision (MAP) was used for performance measurement., Results: On the OMOP cup data, LGPS achieved a MAP of 0.245, and the combination of LGPS and LEOPARD achieved a MAP of 0.260, the highest score in the competition., Conclusions: The sequential use of LGPS and LEOPARD have proven to be a useful novel set of methods for drug safety signal detection on longitudinal health records., (Copyright © 2010 John Wiley & Sons, Ltd.)

Purpose: In this proof-of-concept paper we describe the framework, process, and preliminary results of combining data from European electronic healthcare record (EHR) databases for large-scale monitoring of drug safety., Methods: Aggregated demographic, clinical, and prescription data from eight databases in four countries (Denmark, Italy, Netherlands, the UK) were pooled using a distributed network approach by generation of common input data followed by local aggregation through custom-built software, Jerboa(©). Comparison of incidence rates of upper gastrointestinal bleeding (UGIB) and nonsteroidal anti-inflammatory drug (NSAID) utilization patterns were used to evaluate data harmonization and quality across databases. The known association of NSAIDs and UGIB was employed to demonstrate sensitivity of the system by comparing incidence rate ratios (IRRs) of UGIB during NSAID use to UGIB during all other person-time., Results: The study population for this analysis comprised 19,647,445 individuals corresponding to 59,929,690 person-years of follow-up. 39,967 incident cases of UGIB were identified during the study period. Crude incidence rates varied between 38.8 and 109.5/100,000 person-years, depending on country and type of database, while age-standardized rates ranged from 25.1 to 65.4/100,000 person-years. NSAID use patterns were similar for databases within the same country but heterogeneous among different countries. A statistically significant age- and gender-adjusted association between use of any NSAID and increased risk for UGIB was confirmed in all databases, IRR from 2.0 (95%CI:1.7-2.2) to 4.3 (95%CI: 4.1-4.5)., Conclusions: Combining data from EHR databases of different countries to identify drug-adverse event associations is feasible and can set the stage for changing and enlarging the scale for drug safety monitoring., (Copyright © 2010 John Wiley & Sons, Ltd.)