Search

Background and Purpose: Hemorrhagic transformation (HT) after cerebral infarction is a complex and multifactorial phenomenon in the acute stage of ischemic stroke, and often results in a poor prognosis. Thus, identifying risk factors and making an early prediction of HT in acute cerebral infarction contributes not only to the selections of therapeutic regimen but also, more importantly, to the improvement of prognosis of acute cerebral infarction. The purpose of this study was to develop and validate a model to predict a patient's risk of HT within 30 days of initial ischemic stroke., Methods: We utilized a retrospective multicenter observational cohort study design to develop a Lasso Logistic Regression prediction model with a large, US Electronic Health Record dataset which structured to the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM). To examine clinical transportability, the model was externally validated across 10 additional real-world healthcare datasets include EHR records for patients from America, Europe and Asia., Results: In the database the model was developed, the target population cohort contained 621,178 patients with ischemic stroke, of which 5,624 patients had HT within 30 days following initial ischemic stroke. 612 risk predictors, including the distance a patient travels in an ambulance to get to care for a HT, were identified. An area under the receiver operating characteristic curve (AUC) of 0.75 was achieved in the internal validation of the risk model. External validation was performed across 10 databases totaling 5,515,508 patients with ischemic stroke, of which 86,401 patients had HT within 30 days following initial ischemic stroke. The mean external AUC was 0.71 and ranged between 0.60-0.78., Conclusions: A HT prognostic predict model was developed with Lasso Logistic Regression based on routinely collected EMR data. This model can identify patients who have a higher risk of HT than the population average with an AUC of 0.78. It shows the OMOP CDM is an appropriate data standard for EMR secondary use in clinical multicenter research for prognostic prediction model development and validation. In the future, combining this model with clinical information systems will assist clinicians to make the right therapy decision for patients with acute ischemic stroke., Competing Interests: Dr. Reps, Dr. Ryan, Ms. Voss and Dr. Rao are employees of Janssen Research and Development. This does not alter our adherence to PLOS ONE policies on sharing data and materials.

Background: A systematic risk identification system has the potential to study all marketed drugs. However, the rates of drug exposure and outcome occurrences in observational databases, the database size and the desired risk detection threshold determine the power and therefore limit the feasibility of the application of appropriate analytical methods. Drugs vary dramatically for these parameters because of their prevalence of indication, cost, time on the market, payer formularies, market pressures and clinical guidelines., Objectives: Evaluate (i) the feasibility of a risk identification system based on commercially available observational databases, (ii) the range of drugs that can be studied for certain outcomes, (iii) the influence of underpowered drug-outcome pairs on the performance of analytical methods estimating the strength of their association and (iv) the time required from the introduction of a new drug to accumulate sufficient data for signal detection., Methods: As part of the Observational Medical Outcomes Partnership experiment, we used data from commercially available observational databases and calculated the minimal detectable relative risk of all pairs of marketed drugs and eight health outcomes of interest. We then studied an array of analytical methods for their ability to distinguish between pre-determined positive and negative drug-outcome test pairs. The positive controls contained active ingredients with evidence of a positive association with the outcome, and the negative controls had no such evidence. As a performance measure we used the area under the receiver operator characteristics curve (AUC). We compared the AUC of methods using all test pairs or only pairs sufficiently powered for detection of a relative risk of 1.25. Finally, we studied all drugs introduced to the market in 2003-2008 and determined the time required to achieve the same minimal detectable relative risk threshold., Results: The performance of methods improved after restricting them to fully powered drug-outcome pairs. The availability of drug-outcome pairs with sufficient power to detect a relative risk of 1.25 varies enormously among outcomes. Depending on the market uptake, drugs can generate relevant signals in the first month after approval, or never reach sufficient power., Conclusion: The incidence of drugs and important outcomes determines sample size and method performance in estimating drug-outcome associations. Careful consideration is therefore necessary to choose databases and outcome definitions, particularly for newly introduced drugs.

Background: Observational healthcare data offer the potential to identify adverse drug reactions that may be missed by spontaneous reporting. The self-controlled cohort analysis within the Temporal Pattern Discovery framework compares the observed-to-expected ratio of medical outcomes during post-exposure surveillance periods with those during a set of distinct pre-exposure control periods in the same patients. It utilizes an external control group to account for systematic differences between the different time periods, thus combining within- and between-patient confounder adjustment in a single measure., Objectives: To evaluate the performance of the calibrated self-controlled cohort analysis within Temporal Pattern Discovery as a tool for risk identification in observational healthcare data., Research Design: Different implementations of the calibrated self-controlled cohort analysis were applied to 399 drug-outcome pairs (165 positive and 234 negative test cases across 4 health outcomes of interest) in 5 real observational databases (four with administrative claims and one with electronic health records)., Measures: Performance was evaluated on real data through sensitivity/specificity, the area under receiver operator characteristics curve (AUC), and bias., Results: The calibrated self-controlled cohort analysis achieved good predictive accuracy across the outcomes and databases under study. The optimal design based on this reference set uses a 360 days surveillance period and a single control period 180 days prior to new prescriptions. It achieved an average AUC of 0.75 and AUC >0.70 in all but one scenario. A design with three separate control periods performed better for the electronic health records database and for acute renal failure across all data sets. The estimates for negative test cases were generally unbiased, but a minor negative bias of up to 0.2 on the RR-scale was observed with the configurations using multiple control periods, for acute liver injury and upper gastrointestinal bleeding., Conclusions: The calibrated self-controlled cohort analysis within Temporal Pattern Discovery shows promise as a tool for risk identification; it performs well at discriminating positive from negative test cases. The optimal parameter configuration may vary with the data set and medical outcome of interest.

Background: Observational healthcare data offer the potential to enable identification of risks of medical products, but appropriate methodology has not yet been defined. The new user cohort method, which compares the post-exposure rate among the target drug to a referent comparator group, is the prevailing approach for many pharmacoepidemiology evaluations and has been proposed as a promising approach for risk identification but its performance in this context has not been fully assessed., Objectives: To evaluate the performance of the new user cohort method as a tool for risk identification in observational healthcare data., Research Design: The method was applied to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record) and in 6 simulated datasets with no effect and injected relative risks of 1.25, 1.5, 2, 4, and 10, respectively., Measures: Method performance was evaluated through Area Under ROC Curve (AUC), bias, and coverage probability., Results: The new user cohort method achieved modest predictive accuracy across the outcomes and databases under study, with the top-performing analysis near AUC >0.70 in most scenarios. The performance of the method was particularly sensitive to the choice of comparator population. For almost all drug-outcome pairs there was a large difference, either positive or negative, between the true effect size and the estimate produced by the method, although this error was near zero on average. Simulation studies showed that in the majority of cases, the true effect estimate was not within the 95 % confidence interval produced by the method., Conclusion: The new user cohort method can contribute useful information toward a risk identification system, but should not be considered definitive evidence given the degree of error observed within the effect estimates. Careful consideration of the comparator selection and appropriate calibration of the effect estimates is required in order to properly interpret study findings.

Background: Observational healthcare data offer the potential to enable identification of risks of medical products, but appropriate methodology has not yet been defined. The self-controlled cohort method, which compares the post-exposure outcome rate with the pre-exposure rate among an exposed cohort, has been proposed as a potential approach for risk identification but its performance has not been fully assessed., Objectives: To evaluate the performance of the self-controlled cohort method as a tool for risk identification in observational healthcare data., Research Design: The method was applied to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record) and in 6 simulated datasets with no effect and injected relative risks of 1.25, 1.5, 2, 4, and 10, respectively., Measures: Method performance was evaluated through area under ROC curve (AUC), bias, and coverage probability., Results: The self-controlled cohort design achieved strong predictive accuracy across the outcomes and databases under study, with the top-performing settings exceeding AUC >0.76 in all scenarios. However, the estimates generated were observed to be highly biased with low coverage probability., Conclusions: If the objective for a risk identification system is one of discrimination, the self-controlled cohort method shows promise as a potential tool for risk identification. However, if a system is intended to generate effect estimates to quantify the magnitude of potential risks, the self-controlled cohort method may not be suitable, and requires substantial calibration to be properly interpreted under nominal properties.

Background: The availability of large-scale observational healthcare data allows for the active monitoring of safety of drugs, but research is needed to determine which statistical methods are best suited for this task. Recently, the Longitudinal Gamma Poisson Shrinker (LGPS) and Longitudinal Evaluation of Observational Profiles of Adverse events Related to Drugs (LEOPARD) methods were developed specifically for this task. LGPS applies Bayesian shrinkage to an estimated incidence rate ratio, and LEOPARD aims to detect and discard associations due to protopathic bias. The operating characteristics of these methods still need to be determined., Objective: Establish the operating characteristics of LGPS and LEOPARD for large scale observational analysis in drug safety., Research Design: We empirically evaluated LGPS and LEOPARD in five real observational healthcare databases and six simulated datasets. We retrospectively studied the predictive accuracy of the methods when applied to a collection of 165 positive control and 234 negative control drug-outcome pairs across four outcomes: acute liver injury, acute myocardial infarction, acute kidney injury, and upper gastrointestinal bleeding., Results: In contrast to earlier findings, we found that LGPS and LEOPARD provide weak discrimination between positive and negative controls, although the use of LEOPARD does lead to higher performance in this respect. Furthermore, the methods produce biased estimates and confidence intervals that have poor coverage properties., Conclusions: For the four outcomes we examined, LGPS and LEOPARD may not be the designs of choice for risk identification.

Background: The self-controlled case series (SCCS) offers potential as an statistical method for risk identification involving medical products from large-scale observational healthcare data. However, analytic design choices remain in encoding the longitudinal health records into the SCCS framework and its risk identification performance across real-world databases is unknown., Objectives: To evaluate the performance of SCCS and its design choices as a tool for risk identification in observational healthcare data., Research Design: We examined the risk identification performance of SCCS across five design choices using 399 drug-health outcome pairs in five real observational databases (four administrative claims and one electronic health records). In these databases, the pairs involve 165 positive controls and 234 negative controls. We also consider several synthetic databases with known relative risks between drug-outcome pairs., Measures: We evaluate risk identification performance through estimating the area under the receiver-operator characteristics curve (AUC) and bias and coverage probability in the synthetic examples., Results: The SCCS achieves strong predictive performance. Twelve of the twenty health outcome-database scenarios return AUCs >0.75 across all drugs. Including all adverse events instead of just the first per patient and applying a multivariate adjustment for concomitant drug use are the most important design choices. However, the SCCS as applied here returns relative risk point-estimates biased towards the null value of 1 with low coverage probability., Conclusions: The SCCS recently extended to apply a multivariate adjustment for concomitant drug use offers promise as a statistical tool for risk identification in large-scale observational healthcare databases. Poor estimator calibration dampens enthusiasm, but on-going work should correct this short-coming.

Background: There has been only limited evaluation of statistical methods for identifying safety risks of drug exposure in observational healthcare data. Simulations can support empirical evaluation, but have not been shown to adequately model the real-world phenomena that challenge observational analyses., Objectives: To design and evaluate a probabilistic framework (OSIM2) for generating simulated observational healthcare data, and to use this data for evaluating the performance of methods in identifying associations between drug exposure and health outcomes of interest., Research Design: Seven observational designs, including case-control, cohort, self-controlled case series, and self-controlled cohort design were applied to 399 drug-outcome scenarios in 6 simulated datasets with no effect and injected relative risks of 1.25, 1.5, 2, 4, and 10, respectively., Subjects: Longitudinal data for 10 million simulated patients were generated using a model derived from an administrative claims database, with associated demographics, periods of drug exposure derived from pharmacy dispensings, and medical conditions derived from diagnoses on medical claims., Measures: Simulation validation was performed through descriptive comparison with real source data. Method performance was evaluated using Area Under ROC Curve (AUC), bias, and mean squared error., Results: OSIM2 replicates prevalence and types of confounding observed in real claims data. When simulated data are injected with relative risks (RR) ≥ 2, all designs have good predictive accuracy (AUC > 0.90), but when RR < 2, no methods achieve 100 % predictions. Each method exhibits a different bias profile, which changes with the effect size., Conclusions: OSIM2 can support methodological research. Results from simulation suggest method operating characteristics are far from nominal properties.

Background: Considerable attention now focuses on the use of large-scale observational healthcare data for understanding drug safety. In this context, analysts utilize a variety of statistical and epidemiological approaches such as case-control, cohort, and self-controlled methods. The operating characteristics of these methods are poorly understood., Objective: Establish the operating characteristics of the case-control method for large scale observational analysis in drug safety., Research Design: We empirically evaluated the case-control approach in 5 real observational healthcare databases and 6 simulated datasets. We retrospectively studied the predictive accuracy of the method when applied to a collection of 165 positive controls and 234 negative controls across 4 outcomes: acute liver injury, acute myocardial infarction, acute kidney injury, and upper gastrointestinal bleeding., Results: In our experiment, the case-control method provided weak discrimination between positive and negative controls. Furthermore, the method yielded positively biased estimates and confidence intervals that had poor coverage properties., Conclusions: For the four outcomes we examined, the case-control method may not be the method of choice for estimating potentially harmful effects of drugs.

Background: Observational healthcare data offer the potential to enable identification of risks of medical products, and the medical literature is replete with analyses that aim to accomplish this objective. A number of established analytic methods dominate the literature but their operating characteristics in real-world settings remain unknown., Objectives: To compare the performance of seven methods (new user cohort, case control, self-controlled case series, self-controlled cohort, disproportionality analysis, temporal pattern discovery, and longitudinal gamma poisson shrinker) as tools for risk identification in observational healthcare data., Research Design: The experiment applied each method to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record)., Measures: Method performance was evaluated through Area Under the receiver operator characteristics Curve (AUC), bias, mean square error, and confidence interval coverage probability., Results: Multiple methods offer strong predictive accuracy, with AUC > 0.70 achievable for all outcomes and databases with more than one analytical approach. Self-controlled methods (self-controlled case series, temporal pattern discovery, self-controlled cohort) had higher predictive accuracy than cohort and case-control methods across all databases and outcomes. Methods differed in the expected value and variance of the error distribution. All methods had lower coverage probability than the expected nominal properties., Conclusions: Observational healthcare data can inform risk identification of medical product effects on acute liver injury, acute myocardial infarction, acute renal failure and gastrointestinal bleeding. However, effect estimates from all methods require calibration to address inconsistency in method operating characteristics. Further empirical evaluation is required to gauge the generalizability of these findings to other databases and outcomes.

Background: Biomarkers are considered the method of choice for determining exposure to environmental contaminants and relating such exposures to health outcomes. However, the association between many biomarkers and outcome is not direct because of variability in sensitivity and susceptibility in the individual., Objectives: We explore the relationship between environmental exposures and health outcomes as mitigated by differential susceptibility in individuals or populations and address the question "Can biomarkers enable us to understand and quantify better the population burden of disease and health effects attributable to environmental exposures?", Methods: We use a case-study approach to develop the thesis that biomarkers offer a pathway to disaggregation of health effects into specific, if multiple, risk factors. We offer the point of view that a series or array of biomarkers, including biomarkers of exposure, biomarkers of susceptibility, and biomarkers of effect, used in concert offer the best means by which to effect this disaggregation. We commence our discussion by developing the characteristics of an ideal biomarker, then give some examples of commonly used biomarkers to show the strengths and weaknesses of current usage. We follow this by more detailed case-study assessment outlining the state-of-the-science in specific cases. We complete our work with recommendations regarding the future use of biomarkers and areas for continued development., Conclusions: The case studies provide examples of when and how biomarkers can be used to infer the source and magnitude of exposure among a set of competing sources and pathways. The answer to this question is chemical specific and relates to how well the biomarker matches the characteristics of an "ideal" biomarker-in particular ease of collection and persistence. The use of biomarkers in combination provides a better opportunity to disaggregate both source and pathway contributions.

Information on human time-activity patterns is often required to interpret environmental exposure data fully and to implement exposure assessment models. Data on short-term time-activity patterns for individuals, such as 1-day measurements, are relatively abundant. The reliability of such data for use in chronic exposure (e.g., 1 or more years) assessments performed for evaluation of health risks is not well understood. As part of the NHEXAS-Maryland investigation, daily time budget data for seven microenvironments were collected from 80 people during as many as six 1-week Cycles over a 12-month period. The data were summarized and analyzed statistically by sampling Cycle, day of week, and individual to characterize long-term average microenvironmental time budgets and to identify their determinants. Median times spent in transit, indoors at home, outside at home, indoors at work or school, outdoors at work or school, indoors at other locations, and outdoors at other locations were found to vary significantly, although not substantively in many cases, by time of year (i.e., Cycle), by day of week, and by individuals. Time budgets for most of the microenvironments also exhibited significant variability by gender, age group, education level, annual household income, and work status. The results indicate that short-term (e.g., 1-day) measures of microenvironmental time budgets for individuals are unlikely to be representative of their long-term patterns. Thus, health risk or epidemiological assessments performed for a population mean or specific quantile may be relatively insensitive to when time budget data were collected. However, the accuracy of such assessments performed for individuals is likely to be greatly improved by collection of time budget data from numerous points in time.

As part of a longitudinal investigation of environmental exposures to selected chemical contaminants, concentrations of the pesticide metabolites 1-naphthol (INAP), 3,5,6-trichloro-2-pyridinol (TCPY), malathion dicarboxylic acid (MDA), and atrazine mercapturate (AM) were measured in repeated samples obtained from 80 individuals in Maryland during 1995-1996. Up to six urine samples were collected from each individual at intervals of approximately 8 weeks over a 1-year period (i.e., one sample per participant in each of six cycles). INAP (median=4.2 microg/l and 3.3 microg/g creatinine) and TCPY (median=5.3 microg/l and 4.6. microg/g creatinine) were present in over 80% of the samples, while MDA and AM were detected infrequently (6.6% and <1% of samples, respectively). Geometric mean (GM) concentrations of INAP in urine did not vary significantly among sampling cycles. In contrast, GM concentrations of TCPY were significantly greater in samples collected during the spring and summer of 1996 than in the preceding fall and winter. Repeated measurements of INAP and TCPY from the same individual over time were highly variable. The average range of INAP and TCPY concentrations from the same individual were approximately 200% and 50% greater than the respective population mean levels. Geometric mean (GM) TCPY concentrations differed significantly between Caucasian (n=42, GM=5.7 microg/g creatinine) and African-American (n=11, GM=4.0 microg/g) participants and among education levels, but were not significantly different among groups classified by gender, age, or household income. In future research, environmental measurements of the parent compounds and questionnaire data collected concurrently with the biomarker data will be used to characterize the determinants of variability in the urinary pesticide metabolite levels.