Search

Preeclampsia is a pregnancy‐associated condition posing risks of both fetal and maternal mortality and morbidity that can only resolve following delivery and removal of the placenta. Because in its typical form preeclampsia can arise before delivery, but not after, these two events exemplify the time‐to‐event setting of "semi‐competing risks" in which a non‐terminal event of interest is subject to the occurrence of a terminal event of interest. The semi‐competing risks framework presents a valuable opportunity to simultaneously address two clinically meaningful risk modeling tasks: (i) characterizing risk of developing preeclampsia, and (ii) characterizing time to delivery after onset of preeclampsia. However, some people with preeclampsia deliver immediately upon diagnosis, while others are admitted and monitored for an extended period before giving birth, resulting in two distinct trajectories following the non‐terminal event, which we call "clinically immediate" and "non‐immediate" terminal events. Though such phenomena arise in many clinical contexts, to‐date there have not been methods developed to acknowledge the complex dependencies between such outcomes, nor leverage these phenomena to gain new insight into individualized risk. We address this gap by proposing a novel augmented frailty‐based illness‐death model with a binary submodel to distinguish risk of immediate terminal event following the non‐terminal event. The model admits direct dependence of the terminal event on the non‐terminal event through flexible regression specification, as well as indirect dependence via a shared frailty term linking each submodel. We develop an efficient Bayesian sampler for estimation and corresponding model fit metrics, and derive formulae for dynamic risk prediction. In an extended example using pregnancy outcome data from an electronic health record, we demonstrate the proposed model's direct applicability to address a broad range of clinical questions. [ABSTRACT FROM AUTHOR]

The analysis of streaming time‐to‐event cohorts has garnered significant research attention. Most existing methods require observed cohorts from a study sequence to be independent and identically sampled from a common model. This assumption may be easily violated in practice. Our methodology operates within the framework of online data updating, where risk estimates for each cohort of interest are continuously refreshed using the latest observations and historical summary statistics. At each streaming stage, we introduce parameters to quantify the potential discrepancy between batch‐specific effects from adjacent cohorts. We then employ penalized estimation techniques to identify nonzero discrepancy parameters, allowing us to adaptively adjust risk estimates based on current data and historical trends. We illustrate our proposed method through extensive empirical simulations and a lung cancer data analysis. [ABSTRACT FROM AUTHOR]

Moderate calibration, the expected event probability among observations with predicted probability z being equal to z, is a desired property of risk prediction models. Current graphical and numerical techniques for evaluating moderate calibration of risk prediction models are mostly based on smoothing or grouping the data. As well, there is no widely accepted inferential method for the null hypothesis that a model is moderately calibrated. In this work, we discuss recently‐developed, and propose novel, methods for the assessment of moderate calibration for binary responses. The methods are based on the limiting distributions of functions of standardized partial sums of prediction errors converging to the corresponding laws of Brownian motion. The novel method relies on well‐known properties of the Brownian bridge which enables joint inference on mean and moderate calibration, leading to a unified "bridge" test for detecting miscalibration. Simulation studies indicate that the bridge test is more powerful, often substantially, than the alternative test. As a case study we consider a prediction model for short‐term mortality after a heart attack, where we provide suggestions on graphical presentation and the interpretation of results. Moderate calibration can be assessed without requiring arbitrary grouping of data or using methods that require tuning of parameters. [ABSTRACT FROM AUTHOR]

Introduction: There is currently no guidance on how to assess the calibration of multistate models used for risk prediction. We introduce several techniques that can be used to produce calibration plots for the transition probabilities of a multistate model, before assessing their performance in the presence of random and independent censoring through a simulation. Methods: We studied pseudo‐values based on the Aalen‐Johansen estimator, binary logistic regression with inverse probability of censoring weights (BLR‐IPCW), and multinomial logistic regression with inverse probability of censoring weights (MLR‐IPCW). The MLR‐IPCW approach results in a calibration scatter plot, providing extra insight about the calibration. We simulated data with varying levels of censoring and evaluated the ability of each method to estimate the calibration curve for a set of predicted transition probabilities. We also developed evaluated the calibration of a model predicting the incidence of cardiovascular disease, type 2 diabetes and chronic kidney disease among a cohort of patients derived from linked primary and secondary healthcare records. Results: The pseudo‐value, BLR‐IPCW, and MLR‐IPCW approaches give unbiased estimates of the calibration curves under random censoring. These methods remained predominately unbiased in the presence of independent censoring, even if the censoring mechanism was strongly associated with the outcome, with bias concentrated in low‐density regions of predicted transition probability. Conclusions: We recommend implementing either the pseudo‐value or BLR‐IPCW approaches to produce a calibration curve, combined with the MLR‐IPCW approach to produce a calibration scatter plot. The methods have been incorporated into the "calibmsm" R package available on CRAN. [ABSTRACT FROM AUTHOR]

Mammography remains the primary screening strategy for breast cancer, which continues to be the most prevalent cancer diagnosis among women globally. Because screening mammograms capture both the left and right breast, there is a nonnegligible correlation between the pair of images. Previous studies have explored the concept of averaging between the pair of images after proper image registration; however, no comparison has been made in directly utilizing the paired images. In this paper, we extend the bivariate functional principal component analysis over triangulations to jointly characterize the pair of imaging data bounded in an irregular domain and then nest the extracted features within the survival model to predict the onset of breast cancer. The method is applied to our motivating data from the Joanne Knight Breast Health Cohort at Siteman Cancer Center. Our findings indicate that there was no statistically significant difference in model discrimination performance between averaging the pair of images and jointly modeling the two images. Although the breast cancer study did not reveal any significant difference, it is worth noting that the methods proposed here can be readily extended to other studies involving paired or multivariate imaging data. [ABSTRACT FROM AUTHOR]

We have previously proposed temporal recalibration to account for trends in survival over time to improve the calibration of predictions from prognostic models for new patients. This involves first estimating the predictor effects using data from all individuals (full dataset) and then re‐estimating the baseline using a subset of the most recent data whilst constraining the predictor effects to remain the same. In this article, we demonstrate how temporal recalibration can be applied in competing risk settings by recalibrating each cause‐specific (or subdistribution) hazard model separately. We illustrate this using an example of colon cancer survival with data from the Surveillance Epidemiology and End Results (SEER) program. Data from patients diagnosed in 1995–2004 were used to fit two models for deaths due to colon cancer and other causes respectively. We discuss considerations that need to be made in order to apply temporal recalibration such as the choice of data used in the recalibration step. We also demonstrate how to assess the calibration of these models in new data for patients diagnosed subsequently in 2005. Comparison was made to a standard analysis (when improvements over time are not taken into account) and a period analysis which is similar to temporal recalibration but differs in the data used to estimate the predictor effects. The 10‐year calibration plots demonstrated that using the standard approach over‐estimated the risk of death due to colon cancer and the total risk of death and that calibration was improved using temporal recalibration or period analysis. [ABSTRACT FROM AUTHOR]

Understanding the relationships between exposure and disease incidence is an important problem in environmental epidemiology. Typically, a large number of these exposures are measured, and it is found either that a few exposures transmit risk or that each exposure transmits a small amount of risk, but, taken together, these may pose a substantial disease risk. Further, these exposure effects can be nonlinear. We develop a latent functional approach, which assumes that the individual effect of each exposure can be characterized as one of a series of unobserved functions, where the number of latent functions is less than or equal to the number of exposures. We propose Bayesian methodology to fit models with a large number of exposures and show that existing Bayesian group LASSO approaches are a special case of the proposed model. An efficient Markov chain Monte Carlo sampling algorithm is developed for carrying out Bayesian inference. The deviance information criterion is used to choose an appropriate number of nonlinear latent functions. We demonstrate the good properties of the approach using simulation studies. Further, we show that complex exposure relationships can be represented with only a few latent functional curves. The proposed methodology is illustrated with an analysis of the effect of cumulative pesticide exposure on cancer risk in a large cohort of farmers. [ABSTRACT FROM AUTHOR]

Mammography screening programs are aimed at reducing mortality due to breast cancer by detecting tumors at an early stage. There is currently interest in moving away from the age‐based screening programs, and toward personalized screening based on individual risk factors. To accomplish this, risk prediction models for breast cancer are needed to determine who should be screened, and when. We develop a novel approach using a (random effects) continuous growth model, which we apply to a large population‐based, Swedish screening cohort. Unlike existing breast cancer prediction models, this approach explicitly incorporates each woman's individual screening visits in the prediction. It jointly models invasive breast cancer tumor onset, tumor growth rate, symptomatic detection rate, and screening sensitivity. In addition to predicting the overall risk of invasive breast cancer, this model can make separate predictions regarding specific tumor sizes, and the mode of detection (eg, detected at screening, or through symptoms between screenings). It can also predict how these risks change depending on whether or not a woman will attend her next screening. In our study, we predict, given a future diagnosis, that the probability of having a tumor less than (as opposed to greater than) 10‐mm diameter, at detection, will be, on average, 2.6 times higher if a woman in the cohort attends their next screening. This indicates that the model can be used to evaluate the short‐term benefit of screening attendance, at an individual level. [ABSTRACT FROM AUTHOR]

Summary: This work is motivated by the need to accurately model a vector of responses related to pediatric functional status using administrative health data from inpatient rehabilitation visits. The components of the responses have known and structured interrelationships. To make use of these relationships in modeling, we develop a two‐pronged regularization approach to borrow information across the responses. The first component of our approach encourages joint selection of the effects of each variable across possibly overlapping groups of related responses and the second component encourages shrinkage of effects towards each other for related responses. As the responses in our motivating study are not normally‐distributed, our approach does not rely on an assumption of multivariate normality of the responses. We show that with an adaptive version of our penalty, our approach results in the same asymptotic distribution of estimates as if we had known in advance which variables have non‐zero effects and which variables have the same effects across some outcomes. We demonstrate the performance of our method in extensive numerical studies and in an application in the prediction of functional status of pediatric patients using administrative health data in a population of children with neurological injury or illness at a large children's hospital. [ABSTRACT FROM AUTHOR]

Prediction performance of a risk scoring system needs to be carefully assessed before its adoption in clinical practice. Clinical preventive care often uses risk scores to screen asymptomatic population. The primary clinical interest is to predict the risk of having an event by a prespecified future time t0 . Accuracy measures such as positive predictive values have been recommended for evaluating the predictive performance. However, for commonly used continuous or ordinal risk score systems, these measures require a subjective cutoff threshold value that dichotomizes the risk scores. The need for a cutoff value created barriers for practitioners and researchers. In this paper, we propose a threshold-free summary index of positive predictive values that accommodates time-dependent event status and competing risks. We develop a nonparametric estimator and provide an inference procedure for comparing this summary measure between 2 risk scores for censored time to event data. We conduct a simulation study to examine the finite-sample performance of the proposed estimation and inference procedures. Lastly, we illustrate the use of this measure on a real data example, comparing 2 risk score systems for predicting heart failure in childhood cancer survivors.

Prediction performance of a risk scoring system needs to be carefully assessed before its adoption in clinical practice. Clinical preventive care often uses risk scores to screen asymptomatic population. The primary clinical interest is to predict the risk of having an event by a prespecified future time t0 . Accuracy measures such as positive predictive values have been recommended for evaluating the predictive performance. However, for commonly used continuous or ordinal risk score systems, these measures require a subjective cutoff threshold value that dichotomizes the risk scores. The need for a cutoff value created barriers for practitioners and researchers. In this paper, we propose a threshold-free summary index of positive predictive values that accommodates time-dependent event status and competing risks. We develop a nonparametric estimator and provide an inference procedure for comparing this summary measure between 2 risk scores for censored time to event data. We conduct a simulation study to examine the finite-sample performance of the proposed estimation and inference procedures. Lastly, we illustrate the use of this measure on a real data example, comparing 2 risk score systems for predicting heart failure in childhood cancer survivors.

Calibration is a vital aspect of the performance of risk prediction models, but research in the context of ordinal outcomes is scarce. This study compared calibration measures for risk models predicting a discrete ordinal outcome, and investigated the impact of the proportional odds assumption on calibration and overfitting. We studied the multinomial, cumulative, adjacent category, continuation ratio, and stereotype logit/logistic models. To assess calibration, we investigated calibration intercepts and slopes, calibration plots, and the estimated calibration index. Using large sample simulations, we studied the performance of models for risk estimation under various conditions, assuming that the true model has either a multinomial logistic form or a cumulative logit proportional odds form. Small sample simulations were used to compare the tendency for overfitting between models. As a case study, we developed models to diagnose the degree of coronary artery disease (five categories) in symptomatic patients. When the true model was multinomial logistic, proportional odds models often yielded poor risk estimates, with calibration slopes deviating considerably from unity even on large model development datasets. The stereotype logistic model improved the calibration slope, but still provided biased risk estimates for individual patients. When the true model had a cumulative logit proportional odds form, multinomial logistic regression provided biased risk estimates, although these biases were modest. Nonproportional odds models require more parameters to be estimated from the data, and hence suffered more from overfitting. Despite larger sample size requirements, we generally recommend multinomial logistic regression for risk prediction modeling of discrete ordinal outcomes. [ABSTRACT FROM AUTHOR]

Summary: The nested case‐control (NCC) design has been widely adopted as a cost‐effective sampling design for biomarker research. Under the NCC design, markers are only measured for the NCC subcohort consisting of all cases and a fraction of the controls selected randomly from the matched risk sets of the cases. Robust methods for evaluating prediction performance of risk models have been derived under the inverse probability weighting framework. The probabilities of samples being included in the NCC cohort can be calculated based on the study design ''a previous study'' or estimated non‐parametrically ''a previous study''. Neither strategy works well due to model mis‐specification and the curse of dimensionality in practical settings where the sampling does not entirely follow the study design or depends on many factors. In this paper, we propose an alternative strategy to estimate the sampling probabilities based on a varying coefficient model, which attains a balance between robustness and the curse of dimensionality. The complex correlation structure induced by repeated finite risk set sampling makes the standard resampling procedure for variance estimation fail. We propose a perturbation resampling procedure that provides valid interval estimation for the proposed estimators. Simulation studies show that the proposed method performs well in finite samples. We apply the proposed method to the Nurses' Health Study II to develop and evaluate prediction models using clinical biomarkers for cardiovascular risk. [ABSTRACT FROM AUTHOR]

Clinical prediction models (CPMs) can predict clinically relevant outcomes or events. Typically, prognostic CPMs are derived to predict the risk of a single future outcome. However, there are many medical applications where two or more outcomes are of interest, meaning this should be more widely reflected in CPMs so they can accurately estimate the joint risk of multiple outcomes simultaneously. A potentially naïve approach to multi-outcome risk prediction is to derive a CPM for each outcome separately, then multiply the predicted risks. This approach is only valid if the outcomes are conditionally independent given the covariates, and it fails to exploit the potential relationships between the outcomes. This paper outlines several approaches that could be used to develop CPMs for multiple binary outcomes. We consider four methods, ranging in complexity and conditional independence assumptions: namely, probabilistic classifier chain, multinomial logistic regression, multivariate logistic regression, and a Bayesian probit model. These are compared with methods that rely on conditional independence: separate univariate CPMs and stacked regression. Employing a simulation study and real-world example, we illustrate that CPMs for joint risk prediction of multiple outcomes should only be derived using methods that model the residual correlation between outcomes. In such a situation, our results suggest that probabilistic classification chains, multinomial logistic regression or the Bayesian probit model are all appropriate choices. We call into question the development of CPMs for each outcome in isolation when multiple correlated or structurally related outcomes are of interest and recommend more multivariate approaches to risk prediction. [ABSTRACT FROM AUTHOR]

Receiver operating characteristic (ROC) and predictiveness curves are graphical tools to study the discriminative and predictive power of a continuous‐valued marker in a binary outcome. In this paper, a copula‐based construction of the joint density of the marker and the outcome is developed for plotting and analyzing both curves. The methodology only requires a copula function, the marginal distribution of the marker, and the prevalence rate for the model to be characterized. The adoption of the Gaussian copula and the customization of the margin for the marker are proposed for such characterization. The computation of both curves is numerically more feasible than methods that attempt to obtain one curve in terms of the other. Estimation is carried out using maximum likelihood and resampling‐based methods. Randomized quantile residuals from each conditional distribution are employed for both assessing the adequacy of the model and identifying outliers. The performance of the estimators of both curves and their underlying quantities is evaluated in simulation studies that assume different dependence structures and sample sizes. The methods are illustrated with an analysis of the level of progesterone receptor gene expression for the diagnosis and prediction of estrogen receptor‐positive breast cancer. [ABSTRACT FROM AUTHOR]

Longitudinal biomarker data are often collected in studies, providing important information regarding the probability of an outcome of interest occurring at a future time. With many new and evolving technologies for biomarker discovery, the number of biomarker measurements available for analysis of disease progression has increased dramatically. A large amount of data provides a more complete picture of a patient's disease progression, potentially allowing us to make more accurate and reliable predictions, but the magnitude of available data introduces challenges to most statistical analysts. Existing approaches suffer immensely from the curse of dimensionality. In this article, we propose methods for making dynamic risk predictions using repeatedly measured biomarkers of a large dimension, including cases when the number of biomarkers is close to the sample size. The proposed methods are computationally simple, yet sufficiently flexible to capture complex relationships between longitudinal biomarkers and potentially censored events times. The proposed approaches are evaluated by extensive simulation studies and are further illustrated by an application to a data set from the Nephrotic Syndrome Study Network. [ABSTRACT FROM AUTHOR]

Statistical learning methods are widely used in medical literature for the purpose of diagnosis or prediction. Conventional accuracy assessment via sensitivity, specificity, and ROC curves does not fully account for clinical utility of a specific model. Decision curve analysis (DCA) becomes a novel complement as it incorporates a clinical judgment of the relative value of benefits (treating a true positive case) and harms (treating a false positive case) associated with prediction models. The preference of a patient or a policy-maker is formulated statistically as the underlying threshold probability, above which the patient would choose to be treated. Net benefit is then calculated for possible threshold probability, which places benefits and harms on the same scale. We consider the inference problems for DCA in this paper. Interval estimation procedure and inference methodology are provided after we derive the relevant asymptotic properties. Our formulation can accommodate the classification problems with multiple categories. We carry out numerical studies to assess the performance of the proposed methods. An eye disease dataset is analyzed to illustrate our proposals. [ABSTRACT FROM AUTHOR]

Linear mixed models (LMMs) and their extensions have been widely used for high-dimensional genomic data analyses. While LMMs hold great promise for risk prediction research, the high dimensionality of the data and different effect sizes of genomic regions bring great analytical and computational challenges. In this work, we present a multikernel linear mixed model with adaptive lasso (KLMM-AL) to predict phenotypes using high-dimensional genomic data. We develop two algorithms for estimating parameters from our model and also establish the asymptotic properties of LMM with adaptive lasso when only one dependent observation is available. The proposed KLMM-AL can account for heterogeneous effect sizes from different genomic regions, capture both additive and nonadditive genetic effects, and adaptively and efficiently select predictive genomic regions and their corresponding effects. Through simulation studies, we demonstrate that KLMM-AL outperforms most of existing methods. Moreover, KLMM-AL achieves high sensitivity and specificity of selecting predictive genomic regions. KLMM-AL is further illustrated by an application to the sequencing dataset obtained from the Alzheimer's disease neuroimaging initiative. [ABSTRACT FROM AUTHOR]

When comparing performances of two risk prediction models, several metrics exist to quantify prognostic improvement, including the change in the area under the Receiver Operating Characteristic curve, the Integrated Discrimination Improvement, the Net Reclassification Index at event rate, the change in Standardized Net Benefit, the change in Brier score, and the change in scaled Brier score. We explore the behavior and interrelationships between these metrics under multivariate normality in nested and nonnested model comparisons. We demonstrate that, within the framework of linear discriminant analysis, all six statistics are functions of squared Mahalanobis distance, a robust metric that properly measures discrimination by quantifying the separation between the risk scores of events and nonevents. These relationships are important for overall interpretability and clinical usefulness. Through simulation, we demonstrate that the performance of the theoretical estimators under normality is comparable or superior to empirical estimation methods typically used by investigators. In particular, the theoretical estimators for the Net Reclassification Index and the change in Standardized Net Benefit exhibit less variability in their estimates as compared to their empirically estimated counterparts. Finally, we explore how these metrics behave with potentially nonnormal data by applying these methods in a practical example based on the sex-specific cardiovascular disease risk models from the Framingham Heart Study. Our findings aim to give greater insight into the behavior of these measures and the connections existing among them and to provide additional estimation methods with less variability for the Net Reclassification Index and the change in Standardized Net Benefit. [ABSTRACT FROM AUTHOR]

A property of diagnostic tests and risk models deserving more attention is risk stratification, defined as the ability of a test or model to separate those at high absolute risk of disease from those at low absolute risk. Risk stratification fills a gap between measures of classification (ie, area under the curve (AUC)) that do not require absolute risks and decision analysis that requires not only absolute risks but also subjective specification of costs and utilities. We introduce mean risk stratification (MRS) as the average change in risk of disease (posttest‐pretest) revealed by a diagnostic test or risk model dichotomized at a risk threshold. Mean risk stratification is particularly valuable for rare conditions, where AUC can be high but MRS can be low, identifying situations that temper overenthusiasm for screening with the new test/model. We apply MRS to the controversy over who should get testing for mutations in BRCA1/2 that cause high risks of breast and ovarian cancers. To reveal different properties of risk thresholds to refer women for BRCA1/2 testing, we propose an eclectic approach considering MRS and other metrics. The value of MRS is to interpret AUC in the context of BRCA1/2 mutation prevalence, providing a range of risk thresholds at which a risk model is "optimally informative," and to provide insight into why net benefit arrives to its conclusion. [ABSTRACT FROM AUTHOR]

Shared random parameter models (SRPMs) were first introduced by researchers at the National Heart Lung and Blood Institute (NHLBI) Biostatistics Branch for analyzing longitudinal data with informative dropout (Wu and Carroll, 1987; Wu and Bailey, 1988; Follmann and Wu, 1995; Albert and Follmann, 2000; Albert et al, 2002). This work was all focused on characterizing the longitudinal data process in the presence of an informative missing data mechanism that is treated as a nuisance. Shared random parameter modeling approaches have also been developed from the perspective of characterizing the relationship between longitudinal data and a subsequent outcome that may be an event time, a dichotomous measurement, or another longitudinal outcome. This article will review the early contributions of the NHLBI biostatisticians on SRPMs for analyzing longitudinal data with dropout and demonstrate how these ideas have, more recently, been applied in these other areas of biostatistics. Rather than focus on technical details or specific analyses, this article presents a conceptual framework for SRPMs within a historical context. [ABSTRACT FROM AUTHOR]

With rapidly evolving high-throughput technologies, studies are being initiated to accelerate the process toward precision medicine. The collection of the vast amounts of sequencing data provides us with great opportunities to systematically study the role of a deep catalog of sequencing variants in risk prediction. Nevertheless, the massive amount of noise signals and low frequencies of rare variants in sequencing data pose great analytical challenges on risk prediction modeling. Motivated by the development in spatial statistics, we propose a spatial autoregressive model with adaptive lasso (SARAL) for risk prediction modeling using high-dimensional sequencing data. The SARAL is a set-based approach, and thus, it reduces the data dimension and accumulates genetic effects within a single-nucleotide variant (SNV) set. Moreover, it allows different SNV sets having various magnitudes and directions of effect sizes, which reflects the nature of complex diseases. With the adaptive lasso implemented, SARAL can shrink the effects of noise SNV sets to be zero and, thus, further improve prediction accuracy. Through simulation studies, we demonstrate that, overall, SARAL is comparable to, if not better than, the genomic best linear unbiased prediction method. The method is further illustrated by an application to the sequencing data from the Alzheimer's Disease Neuroimaging Initiative. [ABSTRACT FROM AUTHOR]

Attempts to predict prognosis in cancer patients using high-dimensional genomic data such as gene expression in tumor tissue can be made difficult by the large number of features and the potential complexity of the relationship between features and the outcome. Integrating prior biological knowledge into risk prediction with such data by grouping genomic features into pathways and networks reduces the dimensionality of the problem and could improve prediction accuracy. Additionally, such knowledge-based models may be more biologically grounded and interpretable. Prediction could potentially be further improved by allowing for complex nonlinear pathway effects. The kernel machine framework has been proposed as an effective approach for modeling the nonlinear and interactive effects of genes in pathways for both censored and noncensored outcomes. When multiple pathways are under consideration, one may efficiently select informative pathways and aggregate their signals via multiple kernel learning (MKL), which has been proposed for prediction of noncensored outcomes. In this paper, we propose MKL methods for censored survival outcomes. We derive our approach for a general survival modeling framework with a convex objective function and illustrate its application under the Cox proportional hazards and semiparametric accelerated failure time models. Numerical studies demonstrate that the proposed MKL-based prediction methods work well in finite sample and can potentially outperform models constructed assuming linear effects or ignoring the group knowledge. The methods are illustrated with an application to 2 cancer data sets. [ABSTRACT FROM AUTHOR]

The integrated discrimination improvement (IDI) is commonly used to compare two risk prediction models; it summarizes the extent a new model increases risk in events and decreases risk in non-events. The IDI averages risks across events and non-events and is therefore susceptible to Simpson's paradox. In some settings, adding a predictive covariate to a well calibrated model results in an overall negative (positive) IDI. However, if stratified by that same covariate, the strata-specific IDIs are positive (negative). Meanwhile, the calibration (observed to expected ratio and Hosmer-Lemeshow Goodness of Fit Test), area under the receiver operating characteristic curve, and Brier score improve overall and by stratum. We ran extensive simulations to investigate the impact of an imbalanced covariate upon metrics (IDI, area under the receiver operating characteristic curve, Brier score, and R2), provide an analytic explanation for the paradox in the IDI, and use an investigative metric, a Weighted IDI, to better understand the paradox. In simulations, all instances of the paradox occurred under stratum-specific mis-calibration, yet there were mis-calibrated settings in which the paradox did not occur. The paradox is illustrated on Cancer Genomics Network data by calculating predictions based on two versions of BRCAPRO, a Mendelian risk prediction model for breast and ovarian cancer. In both simulations and the Cancer Genomics Network data, overall model calibration did not guarantee stratum-level calibration. We conclude that the IDI should only assess model performance among a clinically relevant subset when stratum-level calibration is strictly met and recommend calculating additional metrics to confirm the direction and conclusions of the IDI. Copyright © 2016 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Developing new measures of risk model performance is an active line of research, often motivated by the conventional wisdom that area under the ROC curve is an 'insensitive' measure of the additional predictive capacity offered by new biomarkers. Without endorsing area under the ROC curve, we argue that this charge is not substantiated. Three articles in this issue discuss alternative metrics of risk model performance: NRI(p) (two-category net reclassification index at the event rate), integrated discrimination index, and R-squared statistics. Guided by the principle that performance metrics should match the intended use of a risk prediction model, we argue that routine use of these indices is not justified. Instead, we recommend decision-theoretic measures to evaluate risk prediction models for applications in which clinically relevant risk thresholds have been established for classifying individuals. In the absence of established risk thresholds, additional research is needed to develop suitable metrics. Copyright © 2017 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

The change in area under the curve (∆AUC), the integrated discrimination improvement (IDI), and net reclassification index (NRI) are commonly used measures of risk prediction model performance. Some authors have reported good validity of associated methods of estimating their standard errors (SE) and construction of confidence intervals, whereas others have questioned their performance. To address these issues, we unite the ∆AUC, IDI, and three versions of the NRI under the umbrella of the U-statistics family. We rigorously show that the asymptotic behavior of ∆AUC, NRIs, and IDI fits the asymptotic distribution theory developed for U-statistics. We prove that the ∆AUC, NRIs, and IDI are asymptotically normal, unless they compare nested models under the null hypothesis. In the latter case, asymptotic normality and existing SE estimates cannot be applied to ∆AUC, NRIs, or IDI. In the former case, SE formulas proposed in the literature are equivalent to SE formulas obtained from U-statistics theory if we ignore adjustment for estimated parameters. We use Sukhatme-Randles-deWet condition to determine when adjustment for estimated parameters is necessary. We show that adjustment is not necessary for SEs of the ∆AUC and two versions of the NRI when added predictor variables are significant and normally distributed. The SEs of the IDI and three-category NRI should always be adjusted for estimated parameters. These results allow us to define when existing formulas for SE estimates can be used and when resampling methods such as the bootstrap should be used instead when comparing nested models. We also use the U-statistic theory to develop a new SE estimate of ∆AUC. Copyright © 2017 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Recent improvements in sequencing technology have enabled the investigation of so-called missing heritability, and a large number of affected subjects have been sequenced in order to detect significant associations between human diseases and rare variants. However, the cost of genome sequencing is still high, and a statistically powerful strategy for selecting informative subjects would be useful. Therefore, in this report, we propose a new statistical method for selecting cases and controls for sequencing studies based on family history. We assume that disease status is determined by unobserved liability scores. Our method consists of two steps: first, the conditional means of liability are estimated with the liability threshold model given the individual's disease status and those of their relatives. Second, the informative subjects are selected with the estimated conditional means. Our simulation studies showed that statistical power is substantially affected by the subject selection strategy chosen, and power is maximized when affected (unaffected) subjects with high (low) risks are selected as cases (controls). The proposed method was successfully applied to genome-wide association studies for type 2 diabetes, and our analysis results reveal the practical value of the proposed methods. Copyright © 2017 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

We compare the calibration and variability of risk prediction models that were estimated using various approaches for combining information on new predictors, termed 'markers', with parameter information available for other variables from an earlier model, which was estimated from a large data source. We assess the performance of risk prediction models updated based on likelihood ratio (LR) approaches that incorporate dependence between new and old risk factors as well as approaches that assume independence ('naive Bayes' methods). We study the impact of estimating the LR by (i) fitting a single model to cases and non-cases when the distribution of the new markers is in the exponential family or (ii) fitting separate models to cases and non-cases. We also evaluate a new constrained maximum likelihood method. We study updating the risk prediction model when the new data arise from a cohort and extend available methods to accommodate updating when the new data source is a case-control study. To create realistic correlations between predictors, we also based simulations on real data on response to antiviral therapy for hepatitis C. From these studies, we recommend the LR method fit using a single model or constrained maximum likelihood. Copyright © 2016 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

This paper proposes a risk prediction model using semi-varying coefficient multinomial logistic regression. We use a penalized local likelihood method to do the model selection and estimate both functional and constant coefficients in the selected model. The model can be used to improve predictive modelling when non-linear interactions between predictors are present. We conduct a simulation study to assess our method's performance, and the results show that the model selection procedure works well with small average numbers of wrong-selection or missing-selection. We illustrate the use of our method by applying it to classify the patients with early rheumatoid arthritis at baseline into different risk groups in future disease progression. We use a leave-one-out cross-validation method to assess its correct prediction rate and propose a recalibration framework to evaluate how reliable are the predicted risks. Copyright © 2016 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Predicting an individual's risk of experiencing a future clinical outcome is a statistical task with important consequences for both practicing clinicians and public health experts. Modern observational databases such as electronic health records provide an alternative to the longitudinal cohort studies traditionally used to construct risk models, bringing with them both opportunities and challenges. Large sample sizes and detailed covariate histories enable the use of sophisticated machine learning techniques to uncover complex associations and interactions, but observational databases are often 'messy', with high levels of missing data and incomplete patient follow-up. In this paper, we propose an adaptation of the well-known Naive Bayes machine learning approach to time-to-event outcomes subject to censoring. We compare the predictive performance of our method with the Cox proportional hazards model which is commonly used for risk prediction in healthcare populations, and illustrate its application to prediction of cardiovascular risk using an electronic health record dataset from a large Midwest integrated healthcare system. Copyright © 2015 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

In the development of risk prediction models, predictors are often measured with error. In this paper, we investigate the impact of covariate measurement error on risk prediction. We compare the prediction performance using a costly variable measured without error, along with error-free covariates, to that of a model based on an inexpensive surrogate along with the error-free covariates. We consider continuous error-prone covariates with homoscedastic and heteroscedastic errors, and also a discrete misclassified covariate. Prediction performance is evaluated by the area under the receiver operating characteristic curve (AUC), the Brier score (BS), and the ratio of the observed to the expected number of events (calibration). In an extensive numerical study, we show that (i) the prediction model with the error-prone covariate is very well calibrated, even when it is mis-specified; (ii) using the error-prone covariate instead of the true covariate can reduce the AUC and increase the BS dramatically; (iii) adding an auxiliary variable, which is correlated with the error-prone covariate but conditionally independent of the outcome given all covariates in the true model, can improve the AUC and BS substantially. We conclude that reducing measurement error in covariates will improve the ensuing risk prediction, unless the association between the error-free and error-prone covariates is very high. Finally, we demonstrate how a validation study can be used to assess the effect of mismeasured covariates on risk prediction. These concepts are illustrated in a breast cancer risk prediction model developed in the Nurses' Health Study. Copyright © 2015 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

The net reclassification improvement (NRI) has become a popular measure of incremental usefulness of markers added to risk prediction models. However, the expected magnitude of the three-category NRI is not well understood, leading researchers to rely on statistical significance. In this paper, we describe a slight modification to the original definition of the NRI, which weighs each reclassification by the number of categories by which a given individual is reclassified. This modification resolves some recent criticisms of the three-category NRI and at the same time has a minimal impact on its magnitude. Then we show that using this modified definition, the event and nonevent NRIs have simple interpretations as sums of changes in sensitivities and specificities calculated at the risk thresholds. We exploit this relationship to arrive at closed-form solutions for the NRI under normality within the event and nonevent subgroups. We observe that the size of the intermediate risk category and the event rate have limited impact on the magnitude of the NRI. As expected, the NRI increases with the strength of the added marker, and this relationship appears fairly proportional for markers with non-weak net effect size (above 0.25). Furthermore, we conclude that using the NRI as a metric, it is harder to improve models that already perform well. Copyright © 2014 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

It is unclear to what extent the incremental predictive performance of a novel biomarker is impacted by the method used to control for standard predictors. We investigated whether adding a biomarker to a model with a published risk score overestimates its incremental performance as compared to adding it to a multivariable model with individual predictors (or a composite risk score estimated from the sample of interest) and to a null model. We used 1000 simulated datasets (with a range of risk factor distributions and event rates) to compare these methods, using the continuous net reclassification index (NRI), the integrated discrimination index (IDI), and change in the C-statistic as discrimination metrics. The new biomarker was added to the following: null model, model including a published risk score, model including a composite risk score estimated from the sample of interest, and multivariable model with individual predictors. We observed a gradient in the incremental performance of the biomarker, with the null model resulting in the highest predictive performance of the biomarker and the model using individual predictors resulting in the lowest (mean increases in C-statistic between models without and with the biomarker: 0.261, 0.085, 0.030, and 0.031; NRI: 0.767, 0.621, 0.513, and 0.530; IDI: 0.153, 0.093, 0.053 and 0.057, respectively). These findings were supported by the Framingham Study data predicting atrial fibrillation using novel biomarkers. We recommend that authors report the effect of a new biomarker after controlling for standard predictors modeled as individual variables. Copyright © 2014 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Risk prediction models play an important role in prevention and treatment of several diseases. Models that are in clinical use are often refined and improved. In many instances, the most efficient way to improve a successful model is to identify subgroups for which there is a specific biological rationale for improvement and tailor the improved model to individuals in these subgroups, an approach especially in line with personalized medicine. At present, we lack statistical tools to evaluate improvements targeted to specific subgroups. Here, we propose simple tools to fill this gap. First, we extend a recently proposed measure, the Integrated Discrimination Improvement, using a linear model with covariates representing the subgroups. Next, we develop graphical and numerical tools that compare reclassification of two models, focusing only on those subjects for whom the two models reclassify differently. We apply these approaches to BRCAPRO, a genetic risk prediction model for breast and ovarian cancer, using data from MD Anderson Cancer Center. We also conduct a simulation study to investigate properties of the new reclassification measure and compare it with currently used measures. Our results show that the proposed tools can successfully uncover subgroup specific model improvements. Copyright © 2013 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

The importance of developing personalized risk prediction estimates has become increasingly evident in recent years. In general, patient populations may be heterogenous and represent a mixture of different unknown subtypes of disease. When the source of this heterogeneity and resulting subtypes of disease are unknown, accurate prediction of survival may be difficult. However, in certain disease settings, the onset time of an observable short-term event may be highly associated with these unknown subtypes of disease and thus may be useful in predicting long-term survival. One approach to incorporate short-term event information along with baseline markers for the prediction of long-term survival is through a landmark Cox model, which assumes a proportional hazards model for the residual life at a given landmark point. In this paper, we use this modeling framework to develop procedures to assess how a patient's long-term survival trajectory may change over time given good short-term outcome indications along with prognosis on the basis of baseline markers. We first propose time-varying accuracy measures to quantify the predictive performance of landmark prediction rules for residual life and provide resampling-based procedures to make inference about such accuracy measures. Simulation studies show that the proposed procedures perform well in finite samples. Throughout, we illustrate our proposed procedures by using a breast cancer dataset with information on time to metastasis and time to death. In addition to baseline clinical markers available for each patient, a chromosome instability genetic score, denoted by CIN25, is also available for each patient and has been shown to be predictive of survival for various types of cancer. We provide procedures to evaluate the incremental value of CIN25 for the prediction of residual life and examine how the residual life profile changes over time. This allows us to identify an informative landmark point, t0, such that accurate risk predictions of the residual life could be made for patients who survive past t0 without metastasis. Copyright © 2013 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Random forest is a supervised learning method that combines many classification or regression trees for prediction. Here we describe an extension of the random forest method for building event risk prediction models in survival analysis with competing risks. In case of right-censored data, the event status at the prediction horizon is unknown for some subjects. We propose to replace the censored event status by a jackknife pseudo-value, and then to apply an implementation of random forests for uncensored data. Because the pseudo-responses take on values on a continuous scale, the node variance is chosen as split criterion for growing regression trees. In a simulation study, the pseudo split criterion is compared with the Gini split criterion when the latter is applied to the uncensored event status. To investigate the resulting pseudo random forest method for building risk prediction models, we analyze it in a simulation study of predictive performance where we compare it to Cox regression and random survival forest. The method is further illustrated in two real data sets. Copyright © 2013 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Risk prediction procedures can be quite useful for the patient's treatment selection, prevention strategy, or disease management in evidence-based medicine. Often, potentially important new predictors are available in addition to the conventional markers. The question is how to quantify the improvement from the new markers for prediction of the patient's risk in order to aid cost-benefit decisions. The standard method, using the area under the receiver operating characteristic curve, to measure the added value may not be sensitive enough to capture incremental improvements from the new markers. Recently, some novel alternatives to area under the receiver operating characteristic curve, such as integrated discrimination improvement and net reclassification improvement, were proposed. In this paper, we consider a class of measures for evaluating the incremental values of new markers, which includes the preceding two as special cases. We present a unified procedure for making inferences about measures in the class with censored event time data. The large sample properties of our procedures are theoretically justified. We illustrate the new proposal with data from a cancer study to evaluate a new gene score for prediction of the patient's survival. Copyright © 2012 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

When seeking prognostic information for patients, modern technologies provide a huge amount of genomic measurements as a starting point. For single-nucleotide polymorphisms (SNPs), there may be more than one million covariates that need to be simultaneously considered with respect to a clinical endpoint. Although the underlying biological problem cannot be solved on the basis of clinical cohorts of only modest size, some important SNPs might still be identified. Sparse multivariable regression techniques have recently become available for automatically identifying prognostic molecular signatures that comprise relatively few covariates and provide reasonable prediction performance. For illustrating how such approaches can be adapted to the specific features of SNP data, we propose different variants of a componentwise likelihood-based boosting approach. The latter links SNP measurements to a time-to-event endpoint by a regression model that is built up in a large number of steps. The variants allow for strategic choices in dealing with SNPs that differ in variance because of their variation in minor allele frequencies. In addition, we propose a heuristic that allows computationally efficient handling of millions of covariates, thus opening the door for incorporating SNP × treatment interactions. We illustrate this using data from patients with acute myeloid leukemia. We judge the resulting models according to prediction error curves and using resampling data sets. We obtain increased stability by moving interpretation from the SNP to the gene level. By considering these different aspects, we outline a more general strategy for linking SNP measurements to a time-to-event endpoint by means of sparse multivariable regression models. Copyright © 2012 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Discrimination measures for continuous time-to-event outcomes have become an important tool in medical decision making. The idea behind discrimination measures is to evaluate the performance of a prediction model by measuring its ability to distinguish between observations having an event and those having no event. Researchers proposed a variety of approaches to estimate discrimination measures from a set of right-censored data. These approaches rely on different regularity assumptions that are needed to ensure consistency of the respective estimators. Typical examples of regularity assumptions include the proportional hazards assumption in Cox regression and the random censoring assumption. Because regularity assumptions are often violated in practice, conducting a sensitivity analysis of the estimators is of considerable interest. The aim of the paper is to analyze and to compare the most popular estimators of discrimination measures for event time outcomes. On the basis of the results of an extensive simulation study and the analysis of molecular data, we investigate the behavior of the estimators in situations where the underlying regularity assumptions do not hold. We show that violations of the regularity assumptions may induce a nonignorable bias and may therefore result in biased medical decision making. Copyright © 2012 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

The area under the receiver operating characteristics curve (AUC of ROC) is a widely used measure of discrimination in risk prediction models. Routinely, the Mann-Whitney statistics is used as an estimator of AUC, while the change in AUC is tested by the DeLong test. However, very often, in settings where the model is developed and tested on the same dataset, the added predictor is statistically significantly associated with the outcome but fails to produce a significant improvement in the AUC. No conclusive resolution exists to explain this finding. In this paper, we will show that the reason lies in the inappropriate application of the DeLong test in the setting of nested models. Using numerical simulations and a theoretical argument based on generalized U-statistics, we show that if the added predictor is not statistically significantly associated with the outcome, the null distribution is non-normal, contrary to the assumption of DeLong test. Our simulations of different scenarios show that the loss of power because of such a misuse of the DeLong test leads to a conservative test for small and moderate effect sizes. This problem does not exist in cases of predictors that are associated with the outcome and for non-nested models. We suggest that for nested models, only the test of association be performed for the new predictors, and if the result is significant, change in AUC be estimated with an appropriate confidence interval, which can be based on the DeLong approach. Copyright © 2012 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Prognostic models for survival outcomes are often developed by fitting standard survival regression models, such as the Cox proportional hazards model, to representative datasets. However, these models can be unreliable if the datasets contain few events, which may be the case if either the disease or the event of interest is rare. Specific problems include predictions that are too extreme, and poor discrimination between low-risk and high-risk patients. The objective of this paper is to evaluate three existing penalised methods that have been proposed to improve predictive accuracy. In particular, ridge, lasso and the garotte, which use penalised maximum likelihood to shrink coefficient estimates and in some cases omit predictors entirely, are assessed using simulated data derived from two clinical datasets. The predictions obtained using these methods are compared with those from Cox models fitted using standard maximum likelihood. The simulation results suggest that Cox models fitted using maximum likelihood can perform poorly when there are few events, and that significant improvements are possible by taking a penalised modelling approach. The ridge method generally performed the best, although lasso is recommended if variable selection is required. Copyright © 2011 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Shared random parameter models (SRPMs) were first introduced by researchers at the National Heart Lung and Blood Institute (NHLBI) Biostatistics Branch for analyzing longitudinal data with informative dropout (Wu and Carroll, 1987; Wu and Bailey, 1988; Follmann and Wu, 1995; Albert and Follmann, 2000; Albert et al, 2002). This work was all focused on characterizing the longitudinal data process in the presence of an informative missing data mechanism that is treated as a nuisance. Shared random parameter modeling approaches have also been developed from the perspective of characterizing the relationship between longitudinal data and a subsequent outcome that may be an event time, a dichotomous measurement, or another longitudinal outcome. This article will review the early contributions of the NHLBI biostatisticians on SRPMs for analyzing longitudinal data with dropout and demonstrate how these ideas have, more recently, been applied in these other areas of biostatistics. Rather than focus on technical details or specific analyses, this article presents a conceptual framework for SRPMs within a historical context. [ABSTRACT FROM AUTHOR]