Search

Cross-validation is a widely used technique to estimate prediction error, but its behavior is complex and not fully understood. Ideally, one would like to think that cross-validation estimates the prediction error for the model at hand, fit to the training data. We prove that this is not the case for the linear model fit by ordinary least squares; rather it estimates the average prediction error of models fit on other unseen training sets drawn from the same population. We further show that this phenomenon occurs for most popular estimates of prediction error, including data splitting, bootstrapping, and Mallow's C p . Next, the standard confidence intervals for prediction error derived from cross-validation may have coverage far below the desired level. Because each data point is used for both training and testing, there are correlations among the measured accuracies for each fold, and so the usual estimate of variance is too small. We introduce a nested cross-validation scheme to estimate this variance more accurately, and show empirically that this modification leads to intervals with approximately correct coverage in many examples where traditional cross-validation intervals fail. Lastly, our analysis also shows that when producing confidence intervals for prediction accuracy with simple data splitting, one should not refit the model on the combined data, since this invalidates the confidence intervals. for this article are available online. [ABSTRACT FROM AUTHOR]

Professor Efron has presented us with a thought-provoking paper on the relationship between prediction, estimation, and attribution in the modern era of data science. While we appreciate many of his arguments, we see more of a continuum between the old and new methodology, and the opportunity for both to improve through their synergy. [ABSTRACT FROM AUTHOR]

In regression problems where the number of predictors greatly exceeds the number of observations, conventional regression techniques may produce unsatisfactory results. We describe a technique called supervised principal components that can be applied to this type of problem. Supervised principal components is similar to conventional principal components analysis except that it uses a subset of the predictors selected based on their association with the outcome. Supervised principal components can be applied to regression and generalized regression problems, such as survival analysis. It compares favorably to other techniques for this type of problem, and can also account for the effects of other covariates and help identify which predictor variables are most important. We also provide asymptotic consistency results to help support our empirical findings. These methods could become important tools for DNA microarray data, where they may be used to more accurately diagnose and treat cancer. [ABSTRACT FROM AUTHOR]

Fisher's linear discriminant analysis is a valuable tool for multigroup classification With a large number of predictors, one can find a reduced number of discriminant coordinate functions that are "optimal' for separating the groups With two such functions, one can produce a classification map that partitions the reduced space into regions that are identified with group membership, and the decision boundaries are linear This article is about richer nonlinear classification schemes Linear discriminant analysis is equivalent to multiresponse linear regression using optimal scorings to represent the groups In this paper, we obtain nonparametnc versions of discriminant analysis by replacing linear regression by any nonparameinc regression method In this way, any multiresponse regression technique (such as MARS or neural networks) can be postprocessed to improve its classification performance. [ABSTRACT FROM AUTHOR]

Principal curves are smooth one-dimensional curves that pass through the middle of a p-dimensional data set, providing a nonlinear summary of the data. They are nonparametric, and their shape is suggested by the data. The algorithm for constructing principal curves starts with some prior summary, such as the usual principal-component line. The curve in each successive iteration is a smooth or local average of the p-dimensional points, where the definition of local is based on the distance in arc length of the projections of the points onto the curve found in the previous iteration. In this article principal curves are defined, an algorithm for their construction is given, some theoretical results arc presented, and the procedure is compared to other generalizations of principal components. Two applications illustrate the use of principal curves. The first describes how the principal-curve procedure was used to align the magnets of the Stanford linear collider. The collider uses about 950 magnets in a roughly circular arrangement to bend electron and positron beams and bring them to collision. After construction, it was found that some of the magnets had ended up significantly out of place. As a result, the beams had to be bent too sharply and could not be focused. The engineers realized that the magnets did not have to be moved to their originally planned locations, but rather to a sufficiently smooth are through the middle of the existing positions. This arc was found using the principal curve procedure. In the second application, two different assays for gold content in several samples of computer chip waste appear to show some systematic differences that are blurred by measurement error. The classical approach using linear errors in variables regression can detect systematic linear differences but is not able to account for nonlinearities. When the first linear principal component is replaced with a... [ABSTRACT FROM AUTHOR]

A scatterplot smoother is applied to data of the form {(x[sub 1], y[sub 1]), (x[sub 2], y[sub 2]),..., (x[sub n], y[sub n])} and uses local fitting to estimate the dependence of Y on X. A simple example is the running lines smoother, which fits a least squares line to the y values falling in a window around each x value. The value of the estimated function at x is given by the value of the least squares line at x. A smoother generalizes the least squares line, which assumes that the dependence of Y on X is linear. In this article, we extend the idea of local fitting to likelihood-based regression models. One such application is to the class of generalized linear models (Nelder and Wedderburn 1972). We enlarge this class by replacing the covariate form beta[sub 0] + x beta[sub 1] with an unspecified smooth function s(x). This function is estimated from the data by a technique we call local likelihood estimation. The method consists of maximum likelihood estimation for beta[sub 0] and beta[sub 1], applied in a window around each x value. Multiple covariates are incorporated through an iterative algorithm. We also apply the local likelihood technique to the proportional hazards model of Cox (1972), a model for censored data. The proportional hazards assumption lambda(t | x) = lambda[sub 0](t)exp(x beta) is replaced by lambda(t | x) = lambda[sub 0](t)exp(s(x)), and the function s(x) is estimated from the data by the local likelihood method. In some real data examples, the local likelihood technique proves to be effective in uncovering nonlinear dependencies. It is useful as a descriptive tool or to suggest transformations of the covariates. We also discuss some methods for inference. [ABSTRACT FROM AUTHOR]

Correspondence analysis is an exploratory multivariate technique that converts a data matrix into a particular type of graphical display in which the rows and columns are depicted as points. The method has a long and varied history and has appeared in different forms in the psychometric and ecological literature, among others. In this article we review the geometry of correspondence analysis and its geometric interpretation. We also discuss various extensions of correspondence analysis to multivariate categorical data (multiple correspondence analysis) and a variety of other data types. [ABSTRACT FROM AUTHOR]

Generalized additive models have the form eta (x) = alpha + SIGMA f[sub j](x[sub j]), where eta might be the regression function in a multiple regression or the logistic transformation of the posterior probability Pr(y = 1 | x) in a logistic regression. In fact, these models generalize the whole family of generalized linear models eta(x) = beta'x, where eta(x) = g(mu(x)) is some transformation of the regression function. We use the local scoring algorithm to estimate the functions f[sub j](x[sub j]) nonparametrically, using a scatterplot smoother as a building block. We demonstrate the models in two different analyses: a nonparametric analysis of covariance and a logistic regression. The procedure can be used as a diagnostic tool for identifying parametric transformations of the covariates in a standard linear analysis. A variety of inferential tools have been developed to aid the analyst in assessing the relevance and significance of the estimated functions: these include confidence curves, degrees of freedom estimates, and approximate hypothesis tests. The local scoring algorithm is analogous to the iterative reweighted least squares algorithm for solving likelihood and nonlinear regression equations. At each iteration, an adjusted dependent variable is formed and an additive regression model is fit using the backfitting algorithm. The backfitting algorithm cycles through the variables and estimates each coordinated function by smoothing the partial residuals. [ABSTRACT FROM AUTHOR]

A letter to the editor is presented in response to the article by Ian M. Johnstone and Arthur Yu Lu on the contrast between smooth and local variation in functional data in the previous issue.

In 2002, methodological issues around time series analyses of air pollution and health attracted the attention of the scientific community, policy makers, the press, and the diverse stakeholders concerned with air pollution. As the U.S. Environmental Protection Agency (EPA) was finalizing its most recent review of epidemiologic evidence on particulate matter air pollution (PM). statisticians and epidemiologists found that the S-PLUS implementation of generalized additive models (GAMs) can overestimate effects of air pollution and understate statistical uncertainty in time series studies of air pollution and health. This discovery delayed completion of the PM Criteria Document prepared as part of the review of the U.S. National Ambient Air Quality Standard, because the time series findings represented a critical component of the evidence. In addition, it raised concerns about the adequacy of current model formulations and their software implementations. In this article we provide improvements in semiparametric regression directly relevant to risk estimation in time series studies of air pollution. First, we introduce a closed-form estimate of the asymptotically exact covariance matrix of the linear component of a GAM. To ease the implementation of these calculations, we develop the S package gam.exact, an extended version of gam. Use of gam.exact allows a more robust assessment of the statistical uncertainty of the estimated pollution coefficients. Second, we develop a bandwidth selection method to reduce confounding bias in the pollution-mortality relationship due to unmeasured time-varying factors, such as season and influenza epidemics. Third, we introduce a conceptual framework lo fully explore the sensitivity of the air pollution risk estimates to model choice. We apply our methods to data of the National Mortality Morbidity Air Pollution Study, which includes lime series data from the 90 largest U.S. cities for the period 1987-1994. [ABSTRACT FROM AUTHOR]