Search

Cross-validation is a widely used technique to estimate prediction error, but its behavior is complex and not fully understood. Ideally, one would like to think that cross-validation estimates the prediction error for the model at hand, fit to the training data. We prove that this is not the case for the linear model fit by ordinary least squares; rather it estimates the average prediction error of models fit on other unseen training sets drawn from the same population. We further show that this phenomenon occurs for most popular estimates of prediction error, including data splitting, bootstrapping, and Mallow's C p . Next, the standard confidence intervals for prediction error derived from cross-validation may have coverage far below the desired level. Because each data point is used for both training and testing, there are correlations among the measured accuracies for each fold, and so the usual estimate of variance is too small. We introduce a nested cross-validation scheme to estimate this variance more accurately, and show empirically that this modification leads to intervals with approximately correct coverage in many examples where traditional cross-validation intervals fail. Lastly, our analysis also shows that when producing confidence intervals for prediction accuracy with simple data splitting, one should not refit the model on the combined data, since this invalidates the confidence intervals. for this article are available online. [ABSTRACT FROM AUTHOR]

We consider the problem of performing interpretable classification in the high-dimensional setting, in which the number of features is very large and the number of observations is limited. This setting has been studied extensively in the chemometrics literature, and more recently has become commonplace in biological and medical applications. In this setting, a traditional approach involves performing feature selection before classification. We propose sparse discriminant analysis, a method for performing linear discriminant analysis with a sparseness criterion imposed such that classification and feature selection are performed simultaneously. Sparse discriminant analysis is based on the optimal scoring interpretation of linear discriminant analysis, and can be extended to perform sparse discrimination via mixtures of Gaussians if boundaries between classes are nonlinear or if subgroups are present within each class. Our proposal also provides low-dimensional views of the discriminative directions. [ABSTRACT FROM AUTHOR]

Professor Efron has presented us with a thought-provoking paper on the relationship between prediction, estimation, and attribution in the modern era of data science. While we appreciate many of his arguments, we see more of a continuum between the old and new methodology, and the opportunity for both to improve through their synergy. [ABSTRACT FROM AUTHOR]

Principal component analysis (PCA) is widely used in data processing and dimensionality reduction. However, PCA suffers from the fact that each principal component is a linear combination of all the original variables, thus it is often difficult to interpret the results. We introduce a new method called sparse principal component analysis (SPCA) using the lasso (elastic net) to produce modified principal components with sparse loadings. We first show that PCA can be formulated as a regression-type optimization problem; sparse loadings are then obtained by imposing the lasso (elastic net) constraint on the regression coefficients. Efficient algorithms are proposed to fit our SPCA models for both regular multivariate data and gene expression arrays. We also give a new formula to compute the total variance of modified principal components. As illustrations, SPCA is applied to real and simulated data with encouraging results. [ABSTRACT FROM AUTHOR]

In regression problems where the number of predictors greatly exceeds the number of observations, conventional regression techniques may produce unsatisfactory results. We describe a technique called supervised principal components that can be applied to this type of problem. Supervised principal components is similar to conventional principal components analysis except that it uses a subset of the predictors selected based on their association with the outcome. Supervised principal components can be applied to regression and generalized regression problems, such as survival analysis. It compares favorably to other techniques for this type of problem, and can also account for the effects of other covariates and help identify which predictor variables are most important. We also provide asymptotic consistency results to help support our empirical findings. These methods could become important tools for DNA microarray data, where they may be used to more accurately diagnose and treat cancer. [ABSTRACT FROM AUTHOR]

The support vector machine (SVM) is known for its good performance in two-class classification, but its extension to multiclass classification is still an ongoing research is- sue. In this article, we propose a new approach for classification, called the import vector machine (IVM), which is built on kernel logistic regression (KLR). We show that the IVM not only performs as well as the SVM in two-class classification, but also can naturally be generalized to the multiclass case. Furthermore, the IVM provides an estimate of the underlying probability. Similar to the support points of the SVM, the IVM model uses only a fraction of the training data to index kernel basis functions, typically a much smaller fraction than the SVM. This gives the IVM a potential computational advantage over the SVM. [ABSTRACT FROM AUTHOR]

Develops a general method for finding important discriminant directions without assuming the class densities belong to any particular parametric family using a likelihood-based interpretation of linear discriminant analysis criterion. Importance of discriminant directions; Existence of local maxima; Misclassification errors for different one-dimensional Bayes rules.

We propose statistical measures for finding masses in mammograms. The measures are based on fitting broken line regressions to local intensity plots of the images. The method is illustrated on a small database of mammograms that have been read by a radiologist and confirmed by operative data. This work illustrates some of the statistical challenges in working with large diagnostic images, [ABSTRACT FROM AUTHOR]

Provides information on plug-in classification technique (PICT), a method that takes a standard classifier and plugs it into an algorithm to produce a new classifier. Suggested PICT for solving class classification problems using binary classifiers; Discussion on coding matrix; Functions of ECOC PICT.

Presents a reaction to the paper by statisticians Ildiko E. Frank and Jerome H. Friedman regarding two methods used in chemometrics for predictive modeling, partial least squares and principal components regression, from a statistical perspective. Penalized least squares; Smooth basis expansions.

Comments on the article about fitting regression models, smoothing and additive. Difficulty of the task of selecting the number and position of the 'knots' for a regression spline; Common method for fitting nonparametric additive models; Noise examples showing Turbo to be far superior to the alternative methods.

Examines the link between least measures and maximum likelihood estimation. Use of Prediction error and Kulback-Leibler (KL) distance; Reference to the deviance function popular in a generalized linear models literature; Details on the pythagorean property of both KL distance and squared root.

Discusses an alternative algorithm in which the smooth functions are estimated using smoothing spline. The generalized cross-validation (GSV) based techniques used to select the smoothing parameters and number of terms; Structure of the automatic smoothing spline projection pursuit (ASP) algorithm; Examples comparing ASP with the standard supersmoother-based PPR algorithm of Friedman.

Fisher's linear discriminant analysis is a valuable tool for multigroup classification With a large number of predictors, one can find a reduced number of discriminant coordinate functions that are "optimal' for separating the groups With two such functions, one can produce a classification map that partitions the reduced space into regions that are identified with group membership, and the decision boundaries are linear This article is about richer nonlinear classification schemes Linear discriminant analysis is equivalent to multiresponse linear regression using optimal scorings to represent the groups In this paper, we obtain nonparametnc versions of discriminant analysis by replacing linear regression by any nonparameinc regression method In this way, any multiresponse regression technique (such as MARS or neural networks) can be postprocessed to improve its classification performance. [ABSTRACT FROM AUTHOR]

Principal curves are smooth one-dimensional curves that pass through the middle of a p-dimensional data set, providing a nonlinear summary of the data. They are nonparametric, and their shape is suggested by the data. The algorithm for constructing principal curves starts with some prior summary, such as the usual principal-component line. The curve in each successive iteration is a smooth or local average of the p-dimensional points, where the definition of local is based on the distance in arc length of the projections of the points onto the curve found in the previous iteration. In this article principal curves are defined, an algorithm for their construction is given, some theoretical results arc presented, and the procedure is compared to other generalizations of principal components. Two applications illustrate the use of principal curves. The first describes how the principal-curve procedure was used to align the magnets of the Stanford linear collider. The collider uses about 950 magnets in a roughly circular arrangement to bend electron and positron beams and bring them to collision. After construction, it was found that some of the magnets had ended up significantly out of place. As a result, the beams had to be bent too sharply and could not be focused. The engineers realized that the magnets did not have to be moved to their originally planned locations, but rather to a sufficiently smooth are through the middle of the existing positions. This arc was found using the principal curve procedure. In the second application, two different assays for gold content in several samples of computer chip waste appear to show some systematic differences that are blurred by measurement error. The classical approach using linear errors in variables regression can detect systematic linear differences but is not able to account for nonlinearities. When the first linear principal component is replaced with a... [ABSTRACT FROM AUTHOR]

A scatterplot smoother is applied to data of the form {(x[sub 1], y[sub 1]), (x[sub 2], y[sub 2]),..., (x[sub n], y[sub n])} and uses local fitting to estimate the dependence of Y on X. A simple example is the running lines smoother, which fits a least squares line to the y values falling in a window around each x value. The value of the estimated function at x is given by the value of the least squares line at x. A smoother generalizes the least squares line, which assumes that the dependence of Y on X is linear. In this article, we extend the idea of local fitting to likelihood-based regression models. One such application is to the class of generalized linear models (Nelder and Wedderburn 1972). We enlarge this class by replacing the covariate form beta[sub 0] + x beta[sub 1] with an unspecified smooth function s(x). This function is estimated from the data by a technique we call local likelihood estimation. The method consists of maximum likelihood estimation for beta[sub 0] and beta[sub 1], applied in a window around each x value. Multiple covariates are incorporated through an iterative algorithm. We also apply the local likelihood technique to the proportional hazards model of Cox (1972), a model for censored data. The proportional hazards assumption lambda(t | x) = lambda[sub 0](t)exp(x beta) is replaced by lambda(t | x) = lambda[sub 0](t)exp(s(x)), and the function s(x) is estimated from the data by the local likelihood method. In some real data examples, the local likelihood technique proves to be effective in uncovering nonlinear dependencies. It is useful as a descriptive tool or to suggest transformations of the covariates. We also discuss some methods for inference. [ABSTRACT FROM AUTHOR]

Correspondence analysis is an exploratory multivariate technique that converts a data matrix into a particular type of graphical display in which the rows and columns are depicted as points. The method has a long and varied history and has appeared in different forms in the psychometric and ecological literature, among others. In this article we review the geometry of correspondence analysis and its geometric interpretation. We also discuss various extensions of correspondence analysis to multivariate categorical data (multiple correspondence analysis) and a variety of other data types. [ABSTRACT FROM AUTHOR]

Generalized additive models have the form eta (x) = alpha + SIGMA f[sub j](x[sub j]), where eta might be the regression function in a multiple regression or the logistic transformation of the posterior probability Pr(y = 1 | x) in a logistic regression. In fact, these models generalize the whole family of generalized linear models eta(x) = beta'x, where eta(x) = g(mu(x)) is some transformation of the regression function. We use the local scoring algorithm to estimate the functions f[sub j](x[sub j]) nonparametrically, using a scatterplot smoother as a building block. We demonstrate the models in two different analyses: a nonparametric analysis of covariance and a logistic regression. The procedure can be used as a diagnostic tool for identifying parametric transformations of the covariates in a standard linear analysis. A variety of inferential tools have been developed to aid the analyst in assessing the relevance and significance of the estimated functions: these include confidence curves, degrees of freedom estimates, and approximate hypothesis tests. The local scoring algorithm is analogous to the iterative reweighted least squares algorithm for solving likelihood and nonlinear regression equations. At each iteration, an adjusted dependent variable is formed and an additive regression model is fit using the backfitting algorithm. The backfitting algorithm cycles through the variables and estimates each coordinated function by smoothing the partial residuals. [ABSTRACT FROM AUTHOR]

A letter to the editor is presented in response to the article by Ian M. Johnstone and Arthur Yu Lu on the contrast between smooth and local variation in functional data in the previous issue.

Comments on Leo Breiman's article `The II Method for Estimating Multivariate Functions from Noisy Data.' Applaud on Breiman for providing detailed benchmarks as a basis for comparison.

In 2002, methodological issues around time series analyses of air pollution and health attracted the attention of the scientific community, policy makers, the press, and the diverse stakeholders concerned with air pollution. As the U.S. Environmental Protection Agency (EPA) was finalizing its most recent review of epidemiologic evidence on particulate matter air pollution (PM). statisticians and epidemiologists found that the S-PLUS implementation of generalized additive models (GAMs) can overestimate effects of air pollution and understate statistical uncertainty in time series studies of air pollution and health. This discovery delayed completion of the PM Criteria Document prepared as part of the review of the U.S. National Ambient Air Quality Standard, because the time series findings represented a critical component of the evidence. In addition, it raised concerns about the adequacy of current model formulations and their software implementations. In this article we provide improvements in semiparametric regression directly relevant to risk estimation in time series studies of air pollution. First, we introduce a closed-form estimate of the asymptotically exact covariance matrix of the linear component of a GAM. To ease the implementation of these calculations, we develop the S package gam.exact, an extended version of gam. Use of gam.exact allows a more robust assessment of the statistical uncertainty of the estimated pollution coefficients. Second, we develop a bandwidth selection method to reduce confounding bias in the pollution-mortality relationship due to unmeasured time-varying factors, such as season and influenza epidemics. Third, we introduce a conceptual framework lo fully explore the sensitivity of the air pollution risk estimates to model choice. We apply our methods to data of the National Mortality Morbidity Air Pollution Study, which includes lime series data from the 90 largest U.S. cities for the period 1987-1994. [ABSTRACT FROM AUTHOR]