Search

Cross-validation is a widely used technique to estimate prediction error, but its behavior is complex and not fully understood. Ideally, one would like to think that cross-validation estimates the prediction error for the model at hand, fit to the training data. We prove that this is not the case for the linear model fit by ordinary least squares; rather it estimates the average prediction error of models fit on other unseen training sets drawn from the same population. We further show that this phenomenon occurs for most popular estimates of prediction error, including data splitting, bootstrapping, and Mallow's C p . Next, the standard confidence intervals for prediction error derived from cross-validation may have coverage far below the desired level. Because each data point is used for both training and testing, there are correlations among the measured accuracies for each fold, and so the usual estimate of variance is too small. We introduce a nested cross-validation scheme to estimate this variance more accurately, and show empirically that this modification leads to intervals with approximately correct coverage in many examples where traditional cross-validation intervals fail. Lastly, our analysis also shows that when producing confidence intervals for prediction accuracy with simple data splitting, one should not refit the model on the combined data, since this invalidates the confidence intervals. for this article are available online. [ABSTRACT FROM AUTHOR]

Examines the link between least measures and maximum likelihood estimation. Use of Prediction error and Kulback-Leibler (KL) distance; Reference to the deviance function popular in a generalized linear models literature; Details on the pythagorean property of both KL distance and squared root.

A scatterplot smoother is applied to data of the form {(x[sub 1], y[sub 1]), (x[sub 2], y[sub 2]),..., (x[sub n], y[sub n])} and uses local fitting to estimate the dependence of Y on X. A simple example is the running lines smoother, which fits a least squares line to the y values falling in a window around each x value. The value of the estimated function at x is given by the value of the least squares line at x. A smoother generalizes the least squares line, which assumes that the dependence of Y on X is linear. In this article, we extend the idea of local fitting to likelihood-based regression models. One such application is to the class of generalized linear models (Nelder and Wedderburn 1972). We enlarge this class by replacing the covariate form beta[sub 0] + x beta[sub 1] with an unspecified smooth function s(x). This function is estimated from the data by a technique we call local likelihood estimation. The method consists of maximum likelihood estimation for beta[sub 0] and beta[sub 1], applied in a window around each x value. Multiple covariates are incorporated through an iterative algorithm. We also apply the local likelihood technique to the proportional hazards model of Cox (1972), a model for censored data. The proportional hazards assumption lambda(t | x) = lambda[sub 0](t)exp(x beta) is replaced by lambda(t | x) = lambda[sub 0](t)exp(s(x)), and the function s(x) is estimated from the data by the local likelihood method. In some real data examples, the local likelihood technique proves to be effective in uncovering nonlinear dependencies. It is useful as a descriptive tool or to suggest transformations of the covariates. We also discuss some methods for inference. [ABSTRACT FROM AUTHOR]