Search

A dynamical model consists of a continuous self-map $T:\mathcal{X}\to \mathcal{X}$ of a compact state space $\mathcal{X}$ and a continuous observation function $f:\mathcal{X}\to \mathbb{R}$. This paper considers the fitting of a parametrized family of dynamical models to an observed real-valued stochastic process using empirical risk minimization. The limiting behavior of the minimum risk parameters is studied in a general setting. We establish a general convergence theorem for minimum risk estimators and ergodic observations. We then study conditions under which empirical risk minimization can effectively separate signal from noise in an additive observational noise model. The key condition in the latter results is that the family of dynamical models has limited complexity, which is quantified through a notion of entropy for families of infinite sequences that connects covering number based entropies with topological entropy studied in dynamical systems. We establish close connections between entropy and limiting average mean widths for stationary processes, and discuss several examples of dynamical models.

Many results have been proved for various nuclear norm penalized estimators of the uniform sampling matrix completion problem. However, most of these estimators are not robust: in most of the cases the quadratic loss function and its modifications are used. We consider robust nuclear norm penalized estimators using two well-known robust loss functions: the absolute value loss and the Huber loss. Under several conditions on the sparsity of the problem (i.e. the rank of the parameter matrix) and on the regularity of the risk function sharp and non-sharp oracle inequalities for these estimators are shown to hold with high probability. As a consequence, the asymptotic behavior of the estimators is derived. Similar error bounds are obtained under the assumption of weak sparsity, i.e. the case where the matrix is assumed to be only approximately low-rank. In all our results we consider a high-dimensional setting. In this case, this means that we assume $n\leq pq$. Finally, various simulations confirm our theoretical results., Comment: 45 pages, 4 figures

Most high-dimensional estimation methods propose to minimize a cost function (empirical risk) that is a sum of losses associated to each data point (each example). In this paper, we focus on the case of nonconvex losses. Classical empirical process theory implies uniform convergence of the empirical (or sample) risk to the population risk. While under additional assumptions, uniform convergence implies consistency of the resulting M-estimator, it does not ensure that the latter can be computed efficiently. ¶ In order to capture the complexity of computing M-estimators, we study the landscape of the empirical risk, namely its stationary points and their properties. We establish uniform convergence of the gradient and Hessian of the empirical risk to their population counterparts, as soon as the number of samples becomes larger than the number of unknown parameters (modulo logarithmic factors). Consequently, good properties of the population risk can be carried to the empirical risk, and we are able to establish one-to-one correspondence of their stationary points. We demonstrate that in several problems such as nonconvex binary classification, robust regression and Gaussian mixture model, this result implies a complete characterization of the landscape of the empirical risk, and of the convergence properties of descent algorithms. ¶ We extend our analysis to the very high-dimensional setting in which the number of parameters exceeds the number of samples, and provides a characterization of the empirical risk landscape under a nearly information-theoretically minimal condition. Namely, if the number of samples exceeds the sparsity of the parameters vector (modulo logarithmic factors), then a suitable uniform convergence result holds. We apply this result to nonconvex binary classification and robust regression in very high-dimension.

The problem of multiple kernel learning based on penalized empirical risk minimization is discussed. The complexity penalty is determined jointly by the empirical $L_2$ norms and the reproducing kernel Hilbert space (RKHS) norms induced by the kernels with a data-driven choice of regularization parameters. The main focus is on the case when the total number of kernels is large, but only a relatively small number of them is needed to represent the target function, so that the problem is sparse. The goal is to establish oracle inequalities for the excess risk of the resulting prediction rule showing that the method is adaptive both to the unknown design distribution and to the sparsity of the problem., Published in at http://dx.doi.org/10.1214/10-AOS825 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

This paper deals with recovering an unknown vector $\theta$ from the noisy data $Y=A\theta+\sigma\xi$, where $A$ is a known $(m\times n)$-matrix and $\xi$ is a white Gaussian noise. It is assumed that $n$ is large and $A$ may be severely ill-posed. Therefore, in order to estimate $\theta$, a spectral regularization method is used, and our goal is to choose its regularization parameter with the help of the data $Y$. For spectral regularization methods related to the so-called ordered smoothers [see Kneip Ann. Statist. 22 (1994) 835--866], we propose new penalties in the principle of empirical risk minimization. The heuristical idea behind these penalties is related to balancing excess risks. Based on this approach, we derive a sharp oracle inequality controlling the mean square risks of data-driven spectral regularization methods., Comment: Published in at http://dx.doi.org/10.1214/10-AOS803 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

We study estimation of a multivariate function $f:\mathbf{R}^d\to\mathbf{R}$ when the observations are available from the function $Af$, where $A$ is a known linear operator. Both the Gaussian white noise model and density estimation are studied. We define an $L_2$-empirical risk functional which is used to define a $\delta$-net minimizer and a dense empirical risk minimizer. Upper bounds for the mean integrated squared error of the estimators are given. The upper bounds show how the difficulty of the estimation depends on the operator through the norm of the adjoint of the inverse of the operator and on the underlying function class through the entropy of the class. Corresponding lower bounds are also derived. As examples, we consider convolution operators and the Radon transform. In these examples, the estimators achieve the optimal rates of convergence. Furthermore, a new type of oracle inequality is given for inverse problems in additive models., Comment: Published in at http://dx.doi.org/10.1214/09-AOS726 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)