Search

The efficiency of deep convolutional neural networks (DCNNs) has been demonstrated empirically in many practical applications. In this paper, we establish a theory for approximating functions from Korobov spaces by DCNNs. It verifies rigorously the efficiency of DCNNs in approximating functions of many variables with some variable structures and their abilities in overcoming the curse of dimensionality. [ABSTRACT FROM AUTHOR]

In this paper, we study the online learning algorithm without explicit regularization terms. This algorithm is essentially a stochastic gradient descent scheme in a reproducing kernel Hilbert space (RKHS). The polynomially decaying step size in each iteration can play a role of regularization to ensure the generalization ability of online learning algorithm. We develop a novel capacity dependent analysis on the performance of the last iterate of online learning algorithm. This answers an open problem in learning theory. The contribution of this paper is twofold. First, our novel capacity dependent analysis can lead to sharp convergence rate in the standard mean square distance which improves the results in the literature. Second, we establish, for the first time, the strong convergence of the last iterate with polynomially decaying step sizes in the RKHS norm. We demonstrate that the theoretical analysis established in this paper fully exploits the fine structure of the underlying RKHS, and thus can lead to sharp error estimates of online learning algorithm. [ABSTRACT FROM AUTHOR]

The previous results describing the generalization ability of Empirical Risk Minimization (ERM) algorithm are usually based on the assumption of independent and identically distributed (i.i.d.) samples. In this paper we go far beyond this classical framework by establishing the first exponential bound on the rate of uniform convergence of the ERM algorithm with V-geometrically ergodic Markov chain samples, as the application of the bound on the rate of uniform convergence, we also obtain the generalization bounds of the ERM algorithm with V-geometrically ergodic Markov chain samples and prove that the ERM algorithm with V-geometrically ergodic Markov chain samples is consistent. The main results obtained in this paper extend the previously known results of i.i.d. observations to the case of V-geometrically ergodic Markov chain samples. [ABSTRACT FROM AUTHOR]

We study the problem of learning a sparse linear regression vector under additional conditions on the structure of its sparsity pattern. This problem is relevant in machine learning, statistics and signal processing. It is well known that a linear regression can benefit from knowledge that the underlying regression vector is sparse. The combinatorial problem of selecting the nonzero components of this vector can be 'relaxed' by regularizing the squared error with a convex penalty function like the ℓ norm. However, in many applications, additional conditions on the structure of the regression vector and its sparsity pattern are available. Incorporating this information into the learning method may lead to a significant decrease of the estimation error. In this paper, we present a family of convex penalty functions, which encode prior knowledge on the structure of the vector formed by the absolute values of the regression coefficients. This family subsumes the ℓ norm and is flexible enough to include different models of sparsity patterns, which are of practical and theoretical importance. We establish the basic properties of these penalty functions and discuss some examples where they can be computed explicitly. Moreover, we present a convergent optimization algorithm for solving regularized least squares with these penalty functions. Numerical simulations highlight the benefit of structured sparsity and the advantage offered by our approach over the Lasso method and other related methods. [ABSTRACT FROM AUTHOR]

Sampling inequalities give a precise formulation of the fact that a differentiable function cannot attain large values if its derivatives are bounded and if it is small on a sufficiently dense discrete set. Sampling inequalities can be applied to the difference of a function and its reconstruction in order to obtain (sometimes optimal) convergence orders for very general possibly regularized recovery processes. So far, there are only sampling inequalities for finitely smooth functions, which lead to algebraic convergence orders. In this paper, the case of infinitely smooth functions is investigated, in order to derive error estimates with exponential convergence orders. [ABSTRACT FROM AUTHOR]