Search

In the context of finite sums minimization, variance reduction techniques are widely used to improve the performance of state-of-the-art stochastic gradient methods. Their practical impact is clear, as well as their theoretical properties. Stochastic proximal point algorithms have been studied as an alternative to stochastic gradient algorithms since they are more stable with respect to the choice of the step size. However, their variance-reduced versions are not as well studied as the gradient ones. In this work, we propose the first unified study of variance reduction techniques for stochastic proximal point algorithms. We introduce a generic stochastic proximal-based algorithm that can be specified to give the proximal version of SVRG, SAGA, and some of their variants. For this algorithm, in the smooth setting, we provide several convergence rates for the iterates and the objective function values, which are faster than those of the vanilla stochastic proximal point algorithm. More specifically, for convex functions, we prove a sublinear convergence rate of O(1/k). In addition, under the Polyak-łojasiewicz condition, we obtain linear convergence rates. Finally, our numerical experiments demonstrate the advantages of the proximal variance reduction methods over their gradient counterparts in terms of the stability with respect to the choice of the step size in most cases, especially for difficult problems. [ABSTRACT FROM AUTHOR]

Machine learning (ML) is evolving rapidly and has made many theoretical breakthroughs while widely applied in various fields. ML allows systems the ability to access data and use it to enable computers to execute cognitive processes such as learning and improving from previous experiences and solving complicated issues. Many first-order stochastic optimization methods have been used to solve the optimization model of ML. These algorithms adopt Barzilai-Borwein (BB) step size instead of fixed or diminishing step size to improve performance. However, the BB step size format involves fractional calculation, which inevitably leads to a zero denominator, especially when the objective function is non-convex. The BB technique will be violated if the denominator is near 0 or even negative. To improve the computation of the step size, a Positive Defined Stabilized Barzilai-Borwein (PDSBB) approach is introduced in this paper. Integrating PDSBB with the stochastic variance reduced gradient (SVRG) approach, a new method SVRG-PDSBB is proposed. Numerical experiments have shown that the new algorithm has stabilized the performance of the new step size, which successfully avoiding zero denominators and effectively solving the common problems in machine learning. The convergence of SVRG-PDSBB is theoretically and numerically proven, and the effectiveness of the new algorithm is shown by comparison with other algorithms. [ABSTRACT FROM AUTHOR]

In order to solve the problem of imbalanced data classification,this paper proposes an AdaBoostv algorithm based on optimal margin.In this algorithm,the improved SVM is used as the base classifier,the margin mean term is introduced into the optimization model of SVM,and the margin mean term and loss function term are weighted by data imbalance ratio.The stochastic variance reduced gradient (SVRG) is used to solve the optimization model to improve the convergence rate.In the optimal margin AdaBoostv algorithm,a new adaptive cost sensitive function is introduced into the instance weight update formula,the minority instances,the misclassified instances and the borderline minority instances are assigned higher cost values.In addition,a new weight strategy of the base classifier is derived by combining the new weight formula and introducing the estimated value of the optimal margin under the given precision parameter v,so as to further improve the classification accuracy of the algorithm.The experimental results show that the classification accuracy of the AdaBoostv algorithm with optimal margin is better than other algorithms on imbalanced datasets in the case of linear and nonlinear,and it can obtain a larger minimum margin.

The machine learning model converges slowly and has unstable training since large variance by random using a sample estimate gradient in SGD. To this end, we propose a noise reduction method for Stochastic Variance Reduction gradient (SVRG), called N-SVRG, which uses small batches samples instead of all samples for the average gradient calculation, while performing an incremental update of the average gradient. In each round of iteration, a small batch of samples is randomly selected for the average gradient calculation, while the average gradient is updated by rounding of the past model gradients during internal iterations. By suitably reducing the batch size B, the memory storage as well as the number of iterations can be reduced. The experiments are compared with the state-of-the-art Mini-Batch SGD, AdaGrad, RMSProp, SVRG and SCSG, and it is demonstrated that N-SVRG outperforms SVRG and SASG, and is on par with SCSG. Finally, by exploring the relationship between the small values of different parameters n. B and k and the effectiveness of the algorithm, we prove that ourN-SVRG algorithm has some stability and can achieve sufficient accuracy even in the case of small batch size. The advantages and disadvantages of various methods are experimentally compared, and the stability of N-SVRG is explored by parameter settings. [ABSTRACT FROM AUTHOR]

Learning representation from relative similarity comparisons, often called ordinal embedding, gains rising attention in recent years. Most of the existing methods are based on semi-definite programming (SDP), which is generally time-consuming and degrades the scalability, especially confronting large-scale data. To overcome this challenge, we propose a stochastic algorithm called SVRG-SBB, which has the following features: i) achieving good scalability via dropping positive semi-definite (PSD) constraints as serving a fast algorithm, i.e., stochastic variance reduced gradient (SVRG) method, and ii) adaptive learning via introducing a new, adaptive step size called the stabilized Barzilai-Borwein (SBB) step size. Theoretically, under some natural assumptions, we show the O(1/T) rate of convergence to a stationary point of the proposed algorithm, where T is the number of total iterations. Under the further Polyak-Łojasiewicz assumption, we can show the global linear convergence (i.e., exponentially fast converging to a global optimum) of the proposed algorithm. Numerous simulations and real-world data experiments are conducted to show the effectiveness of the proposed algorithm by comparing with the state-of-the-art methods, notably, much lower computational cost with good prediction performance. [ABSTRACT FROM AUTHOR]

The rich data used to train learning models increasingly tend to be distributed and private. It is important to efficiently perform learning tasks without compromising individual users' privacy even considering untrusted learning applications and, furthermore, understand how privacy-preservation mechanisms impact the learning process. To address the problem, we design a differentially private distributed algorithm based on the stochastic variance reduced gradient (SVRG) algorithm, which prevents the learning server from accessing and inferring private training data with a theoretical guarantee. We quantify the impact of the adopted privacy-preservation measure on the learning process in terms of convergence rate, by which it indicates noises added at each gradient update results in a bounded deviation from the optimum. To further evaluate the impact on the trained models, we compare the proposed algorithm with SVRG and stochastic gradient descent using logistic regression and neural nets. The experimental results on benchmark data sets show that the proposed algorithm has minor impact on the accuracy of trained models under a moderate amount of privacy budget. [ABSTRACT FROM AUTHOR]

Stochastic gradient descent is a popular method in large-scale optimization for machine learning but suffers from a slow convergence. In recent years, stochastic variance reduced gradient (SVRG) is proposed to remedy this problem. Although many variants of SVRG have been studied, the analysis of variance has not been thoroughly discussed. In this paper, we propose a general framework denoted by epoch-update-indentification (EUI), which is an abstraction of the existing variants of SVRG. Under this framework i.e., EUI, we then provide a general analysis of the variance reduction technique from a new perspective. Additionally, those previous variants of SVRG have to keep a snapshot of the full gradient for each epoch, which is computationally expensive. In this paper, we also propose a new variant of SVRG named sampleVR which estimates the snapshot of the full gradient by using a sampling strategy, thus leading to decrease the gradient complexity significantly. Both the theoretical analysis and extensive empirical studies show that sampleVR achieves a good tradeoff between convergence performance and gradient complexity, and thus makes the training loss converge faster than its counterparts.

In this work, a descriptor, SVRG (principal component scores vector of radial distribution function descriptors and geometrical descriptors), was derived from principal component analysis (PCA) of a matrix of two structural variables of coded amino acids, including radial distribution function index (RDF) and geometrical index. SVRG scales were then applied in three panels of peptide quantitative structure-activity relationships (QSARs) which were modelled by partial least squares regression (PLS). The obtained models with the correlation coefficient ( [image omitted]), cross-validation correlation coefficient ( [image omitted]) were 0.910 and 0.863 for 48 bitter-tasting dipeptides; 0.968 and 0.931 for 21 oxytocin analogues; and 0.992 and 0.954 for 20 thromboplastin inhibitors. Satisfactory results showed that SVRG contained much chemical information relating to bioactivities. The approach may be a useful structural expression methodology for studies on peptide QSAR. [ABSTRACT FROM AUTHOR]