Search

Online learning is an efficient approach in machine learning and statistics, which iteratively updates models upon the observation of a sequence of training examples. A representative online learning algorithm is the online gradient descent, which has found wide applications due to its low complexity and scalability to large datasets. Kernel-based learning methods have been proven to be quite successful in dealing with nonlinearity in the data and multivariate optimization. In this paper we present a class of kernel-based online gradient descent algorithm for addressing regression problems, which generates sparse estimators in an iterative way to reduce the algorithmic complexity for training streaming datasets and model selection in large-scale learning scenarios. In the setting of support vector regression (SVR), we design the sparse online learning algorithm by introducing a sequence of insensitive distance-based loss functions. We prove consistency and error bounds quantifying the generalization performance of such algorithms under mild conditions. The theoretical results demonstrate the interplay between statistical accuracy and sparsity property during learning processes. We show that the insensitive parameter plays a crucial role in providing sparsity as well as fast convergence rates. The numerical experiments also support our theoretical results. [ABSTRACT FROM AUTHOR]

Several extrapolation strategies have been proposed in the literature to accelerate the EM algorithm, with varying degrees of success. One advantage of extrapolation methods is their ease of implementation, as they only require working with the EM iterations and do not need auxiliary quantities, such as gradient and Hessian. In this paper, we introduce a new family of iterative schemes based on vector extrapolation methods. We also construct and numerically test a randomly relaxed version of the scheme. Our results demonstrate that these new strategies can significantly and stably accelerate the convergence of the EM algorithm compared to existing methods. Moreover, these strategies are highly versatile as they can accelerate any linearly convergent fixed point iteration, including EM-type algorithms. Finally, we provide statistical modeling experiments at the end of the paper to demonstrate the applicability and interest of these convergence acceleration schemes, whether applied to the EM algorithm or one of its variants. [ABSTRACT FROM AUTHOR]

Cluster analysis requires fixing the number of clusters and often many hyper-parameters. In practice, one produces several partitions, and a final one is chosen based on validation or selection criteria. There exist an abundance of validation methods that, implicitly or explicitly, assume a certain clustering notion. In this paper, we focus on groups that can be well separated by quadratic or linear boundaries. The reference cluster concept is defined through the quadratic discriminant function and parameters describing clusters' size, center and scatter. We develop two cluster-quality criteria that are consistent with groups generated from a class of elliptic–symmetric distributions. Using the bootstrap resampling of the proposed criteria, we propose a selection rule that allows choosing among many clustering solutions, eventually obtained from different methods. Extensive experimental analysis shows that the proposed methodology achieves a better overall performance compared to established alternatives from the literature. [ABSTRACT FROM AUTHOR]

This paper proposes a new test for a change point in the mean of high-dimensional data based on the spatial sign and self-normalization. The test is easy to implement with no tuning parameters, robust to heavy-tailedness and theoretically justified with both fixed- n and sequential asymptotics under both null and alternatives, where n is the sample size. We demonstrate that the fixed- n asymptotics provide a better approximation to the finite sample distribution and thus should be preferred in both testing and testing-based estimation. To estimate the number and locations when multiple change-points are present, we propose to combine the p -value under the fixed- n asymptotics with the seeded binary segmentation (SBS) algorithm. Through numerical experiments, we show that the spatial sign based procedures are robust with respect to the heavy-tailedness and strong coordinate-wise dependence, whereas their non-robust counterparts proposed in Wang et al. (2022) [28] appear to under-perform. A real data example is also provided to illustrate the robustness and broad applicability of the proposed test and its corresponding estimation algorithm. [ABSTRACT FROM AUTHOR]

It is well known that, in the presence of outliers, the single linkage algorithm generally fails to identify clusters. In this paper, we construct a new version of this algorithm, less sensitive to outliers, and study both its theoretical properties and its practical behavior. In particular, we provide an oracle-type inequality which guarantees that our procedure recovers clusters with high probability under mild assumptions on the distribution of the outliers. Using this inequality, we prove the consistency of our method and exhibit rates of convergence in various situations. The performance of this approach is also assessed through simulation studies. A thorough comparison with several classical clustering algorithms on simulated data is presented. [ABSTRACT FROM AUTHOR]

Conventional linear mixed-effects modeling is routinely challenging when the validity of necessary assumptions is suspicious. In particular, robustifying model fitting is appealing in the presence of potential outlying points. This paper introduces a robust regression methodology in a parametric setting by constructing a novel multivariate skew-Huber distribution for longitudinal data with heavy-tails and skewed structures. Unlike preceding studies, our model allows for jointly estimating the tuning parameter, which controls the impact of outliers, with all other parameters using an undemanding computational algorithm. Moreover, by promoting an unconstrained parameterization through the modified Cholesky decomposition, the estimate of variance–covariance components can be merely accessible. We also present a spline mixed model to account for the covariate effect. To highlight the usefulness of our methodology, we conducted a simulation study and analyzed a data set collected on type 2 diabetic patients with microalbuminuria over a 6-year prospective cohort study. Findings show that our proposed robust model leads to convincing conclusions in empirical studies. • Introduces the multivariate Huber distribution and studies its main properties. • Presents a robust regression mixed model to analyze correlated data when outliers exist. • Provides case weight for each subject through the tuning parameter estimate. • Offers an easy algorithm for jointly estimate the tuning parameter, variance–covariance components, and fixed effects. • Delivers direct parameterization by utilizing the modified Cholesky decomposition technique. We consider various correlation structures to the model specification. [ABSTRACT FROM AUTHOR]

Variable selection for multi-sample problems is of great interest in statistics. Existing methods for addressing this problem have some limits or disadvantages. In this paper, we propose distance-based variable importance measures to deal with these problems, which are inspired by the Multi-response permutation procedure (MRPP), Energy distance (ED) and Distance components (DISCO) analysis. The proposed variable importance assessments can effectively measure the importance of an individual dimension by quantifying its influence on the differences between multivariate distributions across treatment groups. An importance-measure-based backward selection (IM-BWS) algorithm is developed that can be used in variable selection for multi-sample problems to discover important variables. We propose a modified MRPP based on the IM-BWS procedure for improving the power performance of the original MRPP. Our proposed methods are model-free, work for high-dimensional data, and can capture important variables under different models. Both simulations and real data applications demonstrate that our proposed method enjoys good properties and has advantages over other existing methods. [ABSTRACT FROM AUTHOR]