Search

In this paper, we investigate L 0 -regularized logistic regression models, and design two fast and efficient algorithms for high-dimensional correlated data and massive data, respectively. Our first algorithm, the Variable Sorted Active Set (VSAS) algorithm, is based on the local quadratic approximation of the KKT conditions for L 0 -penalized maximum log-likelihood function in high-dimensional correlated data. We establish an L ∞ error upper bound for the estimator obtained by the VSAS algorithm and prove its optimal convergence rate. Moreover, when the target signal exceeds the detectable level, the estimator obtained by the VSAS algorithm can achieve the oracle estimator with high probability. Our second algorithm, Communication Effective Variable Sorted Active Set (CEVSAS), aims to solve high-dimensional and large-sample L 0 -regularized logistic regression models by reduce computational and communication costs, while maintaining estimation efficiency. Finally, simulations and real data demonstrate the effectiveness of our proposed VSAS and CEVSAS algorithms. • We develop a fast VSAS algorithm for L 0 regularized logistic regression involving high-dimensional correlated data. • An L ∞ upper bound is established on the estimation error of the solution sequence generated by VSAS algorithm. • The CEVSAS altorithm is proposed to reduce computational costs and ensure higher estimation efficiency. • We propose two adaptive versions of the VSAS and CEVSAS algorithms for selecting the optimal tuning parameters. [ABSTRACT FROM AUTHOR]

Highlights • Optimization is an important issue in machine learning because many machine learning models are reformulated as optimization problems. Different kinds of machine learning algorithms mainly focus on minimizing their emiprical loss like deep learning, logisitic regression, and support vector machine. Because data is explosively growing, it is challenging to deal with a large-scale optimization problem. We propose a fast and most condition number free algorithm for train machine learning algorithms. This algorithm has several advantages. • Contrast to previous work which approximates Hessian matrix by sketched or subsampled Hessian directly, we use the sketched or subsampled Hessian matrix as preconditioner and get the direction vector by preconditioned conjugate gradient method. Accordingly, we propose a novel stochastic second-order algorithm called Preconditioned Newton Conjugate Gradient with Sketched Hessian (PNCG). This strategy makes our algorithm almost independent on the condition number. • We numerically demonstrate the effectiveness and robustness of our algorithms in widely used machine learning algorithm Ridge Logistic Regression on several real datasets with different condition number. Our algorithms show good performance compared with existing second order methods in all cases, especially when condition number is large. Abstract Optimization is an important issue in machine learning because many machine learning models are reformulated as optimization problems. Different kinds of machine learning algorithms mainly focus on minimizing their empirical loss like deep learning, logistic regression, and support vector machine. Because data is explosively growing, it is challenging to deal with a large-scale optimization problem. Recently, stochastic second-order methods have emerged to attract much attention due to their efficiency in each iteration. These methods show good performance on training machine learning algorithms like logistic regression and support vector machine. However, the computational complexity of existing stochastic second-order methods heavily depends on the condition number of the Hessian. In this paper, we propose a new Newton-like method called Preconditioned Newton Conjugate Gradient with Sketched Hessian (PNCG). The runtime complexity of PNCG is at most logarithmic in the condition number of the Hessian. PNCG exhibits advantages over existing subsampled Newton methods especially when the Hessian matrix in question is ill-conditioned. We also show that our method has good performance on training machine learning algorithm empirically. The results show consistent improvements in computational efficiency. [ABSTRACT FROM AUTHOR]

This paper proposes a video super-resolution method based on an adaptive superpixel-guided auto-regressive (AR) model. Key-frames are automatically selected and super-resolved by a sparse regression method. Non-key-frames are super-resolved by exploiting the spatio-temporal correlations: the temporal correlation is exploited by an optical flow method while the spatial correlation is modeled by a superpixel-guided AR model. Experimental results show that the proposed method outperforms state-of-the-art methods in terms of both subjective visual quality and objective peak signal-to-noise ratio (PSNR). The proposed method requires less computation and is suitable for practical applications. [ABSTRACT FROM AUTHOR]

• Presenting a new multi-label feature selection method that efficiently combines manifold regularization and dependence maximization. • Introducing a HSIC-based measurement to evaluate the dependence between the manifold space and label space. • Developing an iterative optimization method to solve the objective function of our method MRDM with good convergence. • Conducting extensive experiments on various multi-label data sets to demonstrate the superiority of the proposed method. Feature selection is able to select more discriminative features for classification and plays an important role in multi-label learning to alleviate the effect of the curse of dimensionality. Recently, the multi-label feature selection methods based on the sparse regression model have received increasing attentions. However, most of these methods directly project original data space to label space in the regression model, which is inappropriate because the linear assumption between data space and label space doesn't hold in most cases. In the paper, we propose a feature selection method named multi-label feature selection via manifold regularization and dependence maximization (MRDM). In the regression model of MRDM, the original data space is projected to a low-dimensional manifold space, which not only has the same topological structure with the original data, but also has a strong dependence with the class labels. Then, an objective function involving l 2 , 1 -norm regularization is formulated, and an alternating optimization-based iterative algorithm is designed to obtain the sparse coefficients for multi-label feature selection. Extensive experiments on various multi-label data sets demonstrate the superiority of the proposed method compared with some state-of-the-art multi-label feature selection methods. [ABSTRACT FROM AUTHOR]

Abstract: This paper presents a supervised learning algorithm for image deblurring. The task is addressed into the conceptual framework of matrix regression and gradient evolution. Specifically, given pairs of blurred image patches and their corresponding clear ones, an optimization framework of matrix regression is proposed to learn a matrix mapping. For an image to be deblurred, the learned matrix mapping will be employed to map each of its image patches directly to be a new one with more sharp details. The mapped result is then analyzed in terms of edge profiles, and the image is finally deblurred in way of gradient evolution. The algorithm is fast, and easy to be implemented. Comparative experiments on diverse natural images and the applications to interactive deblurring of real-world out-of-focus images illustrate the validity of our method. [Copyright &y& Elsevier]

Abstract: In this paper we address the problem of robust face recognition by formulating the pattern recognition task as a problem of robust estimation. Using a fundamental concept that in general, patterns from a single object class lie on a linear subspace (Barsi and Jacobs, 2003 ), we develop a linear model representing a probe image as a linear combination of class specific galleries. In the presence of noise, the well-conditioned inverse problem is solved using the robust Huber estimation and the decision is ruled in favor of the class with the minimum reconstruction error. The proposed Robust Linear Regression Classification (RLRC) algorithm is extensively evaluated for two important cases of robustness i.e. illumination variations and random pixel corruption. Illumination invariant face recognition is demonstrated on three standard databases under exemplary evaluation protocols reported in the literature. Comprehensive comparative analysis with the state-of-art illumination tolerant approaches indicates a comparable performance index for the proposed RLRC algorithm. The efficiency of the proposed approach in the presence of severe random noise is validated under several exemplary noise models such as dead-pixel problem, salt and pepper noise, speckle noise and Additive White Gaussian Noise (AWGN). The RLRC algorithm is found to be favorable compared with the benchmark generative approaches. [Copyright &y& Elsevier]

Abstract: Pedestrian counting plays an important role in public safety and intelligent transportation. Most pedestrian counting algorithms based on supervised learning require much labeling work and rarely exploit the topological information of unlabelled data in a video. In this paper, we propose a Semi-Supervised Elastic Net (SSEN) regression method by utilizing sequential information between unlabelled samples and their temporally neighboring samples as a regularization term. Compared with a state-of-the-art algorithm, extensive experiments indicate that our algorithm can not only select sparse representative features from the original feature space without losing their interpretability, but also attain superior prediction performance with only very few labelled frames. [Copyright &y& Elsevier]

Abstract: Recently, mutual interdependence analysis (MIA) has been successfully used to extract representations, or “mutual features”, accounting for samples in the class. For example, a mutual feature is a face signature under varying illumination conditions or a speaker signature under varying channel conditions. A mutual feature is a linear regression that is equally correlated with all samples of the input class. Previous work discussed two equivalent definitions of this problem and a generalization of its solution called generalized MIA (GMIA). Moreover, it showed how mutual features can be computed and employed. This paper uses a parametrized version ) to pursue a deeper understanding of what GMIA features really represent. It defines a generative signal model that is used to interpret ) and visualize its difference to MIA, principal and independent component analysis. Finally, we analyze the effect of on the feature extraction performance of ) in two standard pattern recognition problems: illumination-independent face recognition and text-independent speaker verification. [ABSTRACT FROM AUTHOR]

Biclustering (co-clustering, two-mode clustering), as one of the classical unsupervised learning methods, has been applied in many different fields in recent years. Different types of biclustering methods have been developed such as probabilistic methods, two-way clustering methods, variance minimization methods, and so on. However, few regression-based methods have been proposed to the best of our knowledge. Such methods have been applied in traditional clustering, which can improve both the computational efficiency and the clustering accuracy. In this paper, we present a penalized regression-based method for localizing the biclusters (PRbiclust). By imposing Truncated LASSO Penalty (TLP) and group TLP terms to penalize the column vectors and the row vectors in the regression model, the structure of biclusters in the data matrix is recovered. The model is formulated as an optimization problem with nonconvex penalties, and a computationally efficient algorithm is proposed to solve it. Convergence of the algorithm is proved. To extract the biclusters from the recovered data matrix, we propose a graph-based localization method. An evaluation criterion is also proposed to measure the efficiency of bicluster localization when noise entries exist. We apply the proposed method to both simulated datasets with different setups and a real dataset. Experiments show that this method can well capture the bicluster structure, and performs better than the existing works. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, we develop a semi-supervised regression algorithm to analyze data sets which contain both categorical and numerical attributes. This algorithm partitions the data sets into several clusters and at the same time fits a multivariate regression model to each cluster. This framework allows one to incorporate both multivariate regression models for numerical variables (supervised learning methods) and k-mode clustering algorithms for categorical variables (unsupervised learning methods). The estimates of regression models and k-mode parameters can be obtained simultaneously by minimizing a function which is the weighted sum of the least-square errors in the multivariate regression models and the dissimilarity measures among the categorical variables. Both synthetic and real data sets are presented to demonstrate the effectiveness of the proposed method. [Copyright &y& Elsevier]

Abstract: In this paper, we compare the performances of classifier combination methods (bagging, modified random subspace method, classifier selection, parametric fusion) to logistic regression in consideration of various characteristics of input data. Four factors used to simulate the logistic model are: (a) combination function among input variables, (b) correlation between input variables, (c) variance of observation, and (d) training data set size. In view of typically unknown combination function among input variables, we use a Taguchi design to improve the practicality of our study results by letting it as an uncontrollable factor. Our experimental study results indicate the following: when training set size is large, performances of logistic regression and bagging are not significantly different. However, when training set size is small, the performance of logistic regression is worse than bagging. When training data set size is small and correlation is strong, both modified random subspace method and bagging perform better than the other three methods. When correlation is weak and variance is small, both parametric fusion and classifier selection algorithm appear to be the worst at our disappointment. [Copyright &y& Elsevier]

Abstract: Gaussian fields (GF) have recently received considerable attention for dimension reduction and semi-supervised classification. In this paper we show how the GF framework can be used for semi-supervised regression on high-dimensional data. We propose an active learning strategy based on entropy minimization and a maximum likelihood model selection method. Furthermore, we show how a recent generalization of the LLE algorithm for correspondence learning can be cast into the GF framework, which obviates the need to choose a representation dimensionality. [Copyright &y& Elsevier]

• A novel tensor regression model is introduced to simultaneously capture the underlying low-rank and sparse structure of the coefficient tensor. • Unlike traditional tensor regression models that impose sparse penalty on the factor matrices of the coefficient tensor, our model directly imposes sparse penalty on the coefficient tensor. • An orthonormality constraint is imposed on the factor matrices to make our model identifiable. • We evaluate the proposed model on synthetic and real data sets. The results show that our model achieves competitive performance. Many applications in biomedical informatics deal with data in the tensor form. Traditional regression methods which take vectors as covariates may encounter difficulties in handling tensors due to their ultrahigh dimensionality and complex structure. In this paper, we introduce a novel sparse regularized Tucker tensor regression model to exploit the structure of tensor covariates and perform feature selection on tensor data. Based on Tucker decomposition of the regression coefficient tensor, we reduce the ultrahigh dimensionality to a manageable level. To make our model identifiable, we impose the orthonormality constraint on the factor matrices. Unlike previous tensor regression models that impose sparse penalty on the factor matrices of the coefficient tensor, our model directly imposes sparse penalty on the coefficient tensor to select the relevant features on tensor data. An efficient optimization algorithm based on alternating direction method of multiplier (ADMM) algorithm is designed to solve our proposed model. The performance of our model is evaluated on both synthetic and real genomic data. Experiment results on synthetic data demonstrate that our model could identify the true related signals more accurately than other state-of-the-art regression models. The analysis on genomic data of melanoma demonstrates that our model can achieve better prediction performance and identify markers with important implications. Our model and the associated studies can provide useful insights to disease or pathogenesis mechanisms, and will benefit further studies in variable selection. [ABSTRACT FROM AUTHOR]