Search

In this article, we focus on the unsupervised multiview feature selection, which tries to handle high-dimensional data in the field of multiview learning. Although some graph-based methods have achieved satisfactory performance, they ignore the underlying data structure across different views. Besides, their predefined Laplacian graphs are sensitive to the noises in the original data space and fail to obtain the optimal neighbor assignment. To address the above problems, we propose a novel unsupervised multiview feature selection model based on graph learning, and the contributions are three-fold: 1) during the feature selection procedure, the consensus similarity graph shared by different views is learned. Therefore, the proposed model can reveal the data relationship from the feature subset; 2) a reasonable rank constraint is added to optimize the similarity matrix to obtain more accurate information; and 3) an autoweighted framework is presented to assign view weights adaptively, and an effective alternative iterative algorithm is proposed to optimize the problem. Experiments on various datasets demonstrate the superiority of the proposed method compared to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Binarized neural networks (BNNs) have drawn significant attention in recent years, owing to great potential in reducing computation and storage consumption. While it is attractive, traditional BNNs usually suffer from slow convergence speed and dramatical accuracy-degradation on large-scale classification datasets. To minimize the gap between BNNs and deep neural networks (DNNs), we propose a new framework of designing BNNs, dubbed Hyper-BinaryNet, from the aspect of enhanced information-flow. Our contributions are threefold: 1) Considering the capacity-limitation in the backward pass, we propose an 1-bit convolution module named HyperConv. By exploiting the capacity of auxiliary neural networks, BNNs gain better performance on large-scale image classification task. 2) Considering the slow convergence speed in BNNs, we rethink the gradient accumulation mechanism and propose a hyper accumulation technique. By accumulating gradients in multiple variables rather than one as before, the gradient paths for each weight increase, which escapes BNNs from the gradient bottleneck problem during training. 3) Considering the ill-posed optimization problem, a novel gradient estimation warmup strategy, dubbed STE-Warmup, is developed. This strategy prevents BNNs from the unstable optimization process by progressively transferring neural networks from 32-bit to 1-bit. We conduct evaluations with variant architectures on three public datasets: CIFAR-10/100 and ImageNet. Compared with state-of-the-art BNNs, Hyper-BinaryNet shows faster convergence speed and outperforms existing BNNs by a large margin. [ABSTRACT FROM AUTHOR]

These days, clustering is one of the most classical themes to analyze data structures in machine learning and pattern recognition. Recently, the anchor-based graph has been widely adopted to promote the clustering accuracy of plentiful graph-based clustering techniques. In order to achieve more satisfying clustering performance, we propose a novel clustering approach referred to as the progressive self-supervised clustering method with novel category discovery (PSSCNCD), which consists of three separate procedures specifically. First, we propose a new semisupervised framework with novel category discovery to guide label propagation processing, which is reinforced by the parameter-insensitive anchor-based graph obtained from balanced $K$ -means and hierarchical $K$ -means (BKHK). Second, we design a novel representative point selected strategy based on our semisupervised framework to discover each representative point and endow pseudolabel progressively, where every pseudolabel hypothetically corresponds to a real category in each self-supervised label propagation. Third, when sufficient representative points have been found, the labels of all samples will be finally predicted to obtain terminal clustering results. In addition, the experimental results on several toy examples and benchmark data sets comprehensively demonstrate that our method outperforms other clustering approaches. [ABSTRACT FROM AUTHOR]

Graph-based clustering aims to partition the data according to a similarity graph, which has shown impressive performance on various kinds of tasks. The quality of similarity graph largely determines the clustering results, but it is difficult to produce a high-quality one, especially when data contain noises and outliers. To solve this problem, we propose a robust rank constrained sparse learning (RRCSL) method in this article. The $L_{2,1}$ -norm is adopted into the objective function of sparse representation to learn the optimal graph with robustness. To preserve the data structure, we construct an initial graph and search the graph within its neighborhood. By incorporating a rank constraint, the learned graph can be directly used as the cluster indicator, and the final results are obtained without additional postprocessing. In addition, the proposed method cannot only be applied to single-view clustering but also extended to multiview clustering. Plenty of experiments on synthetic and real-world datasets have demonstrated the superiority and robustness of the proposed framework. [ABSTRACT FROM AUTHOR]

The multiple kernel $k$ -means (MKKM) and its variants utilize complementary information from different sources, achieving better performance than kernel $k$ -means (KKM). However, the optimization procedures of most previous works comprise two stages, learning the continuous relaxation matrix and obtaining the discrete one by extra discretization procedures. Such a two-stage strategy gives rise to a mismatched problem and severe information loss. Even worse, most existing MKKM methods overlook the correlation among prespecified kernels, which leads to the fusion of mutually redundant kernels and bad effects on the diversity of information sources, finally resulting in unsatisfying results. To address these issues, we elaborate a novel Discrete and Parameter-free Multiple Kernel $k$ -means (DPMKKM) model solved by an alternative optimization method, which can directly obtain the cluster assignment results without subsequent discretization procedure. Moreover, DPMKKM can measure the correlation among kernels by implicitly introducing a regularization term, which is able to enhance kernel fusion by reducing redundancy and improving diversity. Noteworthily, the time complexity of optimization algorithm is successfully reduced, through masterly utilizing of coordinate descent technique, which contributes to higher algorithm efficiency and broader applications. What’s more, our proposed model is parameter-free avoiding intractable hyperparameter tuning, which makes it feasible in practical applications. Lastly, extensive experiments conducted on a number of real-world datasets illustrated the effectiveness and superiority of the proposed DPMKKM model. [ABSTRACT FROM AUTHOR]

Sparse unmixing (SU) algorithms use the existing spectral library as prior knowledge to analyze the endmembers and estimate abundance maps. The majority of SU algorithms use loss functions based on the $L_{2,1}$ -norm or $F$ -norm to minimize reconstruction error. They have different advantages and shortcomings. In short, $F$ -norm has a differentiable characteristic, and it is easy to minimize as a loss function. However, it is very sensitive to heavy noise and outliers. While the $L_{2,1}$ -norm emphasizes the reconstruction error on each band and is robust to noise with different intensities in different bands. But the $L_{2,1}$ -norm is nondifferentiable at zero-point. This article introduces an adaptive loss function based on the $\sigma $ -norm for SU, which combines the advantages of the $L_{2,1}$ -norm and $F$ -norm. The adaptive loss function is related to a nonnegative parameter $\sigma $. By adjusting the parameter $\sigma $ , the adaptive loss function can approach the $F$ -norm or $L_{2,1}$ -norm. To the best of our knowledge, it is the first time to apply an adaptive loss function to SU. Moreover, the adaptive loss function is globally differentiable, and we propose an optimization algorithm for the adaptive loss function and verify its convergence. Experiments on real-world and synthetic HSIs show that the adaptive loss function effectively enhances the performance of the SU algorithms. [ABSTRACT FROM AUTHOR]

Unsupervised feature selection (UFS) aims to remove the redundant information and select the most representative feature subset from the original data, so it occupies a core position for high-dimensional data preprocessing. Many proposed approaches use self-expression to explore the correlation between the data samples or use pseudolabel matrix learning to learn the mapping between the data and labels. Furthermore, the existing methods have tried to add constraints to either of these two modules to reduce the redundancy, but no prior literature embeds them into a joint model to select the most representative features by the computed top ranking scores. To address the aforementioned issue, this article presents a novel UFS method via a convex non-negative matrix factorization with an adaptive graph constraint (CNAFS). Through convex matrix factorization with adaptive graph constraint, it can dig up the correlation between the data and keep the local manifold structure of the data. To our knowledge, it is the first work that integrates pseudo label matrix learning into the self-expression module and optimizes them simultaneously for the UFS solution. Besides, two different manifold regularizations are constructed for the pseudolabel matrix and the encoding matrix to keep the local geometrical structure. Eventually, extensive experiments on the benchmark datasets are conducted to prove the effectiveness of our method. The source code is available at: https://github.com/misteru/CNAFS. [ABSTRACT FROM AUTHOR]

In this article, we propose a novel feature selection approach via explicitly addressing the long-standing subspace sparsity issue. Leveraging $\ell _{2,1}$ -norm regularization for feature selection is the major strategy in existing methods, which, however, confronts sparsity limitation and parameter-tuning trouble. To circumvent this problem, employing the $\ell _{2,0}$ -norm constraint to improve the sparsity of the model has gained more attention recently whereas, optimizing the subspace sparsity constraint is still an unsolved problem, which only can acquire an approximate solution and without convergence proof. To address the above challenges, we innovatively propose a novel subspace sparsity discriminative feature selection (S2DFS) method which leverages a subspace sparsity constraint to avoid tuning parameters. In addition, the trace ratio formulated objective function extremely ensures the discriminability of selected features. Most important, an efficient iterative optimization algorithm is presented to explicitly solve the proposed problem with a closed-form solution and strict convergence proof. To the best of our knowledge, such an optimization algorithm of solving the subspace sparsity issue is first proposed in this article, and a general formulation of the optimization algorithm is provided for improving the extensibility and portability of our method. Extensive experiments conducted on several high-dimensional text and image datasets demonstrate that the proposed method outperforms related state-of-the-art methods in pattern classification and image retrieval tasks. [ABSTRACT FROM AUTHOR]

$k$ k -means method using Lloyd heuristic is a traditional clustering method which has played a key role in multiple downstream tasks of machine learning because of its simplicity. However, Lloyd heuristic always finds a bad local minimum, i.e., the bad local minimum makes objective function value not small enough, which limits the performance of $k$ k -means. In this paper, we use coordinate descent (CD) method to solve the problem. First, we show that the $k$ k -means minimization problem can be reformulated as a trace maximization problem, then a simple and efficient coordinate descent scheme is proposed to solve the maximization problem. Two interesting findings through theory are that Lloyd cannot decrease the objective function value of $k$ k -means produced by our CD further, and our proposed method CD to solve $k$ k -means problem can avoid produce empty clusters. In addition, according to the computational complexity analysis, it is verified CD has the same time complexity with original $k$ k -means method. Extensive experiments including statistical hypothesis testing, on several real-world datasets with varying number of clusters, varying number of samples and varying number of dimensions show that CD performs better compared to Lloyd, i.e., lower objective value, better local minimum and fewer iterations. And CD is more robust to initialization than Lloyd whether the initialization strategy is random or initialization of $k$ k -means++. [ABSTRACT FROM AUTHOR]

Due to the exponential growth of multimedia data, multi-modal hashing as a promising technique to make cross-view retrieval scalable is attracting more and more attention. However, most of the existing multi-modal hashing methods either divide the learning process unnaturally into two separate stages or treat the discrete optimization problem simplistically as a continuous one, which leads to suboptimal results. Recently, a few discrete multi-modal hashing methods that try to address such issues have emerged, but they still ignore several important discrete constraints (such as the balance and decorrelation of hash bits). In this paper, we overcome those limitations by proposing a novel method named “Enhanced Discrete Multi-modal Hashing (EDMH)” which learns binary codes and hashing functions simultaneously from the pairwise similarity matrix of data, under the aforementioned discrete constraints. Although the model of EDMH looks a lot more complex than the other models for multi-modal hashing, we are actually able to develop a fast iterative learning algorithm for it, since the subproblems of its optimization all have closed-form solutions after introducing a couple of auxiliary variables. Our experimental results on three real-world datasets have revealed the usefulness of those previously ignored discrete constraints and demonstrated that EDMH not only performs much better than state-of-the-art competitors according to several retrieval metrics but also runs much faster than most of them. [ABSTRACT FROM AUTHOR]

Recently, several robust principle component analysis (RPCA) models have been proposed to improve the robustness of principle component analysis (PCA). But an important problem that the robustness to outliers affects the discrimination of correct samples has not been solved yet. To solve this problem, we propose a truncated robust principle component analysis (T-RPCA) model which treats correct samples and outliers separately. In fact, the proposed model performs an implicitly truncated weighted learning scheme which is more reasonable for robustness learning respective to previous works. Moreover, we propose a re-weighted (RW) optimization framework to solve a general problem and generalize two sub-frameworks upon it. To be specific, the first sub-framework orients a general truncated loss optimization problem which contains the objective problem of T-RPCA, and the second one focuses on a general singular-value based optimization problem. Besides, we provide rigorously theoretical guarantees for the proposed model, RW framework and sub-frameworks. Empirical studies demonstrate that the proposed T-RPCA model outperforms previous RPCA models on reconstruction and classification tasks. [ABSTRACT FROM AUTHOR]

Semisupervised band selection (BS) technique plays an important role in processing hyperspectral images (HSIs) because of its superiority of using the limited labeled data and plentiful unlabeled data to select the discriminative and informative feature, which copes with the problem of high-dimensional and scare labeled samples of HSIs. Among semisupervised BS models, graph-based methods are superior to others in many situations and have received more and more attention. However, traditional graph-based models construct a similarity matrix and select valuable bands independently. The similarity matrix remains constant, which will damage the local manifold structure and lead to a suboptimal result. To solve this problem, we propose a semisupervised band selection with an optimal graph (BSOG) approach, which performs BS and local structure learning simultaneously. Instead of fixing the input similarity matrix, the similarity matrix is updated constantly to learn a better local structure. Besides, the learned similarity matrix is adaptive. Then, the optimal band subset can be selected by analyzing the obtained projection matrix $\boldsymbol W$. An efficient and simple optimization algorithm is proposed to solve this model. Experiments on four real HSIs data validate the effectiveness of the proposed model. [ABSTRACT FROM AUTHOR]

Spectral clustering methods are gaining more and more interests and successfully applied in many fields because of their superior performance. However, there still exist two main problems to be solved: 1) spectral clustering methods consist of two successive optimization stages—spectral embedding and spectral rotation, which may not lead to globally optimal solutions, 2) and it is known that spectral methods are time-consuming with very high computational complexity. There are methods proposed to reduce the complexity for data vectors but not for graphs that only have information about similarity matrices. In this paper, we propose a new method to solve these two challenging problems for graph clustering. In the new method, a new framework is established to perform spectral embedding and spectral rotation simultaneously. The newly designed objective function consists of both terms of embedding and rotation, and we use an improved spectral rotation method to make it mathematically rigorous for the optimization. To further accelerate the algorithm, we derive a low-dimensional representation matrix from a graph by using label propagation, with which, in return, we can reconstruct a double-stochastic and positive semidefinite similarity matrix. Experimental results demonstrate that our method has excellent performance in time cost and accuracy. [ABSTRACT FROM AUTHOR]