Search

• An analysis of lexicographic optimization for feature selection and data classification. • Cooperative co-evolution for feature selection while the classifier hyperparameters are also being optimized. • Optimization of multiple objectives using a simple evolutionary algorithm. • A new lexicographic ranking methodology to compare the results of different feature selection methods. • Quite good results for real high-dimensional datasets. This paper starts with two hypotheses. The first one is that the simultaneous optimization of the hyperparameters regulating the classifier within a wrapper method, while the best subset of features is being determined, should improve the results with respect to those obtained with a pre-parameterized classifier. The second one is that solving these two problems can be formulated as a lexicographic optimization problem, allowing the use of a simple single-objective evolutionary algorithm to solve this multi-objective problem. The fitness function is of key importance for such wrapper methods. It is responsible for guiding the search towards potentially good solutions and it also consumes most of the runtime. Having these issues in mind, this paper also proposes a new lexicographic fitness function, designed to minimize the runtime of the algorithm and also to avoid over-fitting. Furthermore, the execution time and the quality of the results obtained by the wrapper procedure also depend on some algorithmic hyperparameters: the similarity thresholds used when comparing two different solutions lexicographically and the percentage of data samples used for validation during the training process. Thus, an experimental analysis has been carried out to find adequate values for these hyperparameters. Finally, the lexicographic cooperative co-evolutionary wrapper approach, using the new fitness function proposed in this paper, has been tested with several datasets belonging to the University of California, Irvine (UCI) repository and also with some real high-dimensional datasets, obtaining quite good results, compared to other state-of-the-art wrapper methods. The comparison has also been made lexicographically, with a new methodology proposed in this paper. [ABSTRACT FROM AUTHOR]

Feature selection has been recognized for long as an important preprocessing technique to reduce dimensionality and improve the performance of regression and classification tasks. The class of sequential forward feature selection methods based on Mutual Information (MI) is widely used in practice, mainly due to its computational efficiency and independence from the specific classifier. A recent work introduced a theoretical framework for this class of methods which explains the existing proposals as approximations to an optimal target objective function. Such framework made clear the advantages and drawbacks of each proposal. Methods accounting for the redundancy of candidate features using a maximization function and considering the so-called complementary effect are among the best ones. However, they still penalize the complementarity, which is an important drawback. This paper proposes the Decomposed Mutual Information Maximization (DMIM) method, which keeps the good theoretical properties of the best methods proposed so far but overcomes the complementarity penalization by applying the maximization separately to the inter-feature and class-relevant redundancies. DMIM was extensively evaluated and compared with other methods, both theoretically and using two synthetic scenarios and 20 publicly available real datasets applied to specific classifiers. Our results show that DMIM achieves a better classification performance than the remaining forward feature selection methods based on MI. [ABSTRACT FROM AUTHOR]

Traditional approaches that represent observations in vector or matrix notation easily lead to structure information losses and dependency destruction between elements. Tensors are the generalizations of matrices, providing a natural representation of rich structure in real-world multiway arrays. Although tensor decomposition has been verified and used in classification tasks, the heterogeneity gap among different orders is often neglected. In this paper, we exploited tensor decomposition framework and presented a heterogeneous tensor decomposition for robust classification (HTDRC) approach, which integrates nuclear and ℓ 2 , 1 -norm for intrinsic representation learning. Specifically, to obtain the lowest-rank intrinsic representations of tensorial data from an underlying low-dimensional subspace, HTDRC simultaneously learns a set of orthogonality constrained factor matrices and a low-rank constrained representation matrix. To better guide the decomposition process, a robust discriminant feature selection scheme is utilized by imposing ℓ 2 , 1 -norm penalty on the constructed classification loss and the regularized terms simultaneously. Experiments on four datasets demonstrated the superior performance of our proposed approach. [ABSTRACT FROM AUTHOR]

Sparse Bayesian learning (SBL) provides state-of-the-art performance in accuracy, sparsity and probabilistic prediction for classification. In SBL, the regularization priors are automatically determined that avoids an exhaustive hyperparameter selection by cross-validation. However, scalability to large problems is a drawback of SBL because of the inversion of a potentially enormous covariance matrix for updating the regularization priors in every iteration. This paper develops an approximate SBL algorithm called ARP-SBL, where the regularization priors are approximated without inverting the covariance matrix. Therefore, our approach can easily scale up to problems with large data size or feature dimension. It alleviates the long training time and high memory complexity in SBL. Based on ARP-SBL, two scalable nonlinear SBL models: scalable relevance vector machine (ARP-RVM) and scalable sparse Bayesian extreme learning machine (ARP-SBELM) are developed for problems of large data size and large feature size respectively. Experiments on a variety of benchmarks have shown that the proposed models are with competitive accuracy compared to existing methods while i) converging faster; ii) requiring thousands of times less memory; iii) without exhaustive regularized hyperparameter selection; and iv) easily scaling up to large data size and high dimensional features. [ABSTRACT FROM AUTHOR]

This paper investigates a new bacterial colony-based feature selection algorithm to improve the classification accuracy of customers for personalized products recommendation. An attribute learning strategy is developed in this study to update the feature related population. Specifically, the features can be weighted according to their historic contributions to both the individual- and group-based subsets. Additionally, the frequency of appearance is also recorded for the feature candidates to improve the diversity of feature distribution and avoid the over-fitting. Based on the weight-based feature indexes and frequency of appearance records, the performance of feature subsets are enhanced by replacing the features being repeatedly appeared in a same vector. To explore the feasibility of the proposed method for the missing feature problems, the objective of the optimization is to minimize the classification error using the acceptable number of features. K-Nearest Neighbor is employed as the learning technique to cooperate with the proposed feature selection method. The effectiveness of the proposed feature selection method is demonstrated by performing test on the datasets from UCI machine learning repository and real-world data from Amazon customer reviews of products. Compared with other seven feature selections methods, the proposed feature selection algorithm outperforms the other algorithms by achieving higher classification accuracy rate using smaller features. [ABSTRACT FROM AUTHOR]

[Display omitted] • A cosine transform-based similarity calculation method is proposed to cope with information loss in data discretization when the strategy of mutual information is adopted for feature selection. Feature selection (FS) based on mutual information (MI) metrics needs to discretize the data in preprocessing, which is a convenient way to identify correlation between features. However, information loss often occurs in data discretization. In order to solve this information loss problem, this paper proposes a FS algorithm based on cosine similarity coefficient and information measurement criterion (CSCIM_FS). First, the MI between features and tags is calculated, and features are sorted out according to the MI calculated. Then, a feature matrix is constructed to transform the one-dimensional feature sequence into a two-dimensional square matrix. Next, cosine transform is adopted to obtain the high-frequency components of the feature matrix, and sampling is conducted to derive the hash fingerprint of the feature matrix. After that, the similarity between every two features is calculated on the basis of the hash fingerprints of different features. Finally, the feature weight is calculated according to tags, the MI and similarity between features, and a key feature subset is obtained and used to conduct feature selection from the data. The experimental results on several UCI public datasets show that CSCIM_FS algorithm selected a feature subset with high accuracy, and that this algorithm performs better than MIM, CMIM, mRMR and other algorithms. [ABSTRACT FROM AUTHOR]