Search

Many methods exist for solving the problem of evaluating efficiency in different processes. They are divided into two basic groups, parametric and non-parametric methods, which can have significant differences in the results. In this study, the authors consider the process of assessing the business climate depending on realized foreign investments. Due to the expected difference in efficiency assessment using different approaches, the goal of this paper is to create an optimization model of an ensemble for efficiency assessment that uses both types of methods with the aim of creating a symmetrical approach that achieves better results than each type of method individually. The proposed solution simultaneously analyzes the impact of different factors on foreign investments in order to determine the most important factors and thus enable each local government to ensure the best possible efficiency in this process. The innovative idea of this study is in the inclusion of classification and feature selection methods of machine learning to fulfill the set goal. Our research, focused on a specific case study in various cities across the Republic of Serbia, evaluated the effectiveness of that process. This study extends previous research and confirms the published results, highlighting the advantages of the newly proposed model. [ABSTRACT FROM AUTHOR]

When performing classification tasks on high-dimensional data, traditional machine learning algorithms often fail to filter out valid information in the features adequately, leading to low levels of classification accuracy. Therefore, this paper explores the high-dimensional data from both the data feature dimension and the model ensemble dimension. We propose a high-dimensional ensemble learning classification algorithm focusing on feature space reconstruction and classifier ensemble, called the HDELC algorithm. First, the algorithm considers feature space reconstruction and then generates a feature space reconstruction matrix. It effectively achieves feature selection and reconstruction for high-dimensional data. An optimal feature space is generated for the subsequent ensemble of the classifier, which enhances the representativeness of the feature space. Second, we recursively determine the number of classifiers and the number of feature subspaces in the ensemble model. Different classifiers in the ensemble system are assigned mutually exclusive non-intersecting feature subspaces for model training. The experimental results show that the HDELC algorithm has advantages compared with most high-dimensional datasets due to its more efficient feature space ensemble capability and relatively reliable ensemble operation performance. The HDELC algorithm makes it possible to solve the classification problem for high-dimensional data effectively and has vital research and application value. [ABSTRACT FROM AUTHOR]

This survey is devoted to the task of network traffic classification, specifically, to the use of machine learning algorithms in this task. The survey begins with the description of the task, its different statements, and possible real-world applications. It then proceeds to the description of the methods historically used for network traffic classification, as well as their limitations and evolution of traffic making machine learning, which is the main way to solve the problem. The most popular machine learning algorithms used in this task are described and accompanied with examples of research papers that provide insight into their advantages and disadvantages. The problem of feature selection is discussed with subsequent consideration of a more global problem of acquiring a suitable dataset for network traffic classification; examples of popular datasets and their descriptions are provided. The paper concludes with an overview of some current problems in this field of research: model training and comparison, user data protection, and network traffic volatility. [ABSTRACT FROM AUTHOR]

In the development of anti-breast cancer drugs, the quantitative structure-activity relationship model of compounds is usually used to select potential active compounds. However, the existing methods often have problems such as low model prediction performance, lack of overall consideration of the biological activity and related properties of compounds, and difficulty in directly selection candidate drugs. Therefore, this paper constructs a complete set of compound selection framework from three aspects: feature selection, relationship mapping and multi-objective optimization problem solving. In feature selection part, a feature selection method based on unsupervised spectral clustering is proposed. The selected features have more comprehensive information expression ability. In the relationship mapping part, a variety of machine learning algorithms are used for comparative experiments. Finally, the CatBoost algorithm is selected to perform the relationship mapping between each other, and better prediction performance is achieved. In the multi-objective optimization part, based on the analysis of the conflict relationship between the objectives, the AGE-MOEA algorithm is improved and used to solve this problem. Compared with various algorithms, the improved algorithm has better search performance. [ABSTRACT FROM AUTHOR]

We consider a class of problems where the objective function is the sum of a smooth function and a composition of nonconvex and nonsmooth functions. Such optimization problems arise frequently in machine learning and data processing. The proximal iteratively reweighted method has been widely used and popularized in solving these problems. In this paper, we develop an extrapolated proximal iteratively reweighted method that incorporates two different flexible inertial steps at each iteration. We first prove the subsequential convergence of the proposed method under parameter constraints. Moreover, if the objective function satisfies the Kurdyka-Łojasiewicz property, the global convergence of the new method is established. In addition, we analyze the local convergence rate by making assumptions on the Kurdyka-Łojasiewicz exponent of the objective function. Finally, numerical results on l p minimization and feature selection problems are reported to show the effectiveness and superiority of the proposed algorithm. [ABSTRACT FROM AUTHOR]

With unlabeled and high-dimensional data explosion, unsupervised feature selection has become an essential step in many machine learning and data mining tasks. Many dictionary learning based models have been successfully developed for unsupervised feature selection in recent years. These models learn an over-complete dictionary to investigate more data distribution information. However, over-complete dictionary learning will generate redundancy in the latent representations for data. Moreover, if data contain noise, dictionary learning will also yield noise in the latent representations. In this paper, we propose a novel unsupervised feature selection framework, named dictionary learning for unsupervised feature selection via dual sparse regression. In this model, dictionary learning is first embedded into a sparse regression to learn an over-complete dictionary with sparse representations for data, in which the redundancy and noise are eliminated. The data are then projected to the representations to evaluate the significance of features using the other sparse regression. We also offer an efficient algorithm to solve this problem and theoretically analyze its convergence and computational complexity, which is proportional to the data dimensionality. Finally, the evaluation results with the k-means task utilizing the selected features on 9 benchmark datasets demonstrate the superiority of our approach in terms of effectiveness and efficiency. [ABSTRACT FROM AUTHOR]

A malicious URL is a link that is created to spread spams, phishing, malware, ransomware, spyware, etc. A user may download malware that can adversely affect the computer by clicking on an infected URL, or might be convinced to provide confidential information to a fraudulent website causing serious losses. These threats must be identified and handled in a decent time and in an effective way. Detection is traditionally done through the blacklist usage method, which relies on keyword matching with previously known malicious domain names stored in a repository. This method is fast and easy to implement, with the advantage of having low false-positive rates regarding previously recognized malicious URLs. However, this method cannot recognize newly created malicious URLs. To solve this problem, many machine-learning models have been used. In this paper, we introduce an effective machine learning approach that uses an ensemble learner algorithm called AdaBoost (Adaptive Boosting), combined with different algorithms that enhance detection. For datasets filtration, we used CfsSubsetEval technique, which is an algorithm that searches for a subset of features that work well together. Datasets were collected from the UNB repository; divided into four categories: spam, phishing, malware, and defacement URLs; combined with benign URLs, dataset content is based on lexical features. The experimental results indicate that the proposed approach was successful in enhancing the detection accuracy of malicious URLs with less false-positive rates for all experimental algorithms. [ABSTRACT FROM AUTHOR]

This paper is devoted to the problem of class imbalance in machine learning, focusing on the intrusion detection of rare classes in computer networks. The problem of class imbalance occurs when one class heavily outnumbers examples from the other classes. In this paper, we are particularly interested in classifiers, as pattern recognition and anomaly detection could be solved as a classification problem. As still a major part of data network traffic of any organization network is benign, and malignant traffic is rare, researchers therefore have to deal with a class imbalance problem. Substantial research has been undertaken in order to identify these methods or data features that allow to accurately identify these attacks. But the usual tactic to deal with the imbalance class problem is to label all malignant traffic as one class and then solve the binary classification problem. In this paper, however, we choose not to group or to drop rare classes but instead investigate what could be done in order to achieve good multi-class classification efficiency. Rare class records were up-sampled using SMOTE method (Chawla et al., 2002) to a preset ratio targets. Experiments with the 3 network traffic datasets, namely CIC-IDS2017, CSE-CIC-IDS2018 (Sharafaldin et al., 2018) and LITNET-2020 (Damasevicius et al., 2020) were performed aiming to achieve reliable recognition of rare malignant classes available in these datasets. Popular machine learning algorithms were chosen for comparison of their readiness to support rare class detection. Related algorithm hyper parameters were tuned within a wide range of values, different data feature selection methods were used and tests were executed with and without over-sampling to test the multiple class problem classification performance of rare classes. Machine learning algorithms ranking based on Precision, Balanced Accuracy Score, G ¯ , and prediction error Bias and Variance decomposition, show that decision tree ensembles (Adaboost, Random Forest Trees and Gradient Boosting Classifier) performed best on the network intrusion datasets used in this research. [ABSTRACT FROM AUTHOR]

Nature is a great source of inspiration for solving complex problems in real-world. In this paper, a hybrid nature-inspired algorithm is proposed for feature selection problem. Traditionally, the real-world datasets contain all kinds of features informative as well as non-informative. These features not only increase computational complexity of the underlying algorithm but also deteriorate its performance. Hence, there an urgent need of feature selection method that select an informative subset of features from high dimensional without compromising the performance of the underlying algorithm. In this paper, we select an informative subset of features and perform cluster analysis by employing a cross breed approach of binary particle swarm optimization (BPSO) and sine cosine algorithm (SCA) named as hybrid binary particle swarm optimization and sine cosine algorithm (HBPSOSCA). Here, we employ a V-shaped transfer function to compute the likelihood of changing position for all particles. First, the effectiveness of the proposed method is tested on ten benchmark test functions. Second, the HBPSOSCA is used for data clustering problem on seven real-life datasets taken from the UCI machine learning store and gene expression model selector. The performance of proposed method is tested in comparison to original BPSO, modified BPSO with chaotic inertia weight (C-BPSO), binary moth flame optimization algorithm, binary dragonfly algorithm, binary whale optimization algorithm, SCA, and binary artificial bee colony algorithm. The conducted analysis demonstrates that the proposed method HBPSOSCA attain better performance in comparison to the competitive methods in most of the cases. [ABSTRACT FROM AUTHOR]

Transfer learning is proposed to solve a general problem in practical applications faced by traditional machine learning methods, that is, the training and test data have different distributions. This paper provides a novel transfer subspace learning method combining low-rank representation (LRR) and feature selection for unsupervised domain adaptation. The core of the proposed method is to map both the source and target data into a latent subspace by a projection such that the discrepancy between domains is reduced. Specifically, by using LRR, a low-rank constraint is imposed on the reconstruction coefficient matrix, and thus the global structure of data can be preserved. Moreover, a structured sparsity-inducing norm based regularization term is introduced into the domain adaptation, which leads to imposing a row-sparsity constraint on the projection matrix. This constraint can enforce rows of the projection matrix corresponding to inessential feature attributes to be all zeros, and thus select relevant features across two domains. As a result, the proposed method has good interpretability and can adaptively perform feature selection. Furthermore, taking into account that the projected samples should be close to each other in the shared subspace if they belong to the same class, regardless of which domain they originally come from, we introduce graph embedding to characterize the local manifold structures of data so as to preserve the relationships between examples in the subspace. Finally, we mathematically formulate the proposed method and derive an iterative algorithm to solve the corresponding problem. The exhaustive experimental evaluations on public datasets confirm the effectiveness of the proposed method in comparison with several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Selecting relevant feature subsets is vital in machine learning, and multiclass feature selection is harder to perform since most classifications are binary. The feature selection problem aims at reducing the feature set dimension while maintaining the performance model accuracy. Datasets can be classified using various methods. Nevertheless, metaheuristic algorithms attract substantial attention to solving different problems in optimization. For this reason, this paper presents a systematic survey of literature for solving multiclass feature selection problems utilizing metaheuristic algorithms that can assist classifiers selects optima or near optima features faster and more accurately. Metaheuristic algorithms have also been presented in four primary behavior-based categories, i.e., evolutionary-based, swarm-intelligence-based, physics-based, and human-based, even though some literature works presented more categorization. Further, lists of metaheuristic algorithms were introduced in the categories mentioned. In finding the solution to issues related to multiclass feature selection, only articles on metaheuristic algorithms used for multiclass feature selection problems from the year 2000 to 2022 were reviewed about their different categories and detailed descriptions. We considered some application areas for some of the metaheuristic algorithms applied for multiclass feature selection with their variations. Popular multiclass classifiers for feature selection were also examined. Moreover, we also presented the challenges of metaheuristic algorithms for feature selection, and we identified gaps for further research studies. [ABSTRACT FROM AUTHOR]

Iris image-based biometric systems are commonly used in applications that demand security, authentication, recognition and faster login access. In solving these real-time problems, the impact of soft computing techniques which employ cognitive skills is very high. Although this system has been commercialized, the scope for improvement is still plenty. This paper introduces the reader to different segments of an iris recognition system and reviews the techniques involved with each segment. It reports on how research articles validate the robustness of an iris-based recognition system. As these systems are fallible, it also shows the vulnerabilities associated with each segment and provides insights to develop much better intelligent and robust techniques which will make the system more accurate. This paper also shows that in spite of versatility of soft computing techniques, it is not fully exploited for iris recognition systems. The present challenges and directions for future research are also discussed. [ABSTRACT FROM AUTHOR]

Dimension reduction is a preprocessing step in machine learning for eliminating undesirable features and increasing learning accuracy. In order to reduce the redundant features, there are data representation methods, each of which has its own advantages. On the other hand, big data with imbalanced classes is one of the most important issues in pattern recognition and machine learning. In this paper, a method is proposed in the form of a cost-sensitive optimization problem which implements the process of selecting and extracting the features simultaneously. The feature extraction phase is based on reducing error and maintaining geometric relationships between data by solving a manifold learning optimization problem. In the feature selection phase, the cost-sensitive optimization problem is adopted based on minimizing the upper limit of the generalization error. Finally, the optimization problem which is constituted from the above two problems is solved by adding a cost-sensitive term to create a balance between classes without manipulating the data. To evaluate the results of the feature reduction, the multi-class linear SVM classifier is used on the reduced data. The proposed method is compared with some other approaches on 21 datasets from the UCI learning repository, microarrays and high-dimensional datasets, as well as imbalanced datasets from the KEEL repository. The results indicate the significant efficiency of the proposed method compared to some similar approaches. [ABSTRACT FROM AUTHOR]

Algorithmic trading is a common topic researched in the neural network due to the abundance of data available. It is a phenomenon where an approximately linear relationship exists between two or more independent variables. It is especially prevalent in financial data due to the interrelated nature of the data. The existing feature selection methods are not efficient enough in solving such a problem due to the potential loss of essential and relevant information. These methods are also not able to consider the interaction between features. Therefore, we proposed two improvements to apply to the Long Short-Term Memory neural network (LSTM) in this study. It is the Multicollinearity Reduction Module (MRM) based on correlation-embedded attention to mitigate multicollinearity without removing features. The motivation of the improvements is to allow the model to predict using the relevance and redundancy within the data. The first contribution of the paper is allowing a neural network to mitigate the effects of multicollinearity without removing any variables. The second contribution is improving trading returns when our proposed mechanisms are applied to an LSTM. This study compared the classification performance between LSTM models with and without the correlation-embedded attention module. The experimental result reveals that a neural network that can learn the relevance and redundancy of the financial data to improve the desired classification performance. Furthermore, the trading returns of our proposed module are 46.82% higher without sacrificing training time. Moreover, the MRM is designed to be a standalone module and is interoperable with existing models. [ABSTRACT FROM AUTHOR]

This paper proposes a new pre-processing technique to separate the most effective features from those that might deteriorate the performance of the machine learning classifiers in terms of computational costs and classification accuracy because of their irrelevance, redundancy, or less information; this pre-processing process is often known as feature selection. This technique is based on adopting a new optimization algorithm known as generalized normal distribution optimization (GNDO) supported by the conversion of the normal distribution to a binary one using the arctangent transfer function to convert the continuous values into binary values. Further, a novel restarting strategy (RS) is proposed to preserve the diversity among the solutions within the population by identifying the solutions that exceed a specific distance from the best-so-far and replace them with the others created using an effective updating scheme. This strategy is integrated with GNDO to propose another binary variant having a high ability to preserve the diversity of the solutions for avoiding becoming stuck in local minima and accelerating convergence, namely improved GNDO (IGNDO). The proposed GNDO and IGNDO algorithms are extensively compared with seven state-of-the-art algorithms to verify their performance on thirteen medical instances taken from the UCI repository. IGNDO is shown to be superior in terms of fitness value and classification accuracy and competitive with the others in terms of the selected features. Since the principal goal in solving the FS problem is to find the appropriate subset of features that maximize classification accuracy, IGNDO is considered the best. [ABSTRACT FROM AUTHOR]

Multilabel classification has become increasingly important for various use cases. Amongst the existing multilabel classification methods, problem transformation approaches, such as Binary Relevance, Pruned Problem Transformation, and Classifier Chains, are some of the most popular, since they break a global multilabel classification problem into a set of smaller binary or multiclass classification problems. Transformation methods enable the use of two different feature selection approaches: local, where the selection is performed independently for each of the transformed problems, and global, where the selection is performed on the original dataset, meaning that all local classifiers work on the same set of features. While global methods have been widely researched, local methods have received little attention so far. In this paper, we compare those two strategies on one of the most straight forward transformation approaches, i.e., Binary Relevance. We empirically compare their performance on various flat and hierarchical multilabel datasets of different application domains. We show that local outperforms global feature selection in terms of classification accuracy, without drawbacks in runtime performance. [ABSTRACT FROM AUTHOR]

Feature selection is one of the main steps in preprocessing data in machine learning, and its goal is to reduce features by removing additional and noisy features. Feature selection methods and feature reduction in a dataset must consider the accuracy of the classifying algorithms. Meta-heuristic algorithms serve as the most successful and promising methods to solve this problem. Symbiotic Organisms Search (SOS) is one of the most successful meta-heuristic algorithms inspired by organisms' interaction in nature called mutualism, commensalism, and parasitism. In this paper, three SOS-based binary approaches are offered to solve the feature selection problem. In the first and second approaches, several S-shaped transfer functions and several Chaotic Tent Function-based V-shaped transfer functions called BSOSST and BSOSVT are used to make the binary SOS (BSOS). In the third approach, an advanced BSOS based on changing SOS and the chaotic Tent function operators called EBCSOS is provided. The EBCSOS algorithm uses the chaotic Tent function and the Gaussian mutation to increase usefulness and exploration. Moreover, two new operators, i.e., BMPT and BCPT, are suggested to make the commensalism and mutualism stage binary based on a chaotic function to solve the feature selection problem. Finally, the proposed BSOSST and BSOSVT methods and the advanced version of EBCSOS were implemented on 25 datasets than the basic algorithm's binary meta-heuristic algorithms. Various experiments demonstrated that the proposed EBCSOS algorithm outperformed other methods in terms of several features and accuracy. To further confirm the proposed EBCSOS algorithm, the problem of detecting spam E-mails was applied, with the results of this experiment indicating that the proposed EBCSOS algorithm significantly improved the accuracy and speed of all categories in detecting spam E-mails. [ABSTRACT FROM AUTHOR]

Feature selection is a crucial part of machine learning and pattern recognition, which aims at selecting a subset of informative features from the original dataset. Because of label information, supervised feature selection performs better than unsupervised feature selection without label information. However, in the presence of a small number of labeled data and a large number of unlabeled data, it is challenging for supervised feature selection methods to select relevant features. In this paper, we propose three neurodynamics-driven holistic approaches to semi-supervised feature selection via semi-supervised feature redundancy minimization and semi-supervised feature relevancy maximization. We first define information-theoretic semi-supervised similarity coefficient matrix and semi-supervised feature relevancy vector based on multi-information, unsupervised symmetric uncertainty, and entropy to measure feature redundancy and relevancy. We then formulate a fractional programming problem and an iteratively weighted quadratic programming problem based on the semi-supervised similarity coefficient matrix and semi-supervised feature relevancy vector for semi-supervised feature selection. To solve the formulated problems, we delineate three neurodynamic optimization approaches based on two projection neural networks. We elaborate on the experimental results on six benchmark datasets to demonstrate the superior classification performance of the proposed neurodynamic approaches against six existing supervised and semi-supervised feature selection methods. [ABSTRACT FROM AUTHOR]

In this paper, we provide external image features and use the internal attention mechanism to solve the VQA problem given a dataset of textual questions and related images. Most previous models for VQA use a pair of images and questions as input. In addition, the model adopts a question-oriented attention mechanism to extract the features of the entire image and then perform feature fusion. However, the shortcoming of these models is that they cannot effectively eliminate the irrelevant features of the image. In addition, the problem-oriented attention mechanism lacks in the mining of image features, which will bring in redundant image features. In this paper, we propose a VQA model based on adversarial learning and bidirectional attention. We exploit external image features that are not related to the question to form an adversarial mechanism to boost the accuracy of the model. Target detection is performed on the image—that is, the image-oriented attention mechanism. The bidirectional attention mechanism is conducive to promoting model attention and eliminating interference. Experimental results are evaluated on benchmark datasets, and our model performs better than other models based on attention methods. In addition, the qualitative results show the attention maps on the images and leads to predicting correct answers. [ABSTRACT FROM AUTHOR]

Feature selection is a crucial step in data processing and machine learning. While many greedy and sequential feature selection approaches are available, a holistic neurodynamics approach to supervised feature selection is recently developed via fractional programming by minimizing feature redundancy and maximizing relevance simultaneously. In view that the gradient of the fractional objective function is also fractional, alternative problem formulations are desirable to obviate the fractional complexity. In this paper, the fractional programming problem formulation is equivalently reformulated as bilevel and bilinear programming problems without using any fractional function. Two two-timescale projection neural networks are adapted for solving the reformulated problems. Experimental results on six benchmark datasets are elaborated to demonstrate the global convergence and high classification performance of the proposed neurodynamic approaches in comparison with six mainstream feature selection approaches. [ABSTRACT FROM AUTHOR]

In solving classification problems in the field of machine learning and pattern recognition, the pre-processing of data is particularly important. The processing of high-dimensional feature datasets increases the time and space complexity of computer processing and reduces the accuracy of classification models. Hence, the proposal of a good feature selection method is essential. This paper presents a new algorithm for solving feature selection, retaining the selection and mutation operators from traditional genetic algorithms. On the one hand, the global search capability of the algorithm is ensured by changing the population size, on the other hand, finding the optimal mutation probability for solving the feature selection problem based on different population sizes. During the iteration of the algorithm, the population size does not change, no matter how many transformations are made, and is the same as the initialized population size; this spatial invariance is physically defined as symmetry. The proposed method is compared with other algorithms and validated on different datasets. The experimental results show good performance of the algorithm, in addition to which we apply the algorithm to a practical Android software classification problem and the results also show the superiority of the algorithm. [ABSTRACT FROM AUTHOR]

As news events on the same subject occur, our knowledge about the subject will accumulate and become more comprehensive. In this paper, we formally define the problem of incremental knowledge learning from similar news events on the same subject, where each event consists of a set of news articles reporting about it. The knowledge is represented by a topic hierarchy presenting topics at different levels of granularity. Though topic (hierarchy) mining from text has been researched a lot, incremental learning from similar events remains under developed. In this paper, we propose a scalable two-phase framework to incrementally learn a topic hierarchy for a subject from events on the subject as the events occur. First, we recursively construct a topic hierarchy for each event based on a novel topic model considering the named entities and entity types in news articles. Second, we incrementally merge the topic hierarchies through top-down hierarchical topic alignment. Extensive experimental results on real datasets demonstrate the effectiveness and efficiency of the proposed framework in terms of both qualitative and quantitative measures. [ABSTRACT FROM AUTHOR]

Class binarization strategies decompose the original multi-class problem into several binary sub-problems. One versus One (OVO) is one of the most popular class binarization techniques, which considers every pair of classes as a different sub-problem. Usually, the same classifier is applied to every sub-problem and then all the outputs are combined by some voting scheme. In this paper we present a novel idea where for each test instance we try to assign the best classifier in each sub-problem of OVO. To do so, we have used two simple Dynamic Classifier Selection (DCS) strategies that have not been yet used in this context. The two DCS strategies use K-NN to obtain the local region of the test-instance, and the classifier that performs the best for those instances in the local region, is selected to classify the new test instance. The difference between the two DCS strategies remains in the weight of the instance. In this paper we have also proposed a novel approach in those DCS strategies. We propose to use the K-Nearest Neighbor Equality (K-NNE) method to obtain the local accuracy. K-NNE is an extension of K-NN in which all the classes are treated independently: the K nearest neighbors belonging to each class are selected. In this way all the classes take part in the final decision. We have carried out an empirical study over several UCI databases, which shows the robustness of our proposal. [ABSTRACT FROM AUTHOR]

Feature selection is the problem of identifying a subset of the most relevant features in the context of model construction. This problem has been well studied and plays a vital role in machine learning. In this paper we present three randomized algorithms for feature selection. They are generic in nature and can be applied for any learning algorithm. Proposed algorithms can be thought of as a random walk in the space of all possible subsets of the features. We demonstrate the generality of our approaches using three different applications. The simulation results show that our feature selection algorithms outperforms some of the best known algorithms existing in the current literature. [ABSTRACT FROM AUTHOR]

As the high-dimensional heterogeneous visual features extracted from images are intrinsically embedded in a non-linear space, some kernel methods such as SVM have been proposed to solve this problem. Since different kinds of heterogeneous features in images have different intrinsic discriminative powers for image understanding, how to enforce grouping sparsity penalty to effectively select out discriminative heterogeneous visual features is critical for image understanding. Most existing approaches are using a convex penalty for feature selection, which easily leads to inconsistent selection. To guarantee a consistent selection for heterogeneous features embedded in a non-linear space, this paper proposes a new approach called MKL-NOVA (Multiple Kernel Learning with NOn-conVex group spArsity). Because MKL-NOVA conducts a non-convex penalty for the selection of groups of features, it achieves the consistent selection. Furthermore, considering the contextual correlation between multi labels, sparse canonical correlation analysis is conducted to boost the image annotation performance by MKL-NOVA. We have demonstrated the superior performance of MKL-NOVA via two experiments in the paper. First, we showed that MKL-NOVA converges to the true underlying model by using a ground-truth-available generative-model simulation. Second, we compare the proposed MKL-NOVA and the state-of-the-art approaches which showed that MKL-NOVA achieved the best performance. [ABSTRACT FROM AUTHOR]

Feature selection methods are widely used in machine learning tasks to reduce the dimensionality and improve the performance of the models. However, traditional feature selection methods based on regression often suffer from a lack of robustness and generalization ability and are easily affected by outliers in the data. To address this problem, we propose a robust feature selection method based on sparse regression. This method uses a non-square form of the L2,1 norm as both the loss function and regularization term, which can effectively enhance the model's resistance to outliers and achieve feature selection simultaneously. Furthermore, to improve the model's robustness and prevent overfitting, we add an elastic variable to the loss function. We design two efficient convergent iterative processes to solve the optimization problem based on the L2,1 norm and propose a robust joint sparse regression algorithm. Extensive experimental results on three public datasets show that our feature selection method outperforms other comparison methods. [ABSTRACT FROM AUTHOR]

Feature selection is one of the most important tasks in many machine learning and data mining problems. Due to the increasing size of the problems, removing useless, erroneous or noisy features is frequently an initial step that is performed before other data mining algorithms are applied. The aim is to reproduce, or even improve, the performance of the data mining algorithm when all the features are used. Furthermore, the selection of the most relevant features may offer the expert valuable information about the problem to be solved. Over the past few decades, many different feature selection algorithms have been proposed, each with its own strengths and weaknesses. However, as in the case of classification, it is unlikely that a single feature selection algorithm would be able to achieve good results across many different datasets and application fields. Furthermore, when we are dealing with thousands of features, the most powerful feature selection methods are frequently too time consuming to be applied. In classification, one of the most successful ways of consistently improving the performance of a single weak learner is to construct ensembles using boosting methods. In this paper, we propose a general framework for feature selection boosting in the same way boosting is applied to classification. The proposed approach opens a new field of research in which to apply the many techniques developed for boosting classifiers. Using 120 datasets, the experiments reported show a clear improvement in several state-of-the-art feature selection algorithms using the proposed methodology. [ABSTRACT FROM AUTHOR]

Learning useful representations from original features is a key issue in classification tasks. Stacked autoencoders (SAEs) are easy to understand and realize, and they are powerful tools that learn deep features from original features, so they are popular for classification problems. The deep features can further combine the original features to construct more representative features for classification. However, existing SAEs do not consider the original features within the network structure and during training, so the deep features have low complementarity with the original features. To solve the problem, this paper proposes an embedded stacked group sparse autoencoder (ESGSAE) for more effective feature learning. Different from traditional stacked autoencoders, the ESGSAE model considers the complementarity between the original feature and the hidden outputs by embedding the original features into hidden layers. To alleviate the impact of the small sample problem on the generalization of the proposed ESGSAE model, an L 1 regularization-based feature selection strategy is designed to further improve the feature quality. After that, an ensemble model with support vector machine (SVM) and weighted local discriminant preservation projection (w_LPPD) is designed to further enhance the feature quality. Based on the designs above, an embedded stacked group sparse autoencoder ensemble with L 1 regularization and manifold reduction is proposed to obtain deep features with high complementarity in the context of the small sample problem. At the end of this paper, several representative public datasets are used for verification of the proposed algorithm. The results demonstrate that the ESGSAE ensemble model with L 1 regularization and manifold reduction yields superior performance compared to other existing and state-of-the-art feature learning algorithms, including some representative deep stacked autoencoder methods. Specifically, compared with the original features, the representative feature extraction algorithms and the improved autoencoders, the algorithm proposed in this paper can improve the classification accuracy by up to 13.33%, 7.33%, and 9.55%, respectively. The data and codes can be found in: https://share.weiyun.com/Jt7qeORm • A hybrid feature is embedded into the training process to construct a novel deep model. • A group sparsity constraint is introduced to obtain the sparse representations. • The ESGSAE ensemble model is constructed to obtain high complementary features. • A three-step feature learning mechanism is realized. [ABSTRACT FROM AUTHOR]

This paper presents distress classification of class-imbalanced inspection data via correlation-maximizing weighted extreme learning machine (CMWELM). For distress classification, it is necessary to extract semantic features that can effectively distinguish multiple kinds of distress from a small amount of class-imbalanced data. In recent machine learning techniques such as general deep learning methods, since effective feature transformation from visual features to semantic features can be realized by using multiple hidden layers, a large amount of training data are required. However, since the amount of training data of civil structures becomes small, it becomes difficult to perform successful transformation by using these multiple hidden layers. On the other hand, CMWELM consists of two hidden layers. The first hidden layer performs feature transformation, which can directly extract the semantic features from visual features, and the second hidden layer performs classification with solving the class-imbalanced problem. Specifically, in the first hidden layer, the feature transformation is realized by using projections obtained by maximizing the canonical correlation between visual and text features as weight parameters of the hidden layer without designing multiple hidden layers. Furthermore, the second hidden layer enables successful training of our classifier by using weighting factors concerning the class-imbalanced problem. Consequently, CMWELM realizes accurate distress classification from a small amount of class-imbalanced data. [ABSTRACT FROM AUTHOR]

Finite mixture of Gaussian regression (FMR) is a widely-used modeling technique in supervised learning problems. In cases where the number of features is large, feature selection is desirable to enhance model interpretability and to avoid overfitting. In this paper, we propose a robust feature selection method via l 2,1 -norm penalized maximum likelihood estimation (MLE) in FMR, with extension to sparse l 2,1 penalty by combining l 1 -norm with l 2,1 -norm for increasing flexibility. To solve the non-convex and non-smooth problem of (sparse) penalized MLE in FMR, we develop an new EM-based algorithm for numerical optimization, with combination of block coordinate descent and majorizing-minimization scheme in M-step. We finally apply our method in six simulations and one real dataset to demonstrate its superior performance. [ABSTRACT FROM AUTHOR]

Feature selection is an effective pre-processing step to remove possible redundant and irrelevant features for various machine learning paradigms, which can help build a more understandable machine learning based component in hybrid expert systems. Single-label quadratic programming feature selection (QPFS) model is formulated as a QP problem with a unit simplex constraint to both minimize feature-feature redundancy and maximize feature-label relevance simultaneously. Since its redundancy matrix is desired not to be positive semi-definite and then estimated by Nystrom low-rank approximation method, its performance depends greatly on under-sampling rate and random permutation. In this paper, without any approximation, we extend this model to construct a regularized version (rQPFS), resulting in a strictly convex QP problem, to achieve a globally optimal subset of features. To reduce the increment of computational time for entire redundancy matrix, Frank–Wolfe method with a sub-linear convergence rate and its special cases for strictly convex function and unit simplex constraint are applied to solve our rQPFS efficiently. Furthermore, to tackle multi-label FS paradigm, pruned problem transformation trick is used to evaluate feature-label relevance to describe label correlations sufficiently. The detailed experimental study on eight data sets shows that our proposed method performs the best, compared with six state-of-the-art multi-label FS methods, according to six instance-based performance evaluation measures. [ABSTRACT FROM AUTHOR]

Micro-array expression data classification is a challenging task for machine learning researchers due to its tremendous number of features and also the limited size of sample. Selection of feature has been examined as an existing standard in consideration to minimize the input dimensionality at the same time correcting the classification performance. Feature selection procedure helps us in minimizing the computation time, improving performance of prediction and good understanding of micro-array expression data. This paper focuses on outlining the various feature selection procedure developed so far, we have presented an overview of various methods present in literature and focusing mainly on filter, wrapper and embedded methods. [ABSTRACT FROM AUTHOR]

We propose a cost-effective system for the determination of metal ion concentration in water, addressing a central issue in water resources management. The system combines novel luminometric label array technology with a machine learning algorithm that selects a minimal number of array reagents (modulators) and liquid sample dilutions, such that enable accurate quantification. The algorithm is able to identify the optimal modulators and sample dilutions leading to cost reductions since less manual labour and resources are needed. Inferring the ion detector involves a unique type of a structured feature selection problem, which we formalize in this paper. We propose a novel Cartesian greedy forward feature selection algorithm for solving the problem. The novel algorithm was evaluated in the concentration assessment of five metal ions and the performance was compared to two known feature selection approaches. The results demonstrate that the proposed system can assist in lowering the costs with minimal loss in accuracy. [ABSTRACT FROM AUTHOR]

SARS-COV-2 is prevalent all over the world, causing more than six million deaths and seriously affecting human health. At present, there is no specific drug against SARS-COV-2. Protein phosphorylation is an important way to understand the mechanism of SARS -COV-2 infection. It is often expensive and time-consuming to identify phosphorylation sites with specific modified residues through experiments. A method that uses machine learning to make predictions about them is proposed. As all the methods of extracting protein sequence features are knowledge-driven, these features may not be effective for detecting phosphorylation sites without a complete understanding of the mechanism of protein. Moreover, redundant features also have a great impact on the fitting degree of the model. To solve these problems, we propose a feature selection method based on ensemble learning, which firstly extracts protein sequence features based on knowledge, then quantifies the importance score of each feature based on data, and finally uses the subset of important features as the final features to predict phosphorylation sites. [ABSTRACT FROM AUTHOR]

One of the serious challenges in machine learning and pattern recognition is to transfer knowledge from related but different domains to a new unlabeled domain. Feature selection with maximum mean discrepancy (f-MMD) is a novel and effective approach to transfer knowledge from source domain (training set) into target domain (test set) where training and test sets are drawn from different distributions. However, f-MMD has serious challenges in facing datasets with large number of samples and features. Moreover, f-MMD ignores the feature-label relation in finding the reduced representation of dataset. In this paper, we exploit jointly transfer learning and class discrimination to cope with domain shift problem on which the distribution difference is considerably large. We therefore put forward a novel transfer learning and class discrimination approach, referred to as RandOm k-samplesets feature Weighting Approach (ROWA). Specifically, ROWA reduces the distribution difference across domains in an unsupervised manner where no label is available in the test set. Moreover, ROWA exploits feature-label relation to separate various classes alongside the domain transfer, and augments the relation of selected features and source domain labels. In this work, we employ disjoint/overlapping small-sized samplesets to iteratively converge to final solution. Employment of local sets along with a novel optimization problem constructs a robust and effective reduced representation for adaptation across domains. Extensive experiments on real and synthetic datasets verify that ROWA can significantly outperform state-of-the-art transfer learning approaches. [ABSTRACT FROM AUTHOR]

Multi-label learning deals with data associated with a set of labels simultaneously. Like traditional single-label learning, the high-dimensionality of data is a stumbling block for multi-label learning. In this paper, we first introduce the margin of instance to granulate all instances under different labels, and three different concepts of neighborhood are defined based on different cognitive viewpoints. Based on this, we generalize neighborhood information entropy to fit multi-label learning and propose three new measures of neighborhood mutual information. It is shown that these new measures are a natural extension from single-label learning to multi-label learning. Then, we present an optimization objective function to evaluate the quality of the candidate features, which can be solved by approximating the multi-label neighborhood mutual information. Finally, extensive experiments conducted on publicly available data sets verify the effectiveness of the proposed algorithm by comparing it with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Ensemble learning is a system that improves the performance and robustness of the classification problems. How to combine the outputs of base classifiers is one of the fundamental challenges in ensemble learning systems. In this paper, an optimized Static Ensemble Selection (SES) approach is first proposed on the basis of NSGA-II multi-objective genetic algorithm (called SES-NSGAII), which selects the best classifiers along with their combiner, by simultaneous optimization of error and diversity objectives. In the second phase, the Dynamic Ensemble Selection-Performance (DES-P) is improved by utilizing the first proposed method. The second proposed method is a hybrid methodology that exploits the abilities of both SES and DES approaches and is named Improved DES-P (IDES-P). Accordingly, combining static and dynamic ensemble strategies as well as utilizing NSGA-II are the main contributions of this research. Findings of the present study confirm that the proposed methods outperform the other ensemble approaches over 14 datasets in terms of classification accuracy. Furthermore, the experimental results are described from the view point of Pareto front with the aim of illustrating the relationship between diversity and the over-fitting problem. [ABSTRACT FROM AUTHOR]

Feature selection (FS) is recognized as one of the majority public and challenging problems in the Machine Learning domain. FS can be examined as an optimization problem that needs an effective optimizer to determine its optimal subset of more informative features. This paper proposes a wrapper FS method that combines chaotic maps (CMs) and binary Group Search Optimizer (GSO) called CGSO, which is used to solve the FS problem. In this method, five chaotic maps are incorporated with the GSO algorithm's main procedures, namely, Logistic, Piecewise, Singer, Sinusoidal, and Tent. The GSO algorithm is used as a search strategy, while k-NN is employed as an induction algorithm. The objective function is to integrate three main objectives: maximizing the classification accuracy value, minimizing the number of selected features, and minimizing the complexity of generated k-NN models. To evaluate the proposed methods' performance, twenty well-known UCI datasets are used and compared with other well-known published methods in the literature. The obtained results reveal the superiority of the proposed methods in outperforming other well-known methods, especially when using binary GSO with Tent CM. Finally, it is a beneficial method to be utilized in systems that require FS pre-processing. • A novel feature selection method is proposed for classification task. • Five Chaotic Group Search Optimizers are investigated. • Twenty UCI datasets were used in the experiments. • Our methods got a higher accuracy results using a fewer number of features. [ABSTRACT FROM AUTHOR]

Corporate financial distress prediction research has been ongoing for more than half a century, during which many models have emerged, among which ensemble learning algorithms are the most accurate. Most of the state-of-the-art methods of recent years are based on gradient boosted decision trees. However, most of them do not consider using feature importance for feature selection, and a few of them use the feature importance method with bias, which may not reflect the true importance of features. To solve this problem, a heuristic algorithm based on permutation importance (PIMP) is proposed to modify the biased feature importance measure in this paper. This method ranks and filters the features used by machine learning models, which not only improves accuracy but also makes the results more interpretable. Based on financial data from 4,167 listed companies in China between 2001 and 2019, the experiment shows that compared with using the random forest (RF) wrapper method alone, the bias in feature importance is indeed corrected by combining the PIMP method. After the redundant features are removed, the performance of most machine learning models is improved. The PIMP method is a promising addition to the existing financial distress prediction methods. Moreover, compared with traditional statistical learning models and other machine learning models, the proposed PIMP-XGBoost offers higher prediction accuracy and clearer interpretation, making it suitable for commercial use. • The model Combines a corrected feature selection measure and XGBoost. • Permutation importance can correct the bias of feature importance. • The model is validated on Chinese listed companies datasets over five metrics. • The model is proved to outperform several benchmark techniques. • The feature importance and partial dependence plot enhance model interpretation. [ABSTRACT FROM AUTHOR]

Abstract: Traditional cross-domain learning methods transfer learning from a source domain to a target domain. In this paper, we propose the multiple-domain learning problem for several equally treated domains. The multiple-domain learning problem assumes that samples from different domains have different distributions, but share the same feature and class label spaces. Each domain could be a target domain, while also be a source domain for other domains. A novel multiple-domain representation method is proposed for the multiple-domain learning problem. This method is based on nonnegative matrix factorization (NMF), and tries to learn a basis matrix and coding vectors for samples, so that the domain distribution mismatch among different domains will be reduced under an extended variation of the maximum mean discrepancy (MMD) criterion. The novel algorithm — multiple-domain NMF (MDNMF) — was evaluated on two challenging multiple-domain learning problems — multiple user spam email detection and multiple-domain glioma diagnosis. The effectiveness of the proposed algorithm is experimentally verified. [Copyright &y& Elsevier]

Abstract: Since given classification data often contains redundant, useless or misleading features, feature selection is an important pre-processing step for solving classification problems. This problem is often solved by applying evolutionary algorithms to decrease the dimensional number of features involved. Removing irrelevant features in the feature space and identifying relevant features correctly is the primary objective, which can increase classification accuracy. In this paper, a novel QBGSA–K-NN hybrid system which hybridizes the quantum-inspired binary gravitational search algorithm (QBGSA) with the K-nearest neighbor (K-NN) method with leave-one-out cross-validation (LOOCV) is proposed. The main aim of this system is to improve classification accuracy with an appropriate feature subset in binary problems. We evaluate the proposed hybrid system on several UCI machine learning benchmark examples. The experimental results show that the proposed method is able to select the discriminating input features correctly and achieve high classification accuracy which is comparable to or better than well-known similar classifier systems. [Copyright &y& Elsevier]

Lost circulation, one of the most headache problems in drilling engineering, causes a significant increase in non-productive time and increases the uncertainty of well control risk. There is a tremendous challenge for lost circulation prediction/diagnosis using traditional methods in carbonate formation due to its complex and variable leakage zone. To solve this problem of the Mishrif reservoir in the H oil field, in this paper, we propose a whole practical workflow of lost circulation prediction based on the Mixture Density Network using deep learning. Firstly, we choose 16 comprehensive mudlogging parameters that are the most correlated with the mud loss rate. These parameters are selected from a total of 22 comprehensive mudlogging parameters using three different feature selection approach. Then, combining the parameters and the mud loss rate, we get the relationship between comprehensive mudlogging parameters and the mud loss rate based on the Mixture Density Network including five sub-Gaussian models. Finally, the mud loss rate distribution is obtained according to the relationship with input parameters in real-time. The reliability is also evaluated using the uncertainty information in the model. The results show that the parameters had the highest correlation with the mud loss rate, including Measure Depth, Vertical Depth, Rate of Penetration, Hook load, Pump Pressure, Stroke Per Minute, Flow In, Flow Line, Temperature In, Temperature Out, Mud weight In, Mud weight Out, Conductivity Out, Equivalent Circulating Density, Total Gas and Pit Volume Total. The Mixture Density Network has a solid ability to describe data suitable for mud loss prediction and diagnosis. Improving the quality of training data is very important to improve the prediction accuracy of the model. Due to uncertainty in the weights of the sub-Gaussian model obtained from training, the final probability density distribution curve may be single or multi-peaked. According to the prediction results, lost circulation can be controlled by the corresponding method. Thus, the workflow not only allows for real-time assessment of lost circulation risk during drilling but also provides a reference for optimizing lost circulation control. • The 16 comprehensive mudlogging parameters that are the most correlated with the mud loss rate has been selected. • The probability of mud loss rate in different lost circulation has been calculated using Mixture Density Network. • The uncertainty information has been used to analyze the reliability of the prediction results. [ABSTRACT FROM AUTHOR]

• A framework of building energy modeling for data predictive control is developed. • The framework integrates active learning strategy with feature selection technique. • The framework is demonstrated in a virtual testbed (EnergyPlus and BCVTB). • The case study shows improvement in energy modeling accuracy and DPC performance. Three gaps impede the development of cost-effective and accurate data-driven building energy modeling/models (DBEM) for energy forecasting and predictive control strategies. Gap 1: data bias is common in building operation data, but this topic is hardly studied in DBEM; Gap 2: high data dimensionality is common in DBEM, but a systematic and scalable methodology is lacking to solve the problem; Gap 3: the interactions between data bias and high dimensionality have not been systematically studied for DBEM and predictive control in buildings. In this paper, to address the three gaps mentioned above, we develop a framework that integrates active learning and feature selection for DBEM used for whole building data predictive control (or DPC, which is a branch of model predictive control). The framework provides a systematic methodology and automatic workflow that starts with raw data from building automation systems to the establishment of data-driven energy models for DPC controllers. The developed strategies and framework are evaluated in a virtual testbed based on EnergyPlus and BCVTB. Improved performance and reduced computational complexity are observed from the DBEM built with the developed framework, as well as the DPC controller based on that DBEM, indicating the effectiveness of the developed framework. [ABSTRACT FROM AUTHOR]

Abstract: Feature selection is usually a separate procedure which can not benefit from result of the data exploration. In this paper, we propose a unsupervised feature selection method which could reuse a specific data exploration result. Furthermore, our algorithm follows the idea of clustering attributes and combines two state-of-the-art data analyzing methods, that's maximal information coefficient and affinity propagation. Classification problems with different classifiers were tested to validation our method and others. Data experiments result exhibits our unsupervised algorithm is comparable with classical feature selection methods and even outperforms some supervised learning algorithms. Data simulation with one credit dataset of our own from a bank of China shows the capability of our method for real world application. [Copyright &y& Elsevier]

The feature selection process constitutes a commonly encountered problem of global combinatorial optimization. The process reduces the number of features by removing irrelevant and redundant data. This paper proposed a novel immune clonal genetic algorithm based on immune clonal algorithm designed to solve the feature selection problem. The proposed algorithm has more exploration and exploitation abilities due to the clonal selection theory, and each antibody in the search space specifies a subset of the possible features. Experimental results show that the proposed algorithm simplifies the feature selection process effectively and obtains higher classification accuracy than other feature selection algorithms. [ABSTRACT FROM AUTHOR]

Abstract: Recent years, various information theoretic based measurements have been proposed to remove redundant features from high-dimensional data set as many as possible. However, most traditional Information-theoretic based selectors will ignore some features which have strong discriminatory power as a group but are weak as individuals. To cope with this problem, this paper introduces a cooperative game theory based framework to evaluate the power of each feature. The power can be served as a metric of the importance of each feature according to the intricate and intrinsic interrelation among features. Then a general filter feature selection scheme is presented based on the introduced framework to handle the feature selection problem. To verify the effectiveness of our method, experimental comparisons with several other existing feature selection methods on fifteen UCI data sets are carried out using four typical classifiers. The results show that the proposed algorithm achieves better results than other methods in most cases. [Copyright &y& Elsevier]

Abstract: Decision tree has most widely used for classification. However the main influence of decision tree classification performance is attribute selection problem. The paper considers a number of different attribute selection measures and experimentally examines their behavior in classification. The results show that the choice of measure doesn’t affect the classification accuracy, but the size of the tree is influenced significantly. The main effect of the new attribute selection measures which base on normal gain and distance is that they generate smaller trees than traditional attribute selection measures. [Copyright &y& Elsevier]

Abstract: Objectives: Balancing the trade-offs between solution quality, problem-solving efficiency, and transparency is an important challenge in medical applications of conversational case-based reasoning (CCBR). For example, test selection in CCBR is often based on strategies in which the absence of a specific hypothesis (e.g., diagnosis) to be confirmed makes it difficult to explain the relevance of test results that users are asked to provide. In this paper, we present an approach to CCBR in medical classification and diagnosis that aims to increase transparency while also providing high levels of accuracy and efficiency. Methods: We present an algorithm for CCBR called iNN(k) in which feature selection is driven by the goal of confirming a target class and informed by a measure of a feature''s discriminating power in favor of the target class. As we demonstrate in a CCBR system called CBR-Confirm, this enables a CCBR system to explain the relevance of any question it asks the user. We evaluate the algorithm''s accuracy and efficiency on a selection of datasets related to medicine and health care. Results: The performance of iNN(k) on a given dataset is shown to depend on the value of k and on whether local or global feature selection is used in the algorithm. The combination of these parameters for which iNN(k) is most effective in addressing the trade-off between accuracy and efficiency is identified for each of the selected datasets. For example, only 42% and 51% on average of features in a complete problem description were needed by iNN(k) to provide accuracy levels of 86.5% and 84.3% respectively on the lymphography and SPECT heart datasets from the UCI machine learning repository. Conclusion: Our results demonstrate the ability of iNN(k) to provide high levels of accuracy on most of the selected datasets, while often requiring the user to provide only a small subset of the features in a complete problem description, and enabling a CCBR system to explain the relevance of any question it asks the user. [Copyright &y& Elsevier]