Search

Multiobjective evolutionary algorithms (MOEAs) have gained much attention due to their high effectiveness and efficiency in solving multiobjective optimization problems (MOPs). However, when solving MOPs, it is important but difficult to maintain a good balance of exploration and exploitation. In addition, some reference point based MOEAs with fixed reference points perform poorly on MOPs with irregular frontiers. Therefore, this paper proposes a new multistrategy multiobjective differential evolutionary (DE) algorithm, named RLMMDE. In RLMMDE, a multistrategy and multicrossover DE optimizer is utilized to alleviate the exploration and exploitation dilemma. An adaptive reference point activation mechanism based on RL is proposed to activate the adaptive adjustment of reference points. Moreover, a reference point adaptation method is proposed to improve the performance of RLMMDE on irregular frontier problems. Experimental results of RLMMDE tested on some benchmark test suites (i.e., ZDT, DTLZ, UF, WFG, and LSMOP) and two practical mixed-variable optimization problems show that the algorithm outperforms some advanced MOEAs. [ABSTRACT FROM AUTHOR]

Recently, Extreme learning machine (ELM), an efficient training algorithm for single-hidden-layer feedforward neural networks (SLFN), has gained increasing popularity in machine learning communities. In this paper the ELM based Area Under the ROC Curve (AUC) optimization algorithms are studied so as to further improve the performance of ELM for imbalanced datasets. For binary class problems, a novel ELM algorithm is proposed based on an efficient least square method. For multi-class problems, the following works are done in this paper: First of all, theoretical comparison analysis is proposed for the potential multi-class extensions of AUC; Secondly, a unified objective function for multi-class AUC optimization is proposed following the theoretical analysis; Subsequently, two ELM based multi-class AUC optimization algorithms called E L M M A U C and E L M m a c r o A U C respectively are proposed followed with complexity analyses; Finally, the generalization analysis is established for E L M M A U C in search of theoretical supports. Empirical study on a variety of real-world datasets show the effectiveness of our proposed algorithms. [ABSTRACT FROM AUTHOR]

Recently, there has been growing interest in social network analysis. Graph models for social network analysis are usually assumed to be a deterministic graph with fixed weights for its edges or nodes. As activities of users in online social networks are changed with time, however, this assumption is too restrictive because of uncertainty, unpredictability and the time-varying nature of such real networks. The existing network measures and network sampling algorithms for complex social networks are designed basically for deterministic binary graphs with fixed weights. This results in loss of much of the information about the behavior of the network contained in its time-varying edge weights of network, such that is not an appropriate measure or sample for unveiling the important natural properties of the original network embedded in the varying edge weights. In this paper, we suggest that using stochastic graphs, in which weights associated with the edges are random variables, can be a suitable model for complex social network. Once the network model is chosen to be stochastic graphs, every aspect of the network such as path, clique, spanning tree, network measures and sampling algorithms should be treated stochastically. In particular, the network measures should be reformulated and new network sampling algorithms must be designed to reflect the stochastic nature of the network. In this paper, we first define some network measures for stochastic graphs, and then we propose four sampling algorithms based on learning automata for stochastic graphs. In order to study the performance of the proposed sampling algorithms, several experiments are conducted on real and synthetic stochastic graphs. The performances of these algorithms are studied in terms of Kolmogorov-Smirnov D statistics, relative error, Kendall's rank correlation coefficient and relative cost. [ABSTRACT FROM AUTHOR]

Abstract This paper presents a collection of refinement operators for directed labeled graphs (DLGs), and a family of distance and similarity measures based on them. We build upon previous work on refinement operators for other representations such as feature terms and description logic models. Specifically, we present eight refinement operators for DLGs, which will allow for the adaptation of three similarity measures to DLGs: the anti-unification-based, S λ , the property-based, S π , and the weighted property-based, S wπ , similarities. We evaluate the resulting measures empirically, comparing them to existing similarity measures for structured data in the context of instance-based machine learning. [ABSTRACT FROM AUTHOR]

Abstract Addressing imbalanced datasets in classification tasks is a relevant topic in research studies. The main reason is that for standard classification algorithms, the success rate when identifying minority class instances may be adversely affected. Among different solutions to cope with this problem, data level techniques have shown a robust behavior. In this paper, the novel imbalance package is introduced. Written in R and C++, and available at CRAN repository, this library includes recent relevant oversampling algorithms to improve the quality of data in imbalanced datasets, prior to performing a learning task. The main features of the package, as well as some illustrative examples of its use are detailed throughout this manuscript. [ABSTRACT FROM AUTHOR]

Abstract Multi-view unsupervised feature selection has been proven to be an effective approach to reduce the dimensionality of multi-view data. One of its key issues is how to exploit the underlying common structures across different views. In this paper, we propose a consensus learning guided multi-view unsupervised feature selection method, which embeds multi-view feature selection into a non-negative matrix factorization based clustering with sparse constrain. The proposed method learns latent feature matrices from all the views, and optimizes a consensus matrix such that the difference between the cluster indicator matrix of each view and the consensus matrix is minimized. The parameters for balancing the weights of different views are automatically adjusted, and a sparse constraint is imposed on the latent feature matrices to perform feature selection. After that, we design an effective iterative algorithm to solve the resultant optimization problem. Extensive experiments have been conducted on six publicly multi-view datasets, and the results demonstrate that the proposed algorithm outperforms several other state-of-the-art single view and multi-view unsupervised feature selection methods in terms of clustering tasks, validating the effectiveness of the proposed multi-view unsupervised feature selection method. The source code of our algorithm will be available on our on-line page: http://tangchang.net/. [ABSTRACT FROM AUTHOR]

Recently, many excellent algorithms for time series prediction issues have been proposed, most of which are developed based on the assumption that sufficient training data and testing data under the same distribution are available. However, in reality, time-series data usually exhibit some kind of time-varying characteristic, which may lead to a wide variability between old data and new data. Hence, how to transfer knowledge over a long time span, when addressing time series prediction issues, poses serious challenges. To solve this problem, in this paper, a hybrid algorithm based on transfer learning, Online Sequential Extreme Learning Machine with Kernels (OS-ELMK), and ensemble learning, abbreviated as TrEnOS-ELMK, is proposed, along with its precise mathematic derivation. It aims to make the most of, rather than discard, the adequate long-ago data, and constructs an algorithm framework for transfer learning in time series forecasting, which is groundbreaking. Inspired by the preferable performance of models ensemble, ensemble learning scheme is also incorporated into our proposed algorithm, where the weights of the constituent models are adaptively updated according to their performances on fresh samples. Compared to many existing time series prediction methods, the newly proposed algorithm takes long-ago data into consideration and can effectively leverage the latent knowledge implied in these data for current prediction. In addition, TrEnOS-ELMK naturally inherits merits of both OS-ELMK and ensemble learning due to its incorporation of the two techniques. Experimental results on three synthetic and six real-world datasets demonstrate the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

In a semi-supervised domain adaptation (DA) task, one has access to only few labeled target examples. In this case, the success of DA needs the effective utilization of a large number of unlabeled target data to extract more discriminative information that is useful for generalization. To this end, we exploit in this paper the feature space embeddings of the target data as well as multi-source prior models to augment the discrimination space for the target function learning. Therefore, we propose a novel multi-source adaptation learning framework based on Sparse Feature Space Representation (SFSR), or called SFSR-MSAL for short. Specifically, the SFSR algorithm is first presented for the further construction of robust graph, on which the discriminative information can be smoothly propagated into the unlabeled target data by additionally incorporating the geometric structure of the target data. Considering the robustness in the semi-supervised DA, we replace the traditional l 2 -norm based least squares regression with the l 2, 1 -norm sparse regression, and then construct the SFSR-graph based semi-supervised DA framework with multi-source adaptation constraints. Our framework is universal and can be easily degraded into semi-supervised learning by just tuning the regularization parameter. Moreover, to select the discriminative SFSR-graph Laplacians, we also introduce the ensemble SFSR-graph Laplacians regularization into SFSR-MSAL, thus further improving the performance of SFSR-MSAL. The validity of our methods including semi-supervised and DA learning are examined by several visual recognition tasks on some benchmark datasets, which demonstrate the superiority of our methods in comparison with other related state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

The Successor Feature framework for Reinforcement Learning algorithms improves task transfer by decomposing the learned state–action value function. The decomposition involves two components, one that captures future-expected state features and the other that models the task-related reward structure. However, successful transfer between tasks depends heavily on how the reward function changes, possibly leading to failure of the original Successor Feature formulation. This paper proposes the Dynamic Successor Feature framework, DynSF, by extending the mathematical formulation of the original Successor Feature framework to center around a learned state-transition model. Under this formulation, the state-transition model dynamically induces the acting policy. The flexibility of DynSF also extends to the architecture, requiring only a state-transition model and a small vector of parameters. This architecture provides immense flexibility in the choice of the model used to learn the state-transition model. The DynSF framework is evaluated and compared to other baseline algorithms through several experiments in a continuous grid world environment, a robotic Reacher, and pixels in the Doom environment. • DynSF enables a state-transition model to be used in Reinforcement Learning. • Policy and discount factors can be parameterized dynamically. • Different supervised ML algorithms can be used to learn the state transition model. • DynSF can be used for guided exploration and transfer learning. • DynSF performs better during task transfer than the Successor Feature framework. [ABSTRACT FROM AUTHOR]

In this paper, by introducing the statistics of training data into support vector regression (SVR), we propose a minimum deviation distribution regression (MDR). Rather than just minimizing the structural risk, MDR also minimizes both the regression deviation mean and the regression deviation variance, which is able to deal with the different distribution of boundary data and noises. The formulation of minimizing the first and second order statistics in MDR leads to a strongly convex quadratic programming problem (QPP). An efficient dual coordinate descend algorithm is adopted for small sample problem, and an average stochastic gradient algorithm for large scale one. Both theoretical analysis and experimental results illustrate the efficiency and effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

A number of machine learning and knowledge-based algorithms are using a metric, or a distance, in order to compare individuals. The Euclidean distance is usually employed, but it may be more efficient to learn a parametric distance such as Mahalanobis metric. Learning such a metric is a hot topic since more than ten years now, and a number of methods have been proposed to efficiently learn it. However, the nature of the problem makes it quite difficult for large scale data, as well as data for which classes overlap. This paper presents a simple way of improving accuracy and scalability of any iterative metric learning algorithm, where constraints are obtained prior to the algorithm. The proposed approach relies on a loss-dependent weighted selection of constraints that are used for learning the metric. Using the corresponding dedicated loss function, the method clearly allows to obtain better results than state-of-the-art methods, both in terms of accuracy and time complexity. Some experimental results on real world, and potentially large, datasets are demonstrating the effectiveness of our proposition. [ABSTRACT FROM AUTHOR]

Multigranulation rough sets have attracted more and more attentions in recent years. In real-life applications, with the development of information technology, the attribute values often dynamically evolve over time. How to update useful knowledge is of great importance for dynamic information systems. Approximations of a concept are fundamental concepts of multigranulation rough sets, which need to be updated incrementally while refining or coarsening attribute values. Motivated by the requirements of dynamic knowledge acquisition due to refining or coarsening attribute values, in this paper, we present the dynamic mechanisms for updating approximations in multigranulation rough sets while refining or coarsening attribute values. Then, the corresponding dynamic algorithms for updating multigranulation approximations are designed on the basis of the proposed mechanisms. Extensive experiments on six data sets from UCI demonstrate that the proposed dynamic algorithms for updating approximations in multigranulation rough sets are more effective in comparison with the static algorithm. [ABSTRACT FROM AUTHOR]

Feature selection is an important preprocessing technique in the fields of data mining and machine learning. With the promotion of privacy protection awareness, recently it becomes a very practical and challenging issue to select high-quality feature subsets while ensuring the privacy of all participants. However, there is a lack of research results on this issue, i.e., feature selection under privacy protection. Aiming at the issue, this paper proposes a federated feature selection framework for the first time. In the framework, inspiring by the idea of federated learning, a credible third participant is introduced to process and integrate optimal feature subsets from multiple participants. On the basis of the framework, a federated evolutionary feature selection algorithm based on particle swarm optimization is proposed to effectively solve feature selection problems with multiple participants under privacy protection. Two new operators satisfying the requirement of privacy protection, i.e., the feature assembling strategy with multi-participant cooperation and the swarm initialization strategy guided by assembling solution, are designed to improve the ability of the proposed algorithm. Compared with several typical assembling feature selection algorithms on 15 data sets, experimental results show that the proposed algorithm can significantly improve the classification accuracy of the feature subset selected by each participant, while protecting the privacy of data. [ABSTRACT FROM AUTHOR]

The combination of Machine Learning (ML) algorithms is a solution for constructing stronger predictors than a single one. However, some approximations suggest that combining unstable algorithms provides better results than combining stable algorithms. For instance, Generative ensembles, based on re-sampling techniques, have demonstrated high performance by fusing the information of unstable base learners. Random Forest and Gradient Boosting are two well-known examples, both combining Decision Trees and providing better predictions than those obtained using a single tree. However, such successful results have not been achieved by assembling stable algorithms. This paper introduces the notion of limited learner and a new ensemble general framework called Minimally Overfitted Ensemble (MOE), a re-sampling-based ensemble approach that constructs slightly overfitted-based learners. The proposed framework works well with stable and unstable base algorithms, thanks to a Weighted RAndom Bootstrap (WRAB) sampling that provides the necessary diversity for the stable base algorithms. A hyperparameter analysis of the proposal is carried out on artificial data. Besides, its performance is evaluated on real datasets against well-known ML methods. The results confirm that the MOE framework works successfully using stable and unstable base algorithms, improving in most cases the predictive ability of single ML models and other ensemble methods. • New ensemble general framework. • A novel concept of ensemble learners is introduced. • Minimal overfitted models and limited learners are defined. • The framework works with any base machine learning training algorithm. • The proposal compares successfully to well known ensemble approaches. [ABSTRACT FROM AUTHOR]

Clustering ensemble is an unsupervised ensemble learning method that is very important in machine learning, since it integrates multiple weak base clustering results to produce a strong consistency result. This paper proposes the Markov clustering ensemble (MCE) model to solve the weak stability and robustness of soft clustering ensemble. First, the base clustering algorithms are regarded as new features of the original datasets. Then, the results of the base clustering algorithms are the values of these features, which can break through the framework of consensus cluster ensemble. Second, as the base clustering results are discrete data, the maximum information coefficient is applied to measure their similarity. Accordingly, a graph-based cluster ensemble model can be constructed with row vectors as vertices and the similarity between row vectors as edges. Then, the Markov process can be applied to infer the graph-based cluster ensemble model; therefore, the MCE algorithm is designed according to the inference. To test the performance of the MCE algorithm, clustering algorithms and clustering ensemble algorithms are used to conduct comparative experiments on ten datasets. The experimental results show that the MCE algorithm outperforms the other algorithms in terms of accuracy and purity. [ABSTRACT FROM AUTHOR]

Reinforcement learning (RL) is a learning paradigm that can be useful in a wide variety of real-world applications. However, its applicability to complex problems remains problematic due to different causes. Particularly important among these are the high quantity of data required by the agent to learn useful policies and the poor scalability to high-dimensional problems due to the use of local approximators. This paper presents a novel RL algorithm, called online fitted policy iteration (OFPI), that steps forward in both directions. OFPI is based on a semi-batch scheme that increases the convergence speed by reusing data and enables the use of global approximators by reformulating the value function approximation as a standard supervised problem. The proposed method has been empirically evaluated in three benchmark problems. During the experiments, OFPI has employed a neural network trained with the extreme learning machine algorithm to approximate the value functions. Results have demonstrated the stability of OFPI using a global function approximator and also performance improvements over two baseline algorithms (SARSA and Q-learning) combined with eligibility traces and a radial basis function network. [ABSTRACT FROM AUTHOR]

The main purpose of negative correlation learning (NCL) is to produce ensembles with sound generalization capability through controlling the disagreement among base learners’ outputs. This paper uses neural networks with random weights (NNRWs) to implement such learning scheme in the study of face recognition. Particularly, two-dimensional (2D) feed-forward neural networks (2D-FNNs) with random weights (2D-NNRWs) are employed as base components, and which are incorporated with the NCL strategy for building neural network ensembles, where the basis functions of the base networks are generated randomly and the free parameters of the 2D-FNNs can be determined by solving a linear equation system. Also, an analytical solution is derived for these parameters. To examine the merits of the proposed algorithm, a series of comparative experiments are performed. The experimental results indicate that the proposed approach outperforms existing approaches. [ABSTRACT FROM AUTHOR]

This paper presents a novel recurrent kernel algorithm for online learning. It introduces a propagation scheme to recycle the kernel state information. The novel structure keeps records of the training sample information and incorporates it in the learning task over time to preserve the characteristics of the training sequences. In order to ensure the convergence of the algorithm, an adaptive training method is proposed to tune the kernel weight and recurrent weight simultaneously followed by detailed analysis of the weight convergence. Numerical simulations are presented to show the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Learning from imbalanced data is a problem which arises in many real-world scenarios, so does the need to build classifiers able to predict more than one class label simultaneously (multilabel classification). Dealing with imbalance by means of resampling methods is an approach that has been deeply studied lately, primarily in the context of traditional (non-multilabel) classification. In this paper the process of synthetic instance generation for multilabel datasets (MLDs) is studied and MLSMOTE (Multilabel Synthetic Minority Over-sampling Technique), a new algorithm aimed to produce synthetic instances for imbalanced MLDs, is proposed. An extensive review on how imbalance in the multilabel context has been tackled in the past is provided, along with a thorough experimental study aimed to verify the benefits of the proposed algorithm. Several multilabel classification algorithms and other multilabel oversampling methods are considered, as well as ensemble-based algorithms for imbalanced multilabel classification. The empirical analysis shows that MLSMOTE is able to improve the classification results produced by existent proposals. [ABSTRACT FROM AUTHOR]

Decision trees have been widely used in data mining and machine learning as a comprehensible knowledge representation. Minimal cost decision tree construction plays a crucial role in cost sensitive learning. Recently, many algorithms have been developed to tackle this problem. These algorithms choose an appropriate cut point of a numeric attribute by computing all possible cut points and assign a node through test all attributes. Therefore, the efficiency of these algorithms for large data sets is often unsatisfactory. To solve this issue, in this paper we propose a cost sensitive decision tree algorithm with two adaptive mechanisms to learn cost sensitive decision trees from training data sets based on C4.5 algorithm. The two adaptive mechanisms play an important role in cost sensitive decision tree construction. The first mechanism, adaptive selecting the cut point (ASCP) mechanism, selects the cut point adaptively to build a classifier rather than calculates each possible cut point of an attribute. It improves the efficiency of evaluating numeric attributes for cut point selection significantly. The second mechanism, adaptive removing attribute (ARA) mechanism, removes some redundant attributes in the process of selecting node. The effectiveness of the proposed algorithm is demonstrated on fourteen UCI data sets with representative test cost Normal distribution. Compared with the CS-C4.5 algorithm, the proposed algorithm significantly increases efficiency. [ABSTRACT FROM AUTHOR]

Feature selection techniques are attracting more and more attention with the growing number of domains that produce high dimensional data. Due to the absence of class labels, many researchers focus on the unsupervised scenario, attempting to find an optimal feature subset that preserves the original data distribution. However, the existing methods either fail to achieve sparsity or ignore the potential redundancy among features. In this paper, we propose a novel unsupervised feature selection algorithm, which retains the preserving power, and implements high sparsity and low redundancy in a unified manner. On the one hand, to preserve the data structure of the whole feature set, we build the graph Laplacian matrix and learn the pseudo class labels through spectral analysis. By finding a feature weight matrix, we are allowed to map the original data into a low dimensional space based on the pseudo labels. On the other hand, to ensure the sparsity and low redundancy simultaneously, we introduce a novel regularization term into the objective function with the nonnegative constraints imposed, which can be viewed as the combination of the matrix norms | | · | | m 1 and | | · | | m 2 on the weights of features. An iterative multiplicative algorithm is accordingly designed with proved convergence to efficiently solve the constrained optimization problem. Extensive experimental results on different real world data sets demonstrate the promising performance of our proposed method over the state-of-the-arts. [ABSTRACT FROM AUTHOR]

Feature weighting has been an important topic in classification learning algorithms. In this paper, we propose a new paradigm of assigning weights in classification learning, called value weighting method. While the current weighting methods assign a weight to each feature, we assign a different weight to the values of each feature. The proposed method is implemented in the context of naive Bayesian learning, and optimal weights of feature values are calculated using a gradient approach. The performance of naive Bayes learning with value weighting method is compared with that of other state-of-the-art methods for a number of datasets. The experimental results show that the value weighting method could improve the performance of naive Bayes significantly. [ABSTRACT FROM AUTHOR]

Structural balance enables a comprehensive understanding of the potential tensions and conflicts of signed networks, and its computation and transformation have attracted increasing attention in recent years. The balance computation aims at evaluating the distance from an unbalanced network to a balanced one, and the balance transformation is to convert an unbalanced network into a balanced one. In this paper, firstly, we model the balance computation of signed networks as the optimization of an energy function. Secondly, we model the balance transformation as the optimization of a more general energy function incorporated with transformation cost. Finally, a multilevel learning based memetic algorithm, which incorporates network-specific knowledge such as the neighborhoods of node, cluster and partition, is proposed to solve the modeled optimization problems. Systematical experiments in real-world social networks demonstrate the superior performance of the proposed algorithm compared with the state-of-the-art algorithms on the computation and transformation of structural balance. The results also show that our method can resolve the potential conflicts of signed networks with the minimum cost. [ABSTRACT FROM AUTHOR]

Trust is known as an important social concept and an effective factor in all human interactions in social networks. Users tend to interact with trusted people with whom they have had positive experiences. Trust is updated over time as a result of these repeated interactions. Even though dynamicity is a universally accepted property of the social trust, trust networks are often modeled as static digraphs. In this paper, we first propose that a stochastic graph model, where the weights associated with edges are random variables with unknown distributions, may be a better candidate for representing trust networks. Then, we review the literature on analyzing complex networks and determine graph measures which are most appropriate with respect to the special properties of the concept of trust. Considering trust-specific measures, we finally propose a dynamic algorithm for sampling from stochastic trust networks, which is an extension of Frontier sampling. Even though there exist a few sampling methods which address edge weights and their variations over time through the sampling process, these methods are unable to accurately preserve the properties of trust networks. The proposed algorithm in this paper uses learning automata to tackle the disconnectivity problem of sampled subgraphs by Frontier sampling and, at the same time, capture the changes of edge weights through the sampling process. Our experimental results on the well-known trust network datasets indicate that the proposed sampling algorithm preserves more accurately the trust-specific measures of trust networks compared to existing sampling methods. • A stochastic graph model for representing trust networks is recommended. • We present a comprehensive review of the literature on analyzing complex networks. • New extensions of some graph measures are proposed, considering properties of trust. • We develop a dynamic sampling algorithm for stochastic trust networks. • The experimental results show the benefits of our proposed sampling algorithm. [ABSTRACT FROM AUTHOR]

Image segmentation has been an important technique in the field of image processing. Fine-level manual annotations are very limited and difficult for a large collection of imbalanced images, where each image contains quite different small objects. However, we find that these imbalanced images have similar decision graphs obtained by a lightweight and simple clustering algorithm Density Peaks (DPeaks). Hence, in this paper, we propose a weakly supervised image segmentation algorithm. It trains a decision curve from decision graphs of a few sample imbalanced images by SVM and Support Vector Regression (SVR), which is effective for identifying the sparse region of an imbalanced image. Besides, RangeTree is applied to accelerate RDP for large images due to the high complexity of DPeaks Clustering. Experiments prove that the proposed algorithm works well on imbalanced image data sets, not only it can recognize main things, but also has the ability to identify some relatively small objects. [ABSTRACT FROM AUTHOR]

In this paper, we propose support vector-based supervised learning algorithms, called multiclass support vector data description with weighted dynamic time warping kernel function (MSVDD-WDTWK) and multiclass support vector machines with weighted dynamic time warping kernel function (MSVM-WDTWK), which provides a flexible and robust kernel function for time series classification between non-aligned time series data resulting in improved accuracy. The proposed WDTW kernel function provides an optimal match between two time series data by not only allowing a non-linear mapping between two data sequences, but also considering relative significance depending on the phase difference between points on time series data. We validate the proposed approaches using extensive numerical experiments on a number of multiclass UCR time series data mining archive, and demonstrate that our proposed methods provide lower classification error rates compared with existing techniques. [ABSTRACT FROM AUTHOR]

Instance selection is one of the most important preprocessing steps in many machine learning tasks. Due to the increasing size of the problems, removing useless, erroneous or noisy instances is frequently an initial step that is performed before other data mining algorithms are applied. Instance selection as part of this data reduction task is one of the most relevant problems in current data mining research. Over the past decades, many different instance selection algorithms have been proposed, each with its own strengths and weaknesses. However, as in the case of classification, it is unlikely that a single instance selection algorithm would be able to achieve good results across many different datasets and application fields. In classification, one of the most successful ways of consistently improving the performance of a single learner is the construction of ensembles using boosting methods. In this paper, we propose a novel approach for instance selection based on boosting instance selection algorithms in the same way boosting is applied to classification. The proposed approach opens a new field of research in which to apply the many techniques developed for boosting classifiers, for instance selection and other data reduction techniques such as feature selection and simultaneous instance and feature selection. Using 60 datasets for balanced problems and 45 datasets for class-imbalanced problems, the experiments reported show a clear improvement in several state-of-the-art instance selection algorithms using the proposed methodology. [ABSTRACT FROM AUTHOR]

Similarity learning plays a fundamental role in the problem of database retrieval and nearest classification problem. Traditional pairwise similarity measure ignores the contextual information, and the Graph Transduction (GT) has been proposed as a contextual similarity learning algorithm to utilize the contextual information, which is embedded in a nearest neighbor graph. On main shortage of this method is that it is difficult to choose the optimal graph since different graphs may focus on different aspects of the objects. Co-Transduction (CT) is lately proposed by fusing two different graphs. In this paper, we generalize this problem by using the ensemble of many candidate graphs with different models and parameters for transduction, by assuming that the optimal graph could be obtained by the weighted linear ensemble of these candidate graphs. The similarities and graph weights are modeled within one unified objective function, and optimized alternately in an iterative algorithm. The new proposed algorithm, named as Ensemble Transduction (ET), is tested on two challenging tasks and the experimental results show that it can outperform both the GT and CT. [ABSTRACT FROM AUTHOR]

Dragonfly Algorithm (DA) is a recent swarm-based optimization method that imitates the hunting and migration mechanisms of idealized dragonflies. Recently, a binary DA (BDA) has been proposed. During the algorithm iterative process, the BDA updates its five main coefficients using random values. This updating mechanism can be improved to utilize the survival-of-the-fittest principle by adopting different functions such as linear, quadratic, and sinusoidal. In this paper, a novel BDA is proposed. The algorithm uses different strategies to update the values of its five main coefficients to tackle Feature Selection (FS) problems. Three versions of BDA have been proposed and compared against the original DA. The proposed algorithms are Linear-BDA, Quadratic-BDA, and Sinusoidal-BDA. The algorithms are evaluated using 18 well-known datasets. Thereafter, they are compared in terms of classification accuracy, the number of selected features, and fitness value. The results show that Sinusoidal-BDA outperforms other proposed methods in almost all datasets. Furthermore, Sinusoidal-BDA exceeds three swarm-based methods in all the datasets in terms of classification accuracy and it excels in most datasets when compared in terms of the fitness function value. In a nutshell, the proposed Sinusoidal-BDA outperforms the comparable feature selection algorithms and the proposed updating mechanism has a high impact on the algorithm performance when tackling FS problems. • This paper proposes an improved Binary Dragonfly Algorithm (BDA) for Feature Selection. • Three versions of BDA are proposed: LBDA, QBDA, and SBDA. • Eighteen datasets taken from UCI machine learning repository are used for evaluation process. • SBDA excels LBDA and QBDA in terms of accuracy, number of features, and fitness function. • SBDA excels the nine comparative methods in 12 out of 18 dataset in terms of accuracy. [ABSTRACT FROM AUTHOR]

Algorithm evaluation is a very important task for different Machine Learning (ML) problems. In this work, we assume a different perspective in ML evaluation, in which algorithms are evaluated at the instance level. In this perspective, Item Response Theory (IRT) has recently been applied to algorithm evaluation in ML in order to identify which instances are more difficult and discriminating in a dataset, while also evaluating algorithms based on their predictions for instances with different difficulty values. In IRT, a strong algorithm returns accurate predictions for the most difficult instances, while maintaining a consistent behaviour in the easiest instances. The most common IRT models adopted in the literature only deal with dichotomous responses (i.e., a response has to be either correct or incorrect). This is suitable for evaluating classification algorithms, but not adequate in application contexts where responses are recorded in a continuous scale without an upper bound, such as regression. In this paper we propose the Γ -IRT model, particularly designed for dealing with positive unbounded responses, which we model using a Gamma distribution, parameterised according to respondent ability and item difficulty and discrimination parameters. The proposed parameterisation results in item characteristic curves with more flexible shapes compared to the traditional logistic curves adopted in IRT. We apply the proposed model to evaluate student responses (number of errors) in open-ended questions extracted from Statistics exams. Then, we use Γ -IRT to assess regression model abilities, where responses are the absolute errors in test instances. This novel application represents an alternative for evaluating regression performance and for identifying regions in a regression dataset that present different levels of difficulty and discrimination. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, an ensemble-based method for the screening of diabetic retinopathy (DR) is proposed. This approach is based on features extracted from the output of several retinal image processing algorithms, such as image-level (quality assessment, pre-screening, AM/FM), lesion-specific (microaneurysms, exudates) and anatomical (macula, optic disk) components. The actual decision about the presence of the disease is then made by an ensemble of machine learning classifiers. We have tested our approach on the publicly available Messidor database, where 90% sensitivity, 91% specificity and 90% accuracy and 0.989 AUC are achieved in a disease/no-disease setting. These results are highly competitive in this field and suggest that retinal image processing is a valid approach for automatic DR screening. [Copyright &y& Elsevier]

Abstract: Rapid growth of digital information requires automated handling and organization of documents. The two main stages in automated document categorization are (i) term reduction and (ii) classification. In this paper, we present a novel two-stage term reduction strategy based on Information Gain (IG) theory and Geometric Particle Swarm Optimization (GPSO) search. We evaluate performance of the proposed term reduction approach with use of a new classifier, fuzzy unordered rule induction algorithm (FURIA) to categorize multi-label texts. In order to evaluate the performance of FURIA quantitatively, we compared it against two widely used algorithms, Naïve Bayes and Support Vector Machine (SVM). Text Categorization (TC) performance of the proposed term reduction strategy is validated with use of Reuters-21578 and OHSUMED text collection datasets. The experimental results show that performance of the proposed term reduction method is efficient for document organization tasks. [Copyright &y& Elsevier]

Abstract: Side information, like must-link (ML) and cannot-link (CL), has been widely used in single-view classification tasks. However, so far such information has never been applied in multi-view classification tasks. In many real world situations, data with multiple representations or views are frequently encountered, and most proposed algorithms for such learning situations require that all the multi-view data should be paired. Yet this requirement is difficult to satisfy in some settings and the multi-view data could be totally unpaired. In this paper, we propose an learning framework to design the multi-view classifiers by only employing the weak side information of cross-view must-links (CvML) and cross-view cannot-links (CvCL). The CvML and the CvCL generalize the traditional single-view must-link (SvML) and single-view cannot-link (SvCL), and to the best of our knowledge, are first definitely introduced and applied into the multi-view classification situations. Finally, we demonstrate the effectiveness of our method in our experiments. [Copyright &y& Elsevier]

Highlights: [•] This paper proposes a Column Generation Newton (CGN) algorithm for finding solution of the kernel 1-norm SVM. [•] CGN is combining the Column Generation and the Newton Linear Programming SVM method. [•] CGN is fast when solving the kernel 1-norm SVM. [ABSTRACT FROM AUTHOR]

Abstract: Data mining is the process which is used to analyze the large database to find the useful pattern. Data mining can be used to learn about student’s behavior from data collected using the course management system such as Moodle (Modular Object-Oriented Developmental Learning Environment). Here in this paper we show how data mining techniques such as clustering and association rule algorithm is useful in Course Recommendation System which recommends the course to the student based on choice of other students for particular set of courses collected from Moodle. As a result of Course Recommendation System, we can recommend to new student who has recently enrolled for some course e.g. Operating System, the new course to be opted e.g. Distributed System. Our approach uses combination of clustering technique – Simple K-means and association rule algorithm – Apriori and finds the result. These results were compared with the results of open source data mining tool-Weka. The result obtained using combined approach matches with real world interdependencies among the courses. Other combinations of clustering and association rule algorithms are also discussed here to select the best combination. This Course Recommendation System could help in building intelligent recommender system. This approach of recommending courses to new students can be immensely be useful in “MOOC (Massively Open Online Courses)”. [Copyright &y& Elsevier]

One of the crucial processes in machine learning algorithms to improve the performance as well as, in some cases, to reduce the computational cost is feature selection. In other words, when there are many features, some are irrelevant; thus, irrelevant features will badly affect the learning process, so selecting the most fitting features can lead to increasing the accuracy of models in high-dimensional datasets. Recent researches benefit from an integration of various feature selection methods to find relevant features more accurately. Despite the fact that applying several feature selection methods can be efficient, the accuracy of current methods and combining the ranks of different feature selection methods are still challenging problems. In this paper, a novel ensemble of feature selection based on weighted PageRank and fuzzy logic, named EFSPF, is proposed to overcome these drawbacks. First, an M × M matrix, called Euclidean Distance Matrix (EDM), is set up, where M is the number of features. Secondly, a fuzzy Type-I is applied individually to tackle uncertainties in ranks. In other words, to specify a reliability rate for each feature, fuzzy Type-I is utilized. Then, EFSPF forms a reliable weighted graph where each feature is considered as a node. Furthermore, the weight between two nodes expresses a combination of their Euclidean Distance in EDM and the calculated reliability rates through fuzzy Type-I. Finally, the rank of each graph node is calculated via the PageRank algorithm. To demonstrate the performance of EFSPF, we have compared it with five ensemble algorithms and eight primary feature selection algorithms. The proposed method is evaluated based on Accuracy, Precision, Recall, and F1 score metrics. The analysis results show that the proposed method outperforms its rivals. The source code of the proposed method is accessible from https://github.com/mehdijoodaki/EFSPF. [ABSTRACT FROM AUTHOR]

Existing research works in scene image classification have focused on different aspects such as content features (e.g., visual information), context features (e.g., annotations, semantic information, etc.) and both. However, such works suffer from various issues such as higher feature size and lower classification performance. In this paper, we propose a new feature extraction approach for scene image representation using two kinds of rich information: content features and context features. Specifically, the new content features are generated by multi-scale foreground and background information. Similarly, the new context features are generated by the novel compact supervised codebook. Our compact supervised codebook minimizes irrelevant and redundant information, which, in result, achieves the lower-sized contextual feature vector. Finally, we combine both content and context features to represent the scene image. Our experiments on three widely used benchmark scene datasets using Support Vector Machine (SVM) classifier reveal that our proposed context and content features produce better results than existing context and content features, respectively. The fusion of the proposed two types of features significantly outperform numerous state-of-the-art features. • Propose a novel contextual feature extraction method for scene images. • Propose a novel unique tokens extraction algorithm. • Aggregate multi-scale content and contextual features to represent the scene images. • Evaluate our proposed method on three commonly-used benchmark datasets. [ABSTRACT FROM AUTHOR]

Evolutionary Computation (EC) algorithms are powerful techniques for feature selection tasks however, they reach different solutions in each run, and this is known as the stability issue. Existing solutions to finding a stable subset of features when using an EC algorithm include aggregation and frequency-based methods. These methods may return feature subsets that achieve weak or inconsistent classification performance when utilised to build classifiers, and this limitation is known as 'lack of generalisation power'. To address this limitation, this paper proposes a novel algorithm called Generalisation Power Analysis (GPA) that measures the performance of feature subsets in terms of generalisation power and hence evaluates their ability to achieve optimal or near-optimal accuracy over multiple classifiers. GPA has been designed to work with the stochastic nature of EC algorithms. Experiments with eleven benchmark datasets revealed that the proposed GPA approach consistently outperformed alternative methods in finding subsets that achieved high generalisation power. Although GPA requires relatively higher computation time compared to alternative approaches as it embeds multiple classifiers, the advantages of using GPA during feature selection outweigh this limitation since the outcome will be a robust prediction model that has been developed using a subset of features that are not biased towards a specific classifier. • EC algorithms for feature selection may return different feature subsets whenever they are run • Generalisation Power Analysis (GPA) evaluates feature subsets over multiple classifiers • Experiments show that GPA outperforms alternative approaches for finding a stable set of features • GPA can help EC algorithms find a stable set of features needed for constructing machine learning models [ABSTRACT FROM AUTHOR]

Fuzzy rough set theory has been proved to be an effective tool to deal with uncertainty data. Some different forms of fuzzy uncertainty measures have been introduced in fuzzy rough set theory, such as fuzzy information entropy, fuzzy conditional entropy, and fuzzy mutual information. However, as far as we know, most of the above fuzzy conditional entropy and fuzzy mutual information are non-monotonic, which may lead to a non-convergent learning algorithm. For this reason, this paper proposes a novel fuzzy complementary entropy based on the hybrid-kernel function. Then, based on the proposed fuzzy complementary entropy, some corresponding uncertainty measures are also proposed. Furthermore, fuzzy complementary conditional entropy and fuzzy complementary mutual information are proved to change monotonously with attributes. Finally, based on the proposed uncertainty measures, three kinds of evaluation criteria for unsupervised hybrid attribute reduction are defined and a generalized attribute reduction algorithm is designed. The experimental results show that the proposed method is an effective scheme for reducing hybrid attributes. • The novel fuzzy complementary entropy is defined by using hybrid-kernel function. • It is proved that fuzzy complementary conditional entropy and fuzzy complementary mutual information change monotonously with attributes. • The proposed uncertainty measures are applied to unsupervised hybrid attribute reduction. [ABSTRACT FROM AUTHOR]

Deep multi-layer neural networks represent hypotheses of very high degree polynomials to solve very complex problems. Gradient descent optimization algorithms are utilized to train such deep networks through backpropagation, which suffers from permanent problems such as the vanishing gradient problem. To overcome the vanishing problem, we introduce a new anti-vanishing back-propagated learning algorithm called oriented stochastic loss descent (OSLD). OSLD updates a random-initialized parameter iteratively in the opposite direction of its partial derivative sign by a small positive random number, which is scaled by a tuned ratio of the model loss. This paper compares OSLD to stochastic gradient descent algorithm as the basic backpropagation algorithm and Adam as one of the best backpropagation algorithms in five benchmark models. Experimental results show that OSLD is very competitive to Adam in small and moderate depth models, and OSLD outperforms Adam in very long models. Moreover, OSLD is compatible with current backpropagation architectures except for learning rates. Finally, OSLD is stable and opens more choices in front of the very deep multi-layer neural networks. • A new back-propagated gradient descent optimization algorithm without a vanishing problem is proposed. • The proposed algorithm is called oriented stochastic loss descent (OSLD). • OSLD updates w i in the opposite side of its gradient sign by a tuned ratio of random loss. • OSLD is competitive to Adam and is compatible with most backpropagation architectures. • OSLD is stable and opens more choices in front of very deep multi-layer neural networks. [ABSTRACT FROM AUTHOR]

The multi-objective vehicle routing problem with stochastic demand (MO-VRPSD) is much harder to tackle than other traditional vehicle routing problems (VRPs), due to the uncertainty in customer demands and potentially conflicted objectives. In this paper, we present an improved multi-objective learnable evolution model (IMOLEM) to solve MO-VRPSD with three objectives of travel distance, driver remuneration and number of vehicles. In our method, a machine learning algorithm, i.e., decision tree, is exploited to help find and guide the desirable direction of evolution process. To cope with the key issue of "route failure" caused due to stochastic customer demands, we propose a novel chromosome representation based on priority with bubbles. Moreover, an efficient nondominated sort using a sequential search strategy (ENS-SS) in conjunction with some heuristic operations are leveraged to handle the multi-objective property of the problem. Our algorithm is evaluated on the instances of modified Solomon VRP benchmark. Experimental results show that the proposed IMOLEM is capable to find better Pareto front of solutions and also deliver superior performance to other evolutionary algorithms. [ABSTRACT FROM AUTHOR]

Abstract: In reinforcement learning, when samples are limited in some real applications, Least-Squares Temporal Difference (LSTD) learning is prone to over-fitting, which can be overcome by the introduction of regularization. However, the solution of LSTD with regularization still depends on costly matrix inversion operations. In this paper we investigate the L2-norm regularized LSTD learning and propose an efficient algorithm to avoid expensive computational cost. We derive LSTD using Bellman operator along with projection operator. The L2-norm penalty is introduced to avoid over-fitting. We also describe the difference between Bellman residual minimization and LSTD. Then we propose an efficient recursive least-squares algorithm for L2-norm regularized LSTD, which can eliminate matrix inversion operations and decrease computational complexity effectively. We present empirical comparisons on the Boyan chain problem. The results show that the performance of the new algorithm is better than that of regularized LSTD. [Copyright &y& Elsevier]

Abstract: This paper presents a Crotch Ensemble classification model for high dimensional data. A Crotch Ensemble is obtained from a decision cluster tree built by calling a clustering algorithm recursively. A crotch is an inner node of the tree together with its direct children. If the children of a crotch have more than one dominant class, the crotch is defined as a crotch predictor. Each crotch predictor constructs a classifier by itself. A Crotch Ensemble consists of a set of crotch predictors. When classifying a new object, a subset of crotch predictors is selected according to the distances between the object and the crotch predictors. A classification is made on the object as the class predicted by the crotch predictors with the maximum accumulative weights. The experimental results on both synthetic and real data have shown that the Crotch Ensemble model can get better classification results on high dimensional data than other classification methods. [Copyright &y& Elsevier]

Abstract: It has been shown that the structural information of data may contain useful prior domain knowledge for training a classifier. How to apply the structural information of data to build a good classifier is a new research focus recently. As we all know, the all existing structural large margin methods are the common in considering all structural information within classes into one model. In fact, these methods do not balance all structural information’s relationships both infra-class and inter-class, which directly results in these prior information not being exploited sufficiently. In this paper, we design a new Structural Twin Support Vector Machine (called -TWSVM). Unlike existing methods based on structural information, -TWSVM uses two hyperplanes to decide the category of new data, of which each model only considers one class’s structural information and closer to the class at the same time far away from the other class. This makes -TWSVM fully exploit these prior knowledge to directly improve the algorithm’s the capacity of generalization. All experiments show that our proposed method is rigidly superior to the state-of-the-art algorithms based on structural information of data in both computation time and classification accuracy. [Copyright &y& Elsevier]

Abstract: In this paper we introduce (C, n)-strategy which improves the former C-strategy for on-line calibration of Clonal Selection based algorithms. In this approach, we are focused on a trade-off between the intensification and the diversification of the algorithm search. By using our approach, it allows us to reduce the number of the parameters of the algorithm respecting both the original design of the algorithm and its performance. The number of selected cells and the number of clones are dynamically controlled on-line, according to the algorithm’s behavior. We report statistical comparisons using well-known clonalg based algorithms for solving combinatorial optimization problems. From the tests, we conclude that the tuning effort for Clonalg based algorithms is strongly reduced using our technique. Moreover, the dynamic control does not decrease the performance of the original version of the algorithm. On the contrary, it has shown to improve it. [Copyright &y& Elsevier]

Abstract: Feature selection aims at finding a feature subset that has the most discriminative information from the original feature set. In this paper, we firstly present a new scheme for feature relevance, interdependence and redundancy analysis using information theoretic criteria. Then, a dynamic weighting-based feature selection algorithm is proposed, which not only selects the most relevant features and eliminates redundant features, but also tries to retain useful intrinsic groups of interdependent features. The primary characteristic of the method is that the feature is weighted according to its interaction with the selected features. And the weight of features will be dynamically updated after each candidate feature has been selected. To verify the effectiveness of our method, experimental comparisons on six UCI data sets and four gene microarray datasets are carried out using three typical classifiers. The results indicate that our proposed method achieves promising improvement on feature selection and classification accuracy. [Copyright &y& Elsevier]

Abstract: Ensemble pruning is crucial for the consideration of both efficiency and predictive accuracy of an ensemble system. This paper proposes a new Competitive technique for Ensemble Pruning based on Cross-Validation (CEPCV). The data to be learnt by neural computing models are mostly drifting with time and environment, therefore a dynamic ensemble pruning method is indispensable for practical applications, while the proposed CEPCV method is just the kind of dynamic ensemble pruning method, which can realize on-line ensemble pruning and take full advantage of potentially valuable information. The algorithm naturally inherits the predominance of cross-validation technique, which implies that those networks regarded as winners in selective competitions and chosen into the pruned ensemble have the “strongest” generalization capability. It is essentially based on the strategy of “divide and rule, collect the wisdom”, and might alleviate the local minima problem of many conventional ensemble pruning approaches only at the cost of a little greater computational cost, which is acceptable to most applications of ensemble learning. The comparative experiments among the four ensemble pruning algorithms, including: CEPCV and the state-of-the-art Directed Hill Climbing Ensemble Pruning (DHCEP) algorithm and two baseline methods, i.e. BSM, which chooses the Best Single Model in the initial ensemble based on their performances on the pruning set, and ALL, which reserves all network members of the initial ensemble, on ten benchmark classification tasks, demonstrate the effectiveness and validity of CEPCV. [Copyright &y& Elsevier]

Abstract: Affinity propagation (AP) is introduced as an unsupervised learning algorithm for exemplar-based clustering. A few methods are stated to extend the AP model to account for semi-supervised clustering. In this paper, constraint (cannot-link and must-link) projections are illustrated for semi-supervised AP (CPSSAP), a hierarchical semi-supervised clustering algorithm. It is flexible for the relaxation of some constraints during the learning stage. First, the data points of instance-level constraints and other data points are together projected in a lower dimensional space guided by the constraints. Then, AP is performed on the new data points in the lower dimensional space. Finally, a few datasets are chosen for experimentation from the UCI machine learning repository. The results show that CPSSAP performs better than some existing algorithms. Furthermore, visualizations of the original data and data after the projections show that the data points overlap less after the constraint projections of the datasets. [Copyright &y& Elsevier]

Abstract: Translation initiation site (TIS) recognition is one of the first steps in gene structure prediction, and one of the common components in any gene recognition system. Many methods have been described in the literature to identify TIS in transcribed sequences such as mRNA, EST and cDNA sequences. However, the recognition of TIS in DNA sequences is a far more challenging task, and the methods described so far for transcripts achieve poor results in DNA sequences. Most methods approach this problem taking into account its biological characteristics. In this work we try a different view, considering this classification problem from a purely machine learning perspective. From the point of view of machine learning, TIS recognition is a class imbalance problem. Thus, in this paper we approach TIS recognition from this angle, and apply the different methods that have been developed to deal with imbalanced datasets. The proposed approach has two advantages. Firstly, it improves the results using standard classification methods. Secondly, it broadens the set of classification algorithms that can be used, as some of the class-imbalance methods, such as undersampling, are also useful as methods for scaling up data mining algorithms as they reduce the size of the dataset. In this way, classifiers that cannot be applied to the whole dataset, due to long training time or large memory requirements, can be used when undersampling method is applied. Results show an advantage of class imbalance methods with respect to the same methods applied without considering the class imbalance nature of the problem. The applied methods are also able to improve the results obtained with the best method in the literature, which is based on looking for the next in-frame stop codon from the putative TIS that must be predicted. [Copyright &y& Elsevier]

Abstract: Semi-supervised dimensional reduction methods play an important role in pattern recognition, which are likely to be more suitable for plant leaf and palmprint classification, since labeling plant leaf and palmprint often requires expensive human labor, whereas unlabeled plant leaf and palmprint is far easier to obtain at very low cost. In this paper, we attempt to utilize the unlabeled data to aid plant leaf and palmprint classification task with the limited number of the labeled plant leaf or palmprint data, and propose a semi-supervised locally discriminant projection (SSLDP) algorithm for plant leaf and palmprint classification. By making use of both labeled and unlabeled data in learning a transformation for dimensionality reduction, the proposed method can overcome the small-sample-size (SSS) problem under the situation where labeled data are scant. In SSLDP, the labeled data points, combined with the unlabeled data ones, are used to construct the within-class and between-class weight matrices incorporating the neighborhood information of the data set. The experiments on plant leaf and palmprint databases demonstrate that SSLDP is effective and feasible for plant leaf and palmprint classification. [ABSTRACT FROM AUTHOR]