Search

The evaluation of classifiers' performances plays a critical role in construction and selection of classification model. Although many performance metrics have been proposed in machine learning community, no general guidelines are available among practitioners regarding which metric to be selected for evaluating a classifier's performance. In this paper, we attempt to provide practitioners with a strategy on selecting performance metrics for classifier evaluation. Firstly, the authors investigate seven widely used performance metrics, namely classification accuracy, F-measure, kappa statistic, root mean square error, mean absolute error, the area under the receiver operating curve, and the area under the precision-recall curve. Secondly, the authors resort to using Pearson linear correlation and Spearman rank correlation to analyses the potential relationship among these seven metrics. Experimental results show that these commonly used metrics can be divided into three groups, and all metrics within a given group are highly correlated but less correlated with metrics from different groups.

Label ranking studies the issue of learning a model that maps instances to rankings over a finite set of predefined labels. In order to relieve the cost of memory and time during training and prediction, we propose a novel approach for label ranking problem based on Gaussian mixture model in this paper. The key idea of the approach is to divide the label ranking training data into multiple clusters using clustering algorithm, and each cluster is described by a Gaussian prototype. Then, a Gaussian mixture model is introduced to model the mapping from instances to rankings. Finally, a predicted ranking is obtained with maximum posterior probability. In the experiments, we compare our method with two state-of-the-art label ranking approaches. Experimental results show that our method is fully competitive in terms of predictive accuracy. Moreover, the proposed method also provides a measure of the reliability of the corresponding predicted ranking.

The problem of learning label rankings is receiving increasing attention from machine learning and data mining community. Its goal is to learn a mapping from instances to rankings over a finite number of labels. In this paper, we devote to giving an overview of the state-of-the-art in the area of label ranking, and providing a basic taxonomy of the label ranking algorithms. Specifically, we classify these label ranking algorithms into four categories, namely decomposition methods, probabilistic methods, similarity-based methods, and other methods. We pay particular attention to the latest advances in each. Also, we discuss their strengths and weaknesses, and highlight some interesting challenges that remain to be solved.

Non-intrusive appliance load monitoring (NILM) is the process for disaggregating total electricity consumption into its contributing appliances. Unsupervised NILM algorithm is an attractive method as the need for data annotation can be eliminated. In order to evaluate the performance of unsupervised NILM learning algorithm, most of the research work evaluates the algorithm performance using accuracy type metrics. These assessment metrics can not only identify total disaggregation error, but also distinguish the disaggregation error of each other single appliance. In order to better quantify the nature of disaggregation algorithm, we propose two assessment metrics: the total disaggregation accuracy and the disaggregation accuracy for a single appliance. In this paper, we evaluate the performance of unsupervised disaggregation methods using these assessment metrics. The experiment results show that our assessment metrics can evaluate the unsupervised algorithms effectively.

As techniques of data acquisition and data storage rapidly developed, more and larger datasets are very easily faced in machine learning. In order to avoid excessive storage and time consuming, and possibly to improve generalization accuracy by removing noise, several works presented as reduction techniques have been proposed. In this paper, firstly, we will review most traditional and typical reduction algorithms and find out their strengths and weaknesses, respectively. In addition, nine typical reduction algorithms are compared performing on 16 classification tasks. At last, some valuable directions for further research are proposed based on discussions and conclusion of traditional algorithms mentioned.

Learning is obtaining an underlying rule by using training data sampled from the environment. In many practical situations in inductive learning algorithms, it is often expected to further improve the generalization capability after the learning process has been completed if new data are available. One of the common approaches is to add training data to the learning algorithm and retrain it, but retraining for each new data point or data set can be very expensive. In view of the learning methods of human beings, it seems natural to build posterior learning results upon prior results. Firstly, in this paper, we proposed an updating procedure for least square support vector machine(LS–SVM). If initial concept would be built up by LS–SVM inductive algorithm, then concept updated is the normal solution corresponding to the initial concept learned. Secondly, we discuss a general framework for updating learned concept. Finally, we illustrate the updating method and evaluate it on toy data and real data, their results show that the performance after updating is improved and almost equal to the performance of LS–SVM retrained on whole data.

As a kind of one-class classification algorithm, support vector data description (SVDD) was used to distinguish target objects from outliers objects. By introducing the mathematics concept of curvature, we reduced the training samples according to the value of support vectors locating on the classification boundary, then a reduced learning support vector data description (RSVDD) algorithm based on the reduced support machines was presented in this paper. Compared with the traditional SVDD, RSVDD only used reduced support machines to construct the final classification boundary, so the training time was decreased greatly, meanwhile, the classification performance of the RSVDD can hardly be dropped obviously, and RSVDD is useful and effective especially in large scale training samples problem.

Traditional support vector regression (SVR) algorithm can not handle training data which contain incomplete information. In order to overcome this shortcoming, this paper introduced the interval number to represent the incomplete information, and uses interval operation to replace the real operation, then an uncertainty support vector regression algorithm (USVR) was proposed, which can extend the SVR's application area and improve its learning ability. We used the USVR algorithm to web information mining experiment, and the results show that this algorithm is feasible and effective.

In many practical situations in support vector machine learning, it is often expected to further improve the generalization capability after the learning process has been completed. One of the common approaches is to add training data to the support vector machine (SVM) and retrain SVM, but retraining for each new data point or data set can be very expensive. In view of the learning method of human beings, it seems natural to build posterior learning results upon prior results. In this paper, we propose an incremental batch training method called support vector pursuit learning (SVPL). The SVPL uses an incremental updating model similar to standard SVM to update the trained SVM parameters. SVPL provides the same learning performance as that obtained by batch learning, but is faster than other methods. The effectiveness of the presented method is demonstrated through experiments.

In many practical situations in inductive learning algorithms, it is often expected to further improve the generalization capability after the learning process has been completed if new data are available. One of the common approaches is to add training data to the learning algorithm and retrain it, but retraining for each new data point or data set can be very expensive. In view of the learning methods of human beings, it seems natural to build posterior learning results upon prior results. In this paper, we apply Support Vector Machine(SVM) to the concept updating procedure. If initial concept would be built up by inductive algorithm, then concept updated is the normal solution corresponding to the initial concept learned. It was shown that concept learned would not change if the new available data located in error-insensitive zone. Especially, concept initially learned and updated by SVR induces an incremental SVR approximately learning method for large scale data. We tested our method on toys data sets and 7 regression bench mark data set. It shown that generalization capacity after updating with SVR was improved according to FVU or MSE on the independent test set.

Learning is obtaining an underlying rule by using training data sampled from the environment. In many practical situations in inductive learning algorithms, it is often expected to further improve the generalization capability after the learning process has been completed if new data are available. One of the common approaches is to add training data to the learning algorithm and retrain it, but retraining for each new data point or data set can be very expensive. In view of the learning methods of human beings, it seems natural to build posterior learning results upon prior results. In this paper, we proposed an updating method for support vector machine (SVM). If initial concept would be built up by inductive algorithm, then concept updated is the normal solution corresponding to the initial concept learned. It was shown that concept learned would not change if the new available data located in error-insensitive zone. Especially, concept initially learned and updated by SVR induces an incremental SVR approximately learning method for large scale data. We tested our method on toys data sets and 7 regression bench mark data set. It is shown that generalization capacity after updating with SVR was improved according to FVU or MSE on the independent test set.

In this paper, we develop the conditional evidence theory and apply it to knowledge discovery in database. In this theory, we assume that a priori knowledge about generic situation and evidence about situation at hand can be modelled by two independent random sets. Dempster’s rule of combination is a popular method used in evidence theory, we think that this rule can be applied to knowledge revision, but isn’t appropriate for knowledge updating. Based on random set theory, we develop a new bayesian updating rule in evidence theory. More importantly, we show that bayesian updating rule can be performed incrementally by using Mobius transforms.

Support Vector Machines (SVMs) have become a popular tool for learning with large amounts of high dimensional data. But it may sometimes be preferable to learn incrementally from previous SVM results, as SVMs which involve the solution of a quadratic programming problem suffer from the problem of large memory requirement and CPU time when trained in batch mode on large data sets. And the SVMs may be used in online learning setting. In this paper an approach for incremental learning with Support Vector Machines is presented. We define the normal solution of the incremental learning for SVMs which is defined as the solution minimizing a given positive-definite quadratic form in the coordinates of the difference vector between the normal vectors at the (k-1)-th and k-th incremental step and discuss the relation to standard SVM. It was shown that concept learned at last step will not change if new data satisfy separable condition and empirical evidence is given to prove that this approach can effectively deal with changes in the target concept that are results of the incremental learning setting according to three evaluation criteria: stability, improvement and recoverability.

The correct selection of performance metrics is one of the most key issues in evaluating classifier’s performance. Although many performance metrics have been proposed and used in machine learning community, there is not any common conclusions among practitioners regarding which metric to choose for evaluating a classifier’s performance. In this paper, we attempt to investigate the potential relationship among some common used performance metrics. Based on definitions, We first classify seven most widely performance metrics into three groups, namely threshold metrics, rank metrics, and probability metrics. Then, we focus on using Pearson linear correlation and Spearman rank correlation to investigate the relationship among these metrics. Experimental results show the reasonableness of classifying seven common used metrics into three groups. This can be useful for helping practitioners enhance understanding about the different relationships and groupings among the performance metrics.