Search

Ensemble methods have shown to be more effective than monolithic classifiers, in particular when diversity holds among their components. How to enforce diversity in classifier ensembles has received much attention from machine learning researchers, yielding a variety of different techniques and algorithms. In this paper, a novel algorithm for ensemble classifiers is proposed, in which ensemble components are trained with focus on different regions of the sample space. In so doing, diversity is mainly a consequence of the intention to limit the scope of base classifiers. The algorithm proposed in this paper shares roots with several ensemble paradigms, in particular with random forests, as it generates forests of decision trees as well. As decision trees are trained with focus on specific subsets of the sample space, the resulting ensemble is in fact a forest of “local” trees. Comparative experimental results highlight that, on average, these ensembles perform better than other relevant kinds of ensemble classifiers, including random forests. [ABSTRACT FROM AUTHOR]

• The first attempt to design a GEP-based codematrix generation algorithm. The individual structure of GEP allows the codematrix to be optimized in a much faster and more effective way. • The feature selection scheme is combined with the design of codematrix, to accomplish the feature selection and codematrix optimization simultaneously. • A greedy strategy-based adaptive optimization method to further improve the individual's quality to deal with hard classes by appending proper columns to the codematrix. The multi-class classification task is one of the most common tasks in machine learning. As a typical solution based on a partitioning strategy, Error-Correcting Output Codes (ECOC) can transform a multi-class classification problem into multiple binary classification problems. The key of ECOC is to construct an effective codematrix to represent a set of class decomposition schemes, which transforms a multiclass problem into a group of binary class problems. Consequently, the design of a fast and effective ECOC codematrix generation method is of great research significance and value for solving multi-class classification problems. In ECOC algorithms, the design of codematrix is treated as a combination problem between different code columns, in which the evolutionary algorithm shows a great advantage. Based on this consideration, the Gene Expression Programming (GEP) is applied to search for the codematrix with high performance because its expressive tree structure makes it well represent codematrcies for subsequent optimization operations. This paper proposes an adaptive ECOC algorithm based on Gene Expression Programming (GEP) and similarity measurement matrix, named GEP-ECOC. In our GEP, each individual represents a set of columns to form a random ECOC codematrix, which is optimized in the evolutionary process. Meanwhile, the crossover and mutation operations are modified to include a legality checking process to ensure that the generated codematrix satisfies the ECOC constraints. The GEP-based ECOC codematrix generation algorithm can quickly produce a codematrix with better performance, which ensures the efficiency of the algorithm to a certain extent. In addition, an adaptive algorithm based on a similarity measurement matrix is proposed to add new columns to the current codematrix, aiming to better handle hard classes. Our algorithm is compared with other algorithms on various data sets, and the experimental results confirm that our GEP-ECOC can balance the efficiency and performance of the algorithm and achieve higher performance. [ABSTRACT FROM AUTHOR]

• Convolutional Neural Networks can be improved in terms of the classification performance and robustness by using variable weight structures. • Analysis of different data processing methods, models' robustness and statistical properties. • Comparative analysis of variable weight convolutional neural networks and other widely used machine learning techniques. • Medical applications to the classification of seizure phases and types. Deep learning techniques have recently achieved impressive results and raised expectations in the domains of medical diagnosis and physiological signal processing. The widely adopted methods include convolutional neural networks (CNNs) and recurrent neural networks (RNNs). However, the existing models possess static connection weights between layers, which might limit the generalization capability and the classification performance of the models as the weights of different layers are fixed after training. Furthermore, to deal with a large amount of data, a neural network with a sufficiently large size is required. This paper proposes the variable weight convolutional neural networks (VWCNNs), which are a type of network structure employing dynamic weights instead of static weights in their convolutional layers and fully-connected layers. VWCNNs are able to adapt to different characteristics of input data and can be viewed as an infinite number of traditional, fixed-weight CNNs. We will show that the proposed VWCNN structure outperforms the conventional CNN in terms of the classification accuracy, generalization capability, and robustness when the inputs are contaminated by noise. In this paper, VWCNNs are applied to the classification of three seizure phases (seizure-free, pre-seizure and seizure) based on measured electroencephalography (EEG) data. VWCNNs achieve 100% test accuracy and show strong robustness in the classification of the three seizure phases, and thus show the potential to be a useful classification tool for medical diagnosis. Furthermore, the classification of seven types of seizures is investigated in this paper using the world's largest open source database of seizure recordings, TUH EEG seizure corpus. Comparisons with conventional CNNs, RNN, MobileNet, ResNet, DenseNet and traditional machine learning methods including random forest, decision tree, support vector machine, K-nearest neighbours, standard neural networks, and Naïve Bayes are being conducted using realistic test data sets. The results demonstrate that VWCNNs have advantages over other classifiers in terms of classification accuracy and robustness. [ABSTRACT FROM AUTHOR]

The paper is devoted to two problems: (1) reinforcement of SVM algorithms, and (2) justification of memorization mechanisms for generalization. (1) Current SVM algorithm was designed for the case when the risk for the set of nonnegative slack variables is defined by l 1 norm. In this paper, along with that classical l 1 norm, we consider risks defined by l 2 norm and l ∞ norm. Using these norms, we formulate several modifications of the existing SVM algorithm and show that the resulting modified SVM algorithms can improve (sometimes significantly) the classification performance. (2) Generalization ability of existing learning algorithms is usually explained by arguments involving uniform convergence of empirical losses to the corresponding expected losses over a given set of functions. However, along with bounds for uniform convergence of empirical losses to the expected losses, the VC theory also provides bounds for relative uniform convergence. These bounds lead to a more accurate estimate of the expected loss. Advanced methods of estimating of expected risk of error have to leverage these bounds, which also support mechanisms of training data memorization, which, as the paper demonstrates, can improve classification performance. [ABSTRACT FROM AUTHOR]

Abstract: Incremental learning of neural networks has attracted much interest in recent years due to its wide applicability to large scale data sets and to distributed learning scenarios. Moreover, nonstationary learning paradigms have also emerged as a subarea of study in Machine Learning literature due to the problems of classical methods when dealing with data set shifts. In this paper we present an algorithm to train single layer neural networks with nonlinear output functions that take into account incremental, nonstationary and distributed learning scenarios. Moreover, it is demonstrated that introducing a regularization term into the proposed model is equivalent to choosing a particular initialization for the devised training algorithm, which may be suitable for real time systems that have to work under noisy conditions. In addition, the algorithm includes some previous models as special cases and can be used as a block component to build more complex models such as multilayer perceptrons, extending the capacity of these models to incremental, nonstationary and distributed learning paradigms. In this paper, the proposed algorithm is tested with standard data sets and compared with previous approaches, demonstrating its higher accuracy. [Copyright &y& Elsevier]

Abstract: Graph based methods are among the most active and applicable approaches studied in semi-supervised learning. The problem of neighborhood graph construction for these methods is addressed in this paper. Neighborhood graph construction plays a key role in the quality of the classification in graph based methods. Several unsupervised graph construction methods have been proposed that have addressed issues such as data noise, geometrical properties of the underlying manifold and graph hyper-parameters selection. In contrast, in order to adapt the graph construction to the given classification task, many of the recent graph construction methods take advantage of the data labels. However, these methods are not efficient since the hypothesis space of their possible neighborhood graphs is limited. In this paper, we first prove that the optimal neighborhood graph is a subgraph of a graph for a large enough , which is much smaller than the total number of data points. Therefore, we propose to use all the subgraphs of graph as the hypothesis space. In addition, we show that most of the previous supervised graph construction methods are implicitly optimizing the smoothness functional with respect to the neighborhood graph parameters. Finally, we provide an algorithm to optimize the smoothness functional with respect to the neighborhood graph in the proposed hypothesis space. Experimental results on various data sets show that the proposed graph construction algorithm mostly outperforms the popular k-NN based construction and other state-of-the-art methods. [Copyright &y& Elsevier]

Highlights • Optimization is an important issue in machine learning because many machine learning models are reformulated as optimization problems. Different kinds of machine learning algorithms mainly focus on minimizing their emiprical loss like deep learning, logisitic regression, and support vector machine. Because data is explosively growing, it is challenging to deal with a large-scale optimization problem. We propose a fast and most condition number free algorithm for train machine learning algorithms. This algorithm has several advantages. • Contrast to previous work which approximates Hessian matrix by sketched or subsampled Hessian directly, we use the sketched or subsampled Hessian matrix as preconditioner and get the direction vector by preconditioned conjugate gradient method. Accordingly, we propose a novel stochastic second-order algorithm called Preconditioned Newton Conjugate Gradient with Sketched Hessian (PNCG). This strategy makes our algorithm almost independent on the condition number. • We numerically demonstrate the effectiveness and robustness of our algorithms in widely used machine learning algorithm Ridge Logistic Regression on several real datasets with different condition number. Our algorithms show good performance compared with existing second order methods in all cases, especially when condition number is large. Abstract Optimization is an important issue in machine learning because many machine learning models are reformulated as optimization problems. Different kinds of machine learning algorithms mainly focus on minimizing their empirical loss like deep learning, logistic regression, and support vector machine. Because data is explosively growing, it is challenging to deal with a large-scale optimization problem. Recently, stochastic second-order methods have emerged to attract much attention due to their efficiency in each iteration. These methods show good performance on training machine learning algorithms like logistic regression and support vector machine. However, the computational complexity of existing stochastic second-order methods heavily depends on the condition number of the Hessian. In this paper, we propose a new Newton-like method called Preconditioned Newton Conjugate Gradient with Sketched Hessian (PNCG). The runtime complexity of PNCG is at most logarithmic in the condition number of the Hessian. PNCG exhibits advantages over existing subsampled Newton methods especially when the Hessian matrix in question is ill-conditioned. We also show that our method has good performance on training machine learning algorithm empirically. The results show consistent improvements in computational efficiency. [ABSTRACT FROM AUTHOR]

Abstract In streaming data classification, most of the existing methods assume that all arrived evolving data are completely labeled. One challenge is that some applications where only small amount of labeled examples are available for training. Incremental semi-supervised learning algorithms have been proposed for regularizing neural networks by incorporating various side information, such as pairwise constraints or user-provided labels. However, it is hard to put them into practice, especially for non-stationary environments due to the effectiveness and parameter sensitivity of such algorithms. In this paper, we propose a novel incremental semi-supervised learning framework on streaming data. Each layer of model is comprised of a generative network, a discriminant structure and the bridge. The generative network uses dynamic feature learning based on autoencoders to learn generative features from streaming data which has been demonstrated its potential in learning latent feature representations. In addition, the discriminant structure regularizes the network construction via building pairwise similarity and dissimilarity constraints. It is also used for facilitating the parameter learning of the generative network. The network and structure are integrated into a joint learning framework and bridged by enforcing the correlation of their parameters, which balances the flexible incorporation of supervision information and numerical tractability for non-stationary environments as well as explores the intrinsic data structure. Moreover, an efficient algorithm is designed to solve the proposed optimization problem and we also give an ensemble method. Particularly, when multiple layers of model are stacked, the performance is significantly boosted. Finally, to validate the effectiveness of the proposed method, extensive experiments are conducted on synthetic and real-life datasets. The experimental results demonstrate that the performance of the proposed algorithms is superior to some state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Highlights • Networks and graphs are naturally embedded in Euclidean hyperspheres. • Communicability embedding of networks/graphs reveals clusters in networks. • Nonmetric multidimensional scaling allows visualization of networks in 3D communicability space. • Communicability clusters of papers in a citation network reveal levels of mathematization. • Communicability clusters in a gene-gene network reveal genes that co-participate in cancer and other diseases. Abstract A complex network is a condensed representation of the relational topological framework of a complex system. A main reason for the existence of such networks is the transmission of items through the entities of these complex systems. Here, we consider a communicability function that accounts for the routes through which items flow on networks. Such a function induces a natural embedding of a network in a Euclidean high-dimensional sphere. We use one of the geometric parameters of this embedding, namely the angle between the position vectors of the nodes in the hyperspheres, to extract structural information from networks. First we propose a simple method for visualizing networks by reducing the dimensionality of the communicability space to 3-dimensional spheres. Secondly, we use clustering analysis to cluster the nodes of the networks based on their similarities in terms of their capacity to successfully deliver information through the network. After testing these approaches in benchmark networks and compare them with the most used clustering methods in networks we analyze two real-world examples. In the first, consisting of a citation network, we discover citation groups that reflect the level of mathematics used in their publications. In the second, we discover groups of genes that coparticipate in human diseases, reporting a few genes that coparticipate in cancer and other diseases. Both examples emphasize the potential of the current methodology for the discovery of new patterns in relational data. [ABSTRACT FROM AUTHOR]

Abstract In this paper, we propose the use of data symmetries, in the sense of equivalences under signal transformations, as priors for learning symmetry-adapted data representations, i.e., representations that are equivariant to these transformations. We rely on a group-theoretic definition of equivariance and provide conditions for enforcing a learned representation, for example the weights in a neural network layer or the atoms in a dictionary, to have the structure of a group and specifically the group structure in the distribution of the input. By reducing the analysis of generic group symmetries to permutation symmetries, we devise a regularization scheme for representation learning algorithm, using an unlabeled training set. The proposed regularization is aimed to be a conceptual, theoretical and computational proof of concept for symmetry-adapted representation learning, where the learned data representations are equivariant or invariant to transformations, without explicit knowledge of the underlying symmetries in the data. [ABSTRACT FROM AUTHOR]

Abstract: Class imbalance limits the performance of most learning algorithms since they cannot cope with large differences between the number of samples in each class, resulting in a low predictive accuracy over the minority class. In this respect, several papers proposed algorithms aiming at achieving more balanced performance. However, balancing the recognition accuracies for each class very often harms the global accuracy. Indeed, in these cases the accuracy over the minority class increases while the accuracy over the majority one decreases. This paper proposes an approach to overcome this limitation: for each classification act, it chooses between the output of a classifier trained on the original skewed distribution and the output of a classifier trained according to a learning method addressing the course of imbalanced data. This choice is driven by a parameter whose value maximizes, on a validation set, two objective functions, i.e. the global accuracy and the accuracies for each class. A series of experiments on ten public datasets with different proportions between the majority and minority classes show that the proposed approach provides more balanced recognition accuracies than classifiers trained according to traditional learning methods for imbalanced data as well as larger global accuracy than classifiers trained on the original skewed distribution. [Copyright &y& Elsevier]

Due to the ability of capturing the geometry structure of data manifold, context-sensitive similarity has demonstrated impressive performances in the retrieval task. The key idea of context-sensitive similarity is that the similarity between two data points can be more reliably estimated with the local context of other points in the affinity graph. Therefore, neighborhood selection is a crucial factor for those algorithms, which affects the performance dramatically. In this paper, we propose a new algorithm called Smooth Neighborhood (SN) that mines the neighborhood structure to satisfy the manifold assumption. By doing so, nearby points on the underlying manifold are guaranteed to yield similar neighbors as much as possible. Moreover, SN is adjusted to tackle multiple affinity graphs by imposing a weight learning paradigm, and this is the primary difference compared with related works which are only applicable with one affinity graph. Finally, we integrate SN with Sparse Contextual Activation (SCA), a representative context-sensitive similarity proposed recently. Extensive experimental results and comparisons manifest that with the neighborhood structure generated by SN, the proposed framework can yield state-of-the-art performances on shape retrieval, image retrieval and 3D model retrieval. [ABSTRACT FROM AUTHOR]

The convex clustering formulation of Chi and Lange (2015) is revisited. While this formulation can be precisely and efficiently solved, it uses the standard Euclidean metric to measure the distance between the data points and their corresponding cluster centers and hence its performance deteriorates significantly in the presence of outlier features. To address this issue, this paper considers a formulation that combines convex clustering with metric learning. It is shown that: (1) for any given positive definite Mahalanobis distance metric, the problem of convex clustering can be precisely and efficiently solved using the Alternating Direction Method of Multipliers; (2) the problem of learning a positive definite Mahalanobis distance metric admits a closed-form solution; (3) an algorithm that alternates between convex clustering and metric learning can provide a significant performance boost over not only the original convex clustering formulation but also the recently proposed robust convex clustering formulation of Wang et al. (2017). [ABSTRACT FROM AUTHOR]

Many real-world applications propose the request for sharing knowledge among different tasks or datasets. Transfer learning has been proposed to solve this kind of problems and it has been successfully applied in supervised learning and semi-supervised learning settings. However, its adoption in clustering, one of the most classical research problems in machine learning and data mining, is still scarce. Spectral clustering, as a major clustering algorithm with wide applications and better performance than k -means typically, has not been well incorporated with knowledge transfer. In this paper, we first consider the problem of learning from only one auxiliary unlabeled dataset for spectral clustering and propose a novel algorithm called transfer spectral clustering (TSC). Then, it is extended to the settings with multiple auxiliary tasks. TSC assumes the feature embeddings being shared with the auxiliary tasks and utilizes co-clustering to extract useful information from the auxiliary datasets to improve the clustering performance. TSC involves not only the data manifold information of individual task but also the feature manifold information shared between related tasks. An in-depth explanation of our algorithm together with a convergence analysis are provided. As demonstrated by the extensive experiments, TSC can effectively improve the clustering performance by using auxiliary unlabeled data when compared with other state-of-the-art clustering algorithms. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel approach for learning multi-label classifiers with the help of privileged information. Specifically, we use similarity constraints to capture the relationship between available information and privileged information, and use ranking constraints to capture the dependencies among multiple labels. By integrating similarity constraints and ranking constraints into the learning process of classifiers, the privileged information and the dependencies among multiple labels are exploited to construct better classifiers during training. A maximum margin classifier is adopted, and an efficient learning algorithm of the proposed method is also developed. We evaluate the proposed method on two applications: multiple object recognition from images with the help of implicit information about object importance conveyed by the list of manually annotated image tags; and multiple facial action unit detection from low-resolution images augmented by high-resolution images. Experimental results demonstrate that the proposed method can effectively take full advantage of privileged information and dependencies among multiple labels for better object recognition and better facial action unit detection. [ABSTRACT FROM AUTHOR]

Abstract: Distance metric is a key issue in many machine learning algorithms. This paper considers a general problem of learning from pairwise constraints in the form of must-links and cannot-links. As one kind of side information, a must-link indicates the pair of the two data points must be in a same class, while a cannot-link indicates that the two data points must be in two different classes. Given must-link and cannot-link information, our goal is to learn a Mahalanobis distance metric. Under this metric, we hope the distances of point pairs in must-links are as small as possible and those of point pairs in cannot-links are as large as possible. This task is formulated as a constrained optimization problem, in which the global optimum can be obtained effectively and efficiently. Finally, some applications in data clustering, interactive natural image segmentation and face pose estimation are given in this paper. Experimental results illustrate the effectiveness of our algorithm. [Copyright &y& Elsevier]

• Providing the condition under which an AHC does not effectively extract clusters. • Introducing the entanglement to measure the dendrogram quality. • Identifying the root cause of a density bias of traditional AHC algorithms. • Improving AHC performance with a data-dependent kernel. Agglomerative hierarchical clustering (AHC) is one of the popular clustering approaches. AHC generates a dendrogram that provides richer information and insights from a dataset than partitioning clustering. However, a major problem with existing distance-based AHC methods is: it fails to effectively identify adjacent clusters with varied densities, regardless of the cluster extraction methods applied to the resultant dendrogram. This paper aims to reveal the root cause of this issue and provides a solution by using a data-dependent kernel. We analyse the condition under which existing AHC methods fail to effectively extract clusters, and give the reason why the data-dependent kernel is an effective remedy. This leads to a new approach to kernerlise existing hierarchical clustering algorithms including the traditional AHC algorithms, HDBSCAN, GDL, PHA and HC-OT. Our extensive empirical evaluation shows that the recently introduced Isolation Kernel produces a higher quality or purer dendrogram than distance, Gaussian Kernel and adaptive Gaussian Kernel in all the above mentioned AHC algorithms. [ABSTRACT FROM AUTHOR]

Conventional Gaussian kernel c-means clustering algorithms are widely used in applications. However, Gaussian kernel functions have an important parameter, the width hyper-parameter, which needs to be tuned. Usually this parameter is tuned once and for all and it is the same for all variables. Thus, implicitly, all the variables are equally rescaled and therefore, they have equal importance on the clustering task. This paper presents Gaussian kernel c-means hard clustering algorithms with automated computation of the width hyper-parameters. In these kernel-based clustering algorithms, the hyper-parameters change at each iteration of the algorithm, they differ from variable to variable and can differ from cluster to cluster. Because each variable is rescaled differently according to its own hyper-parameter, these algorithms can select the important variables in the clustering process. Experiments using synthetic data sets and using UCI machine learning repository data sets corroborate the usefulness of the proposed algorithms. [ABSTRACT FROM AUTHOR]

Manifold learning aims to discover the low dimensional space where the input high dimensional data are embedded by preserving the geometric structure. Unfortunately, almost all the existing manifold learning methods were proposed under single view scenario, and they cannot be straightforwardly applied to multiple feature sets. Although concatenating multiple views into a single feature provides a plausible solution, it remains a question on how to better explore the independence and interdependence of different views while conducting manifold learning. In this paper, we propose a multi-view manifold learning with locality alignment (MVML-LA) framework to learn a common yet discriminative low-dimensional latent space that contain sufficient information of original inputs. Both supervised algorithm (S-MVML-LA) and unsupervised algorithm (U-MVML-LA) are developed. Experiments on benchmark real-world datasets demonstrate the superiority of our proposed S-MVML-LA and U-MVML-LA over existing state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Adaptive graph learning methods for clustering, which adjust a data similarity matrix while taking into account its clustering capability, have drawn increasing attention in recent years due to their promising clustering performance. Existing adaptive graph learning methods are based on either original data or linearly projected data and thus rely on the assumption that either representation is a good indicator of the underlying data structure. However, this assumption is sometimes not met in high dimensional data. Studies have shown that high-dimensional data in many problems tend to lie on an embedded nonlinear manifold structure. Motivated by this observation, in this paper, we develop dual data representations, i.e., original data and a nonlinear embedding of the data obtained via an Extreme Learning Machine (ELM)-based neural network, and propose to use them as the more reliable basis for graph learning. The resulting algorithm based on ELM and Constrained Laplacian Rank (ELM-CLR) further improves the clustering capability and robustness, while retaining the advantages of adaptive graph learning, such as not requiring any post-processing to extract cluster indicators. The empirical study shows that the proposed algorithm outperforms the state-of-the-art graph-based clustering methods on a broad range of benchmark datasets. [ABSTRACT FROM AUTHOR]

• First, we propose a fast implementation on the ESDE method. The key is to equivalently transform the large matrix problem of size d into a much smaller one of size n, where d is the data dimension and n is the number of training samples, with d ≫ n in practice. Numerical results demonstrate that the proposed algorithms are at least two orders of magnitude faster than their original counterparts, with no recognition accuracy lost. • Second, to the best of our knowledge, there are no incremental algorithms for matrix exponential discriminant methods till now. To fill in this gap, the second contribution of this paper is to propose incremental ESDE algorithms for incremental learning problems. • Third, the proposed fast implementation strategy and the incremental techniques also apply to other exponential discriminant analysis methods. In various pattern classification problems, semi-supervised learning methods have shown its effectiveness in utilizing unlabeled data to yield better performance than some supervised and unsupervised learning methods. Semi-supervised discriminant embedding (SDE) is a semi-supervised extension of local discriminant embedding (LDE). However, when dealing with high dimensional data, SDE often suffers from the small-sample-size (SSS) problem. In order to settle this problem, an exponential semi-supervised discriminant embedding (ESDE) method was proposed in F. Dornaika, Y. EI Traboulsi. Matrix exponential based semi-supervised discriminant embedding for image classification , Pattern Recognition, 61 (2017): 92–103], which makes use of the tool of matrix exponential. Despite its high discriminative ability, the computational overhead of ESDE is very large for high dimensional data. In order to cure this drawback, the first contribution of this paper is to propose a fast implementation on the ESDE method. The key is to equivalently transform the large matrix problem of size d into a much smaller one of size n , where d is the data dimension and n is the number of training samples, with d ≫ n in practice. On the other hand, in many real world applications, it is likely that whole labeled training set is unavailable beforehand, and the training data is obtained incrementally. Many incremental semi-supervised learning methods have been proposed to deal with this problem, to the best of our knowledge, however, there are no incremental algorithms for matrix exponential discriminant methods till now. To fill in this gap, the second contribution of this paper is to propose incremental ESDE algorithms for incremental learning problems. Numerical experiments on some real-world data sets show the numerical behavior of the proposed algorithms. [ABSTRACT FROM AUTHOR]

Kernel Fisher discriminant analysis (KFDA) is a very popular learning method for the purpose of classification. In this paper, we propose a novel learning algorithm to improve KFDA and make it very suitable for dealing with the large-scale and high-dimensional data sets. The proposed algorithm is termed individualized KFDA (IKFDA). IKFDA is based on individualized learning , i.e., a strategy to learn and classify the individual test samples one by one. Our approach seeks to find the appropriate training subset, referred to as learning area , for each individual test sample, and then employ the learning area to construct the KFDA model for the test sample. For each individual test sample, IKFDA exploits some types of similarity measures to determine a learning area that consists of the training samples that are most similar to the test sample. Compared with the traditional learning algorithms that often exploit the whole training set to construct the learning models without considering the distribution property of the test samples, IKFDA can adaptively learn the individual test samples. It is a powerful tool to deal with the real-world complicated data sets that are often very large-scale and high-dimensional, and are usually drawn from the different distributions. Extensive experiments show that the proposed algorithm can obtain good classification results. [ABSTRACT FROM AUTHOR]

• A hybrid Machine Learning (ML) approach for an efficient wrapper feature selection is proposed for enhancing the performance and reducing the computation time of IDS while increasing the accuracy to detect the intruder in IoT networks. • The hybrid ML method consists of two techniques: Differential Evolution (DE) and Extreme Learning Machine (ELM). The DE is responsible for choosing the beneficial features while the ELM is employed to assess the selected features. • Extensive experiments are implemented to evaluate the proposed approach using different performance metrics. In addition, to verify the efficiency of proposed model, the comparison is achieved with some related existing methods such as CFS + SVM [20] , CFS + ANN [19] , GA + Bagging [18] , CFS + RF [30] , and Chi-squared + RF [30]. The intrusion detection system (IDS) has gained a rapid increase of interest due to its widely recognized potential in various security fields, however, it suffers from several challenges. Different network datasets have several redundant and irrelevant features that affect the decision of the IDS classifier. Therefore, it is essential to decrease these features to improve the system performance. In this paper, an efficient wrapper feature selection method is proposed for improving the performance and decreasing the processing time of the IDS. The proposed approach employs a differential evaluation algorithm to select the useful features whilst the extreme learning machine classifier is applied after feature selection to evaluate the selected features. Many experiments are performed using the full NSL-KDD dataset to evaluate the performance of the proposed method. The results prove that the proposed approach can efficiently reduce the features, increase the accuracy, reduce the false alarm rates, and improve the processing time of the IDS in comparison to other recent related works. [ABSTRACT FROM AUTHOR]

This paper proposes a framework for binary classification of the time series with cyclic characteristics. The framework presents an iterative algorithm for learning the cyclic characteristics by introducing the discriminative frequency bands (DFBs) using the discriminant analysis along with k-means clustering method. The DFBs are employed by a hybrid model for learning dynamic characteristics of the time series within the cycles, using statistical and structural machine learning techniques. The framework offers a systematic procedure for finding the optimal design parameters associated with the hybrid model. The proposed model is optimized to detect the changes of the heart sound recordings (HSRs) related to aortic stenosis. Experimental results show that the proposed framework provides efficient tools for the classification of the HSRs based on the heart murmurs. It is also evidenced that the hybrid model, proposed by the framework, substantially improves the classification performance when it comes to detection of the heart disease. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, we propose a robust tracking algorithm to handle drifting problem. This algorithm consists of two parts: the first part is the G&D part that combines Generative model and Discriminative model for tracking, and the second part is the View-Based model for target appearance that corrects the result of the G&D part if necessary. In G&D part, we use the Maximum Margin Projection (MMP) to construct a graph model to preserve both local geometrical and discriminant structures of the data manifold in low dimensions. Therefore, such discriminative subspace combined with traditional generative subspace can benefit from both models. In addition, we address the problem of learning maximum margin projection under the Spectral Regression (SR) which results in significant savings in computational time. To further solve the drift, an online learned sparsely represented view-based model of the target is complementary to the G&D part. When the result of G&D part is unreliable, the view-based model can rectify the result in order to avoid drifting. Experimental results on several challenging video sequences demonstrate the effectiveness and robustness of our approach. [Copyright &y& Elsevier]

Abstract: In this paper, we introduce a novel algorithm to solve the problem of object tracking across multiple non-overlapping cameras by learning inter-camera transfer models. The transfer models are divided into two parts according to different kinds of cues, i.e. spatio-temporal cues and appearance cues. To learn spatio-temporal transfer models across cameras, an unsupervised topology recovering approach based on N-neighbor accumulated cross-correlations is proposed, which estimates the topology of a non-overlapping multi-camera network. Different from previous methods, the proposed topology recovering method can deal with large amounts of data without considering the size of time window. To learn inter-camera appearance transfer models, a color transfer method is used to model the changes of color characteristics across cameras, which has an advantage of low requirements to training samples, making update efficient when illumination conditions change. The experiments are performed on different datasets. Experimental results demonstrate the effectiveness of the proposed algorithm. [Copyright &y& Elsevier]

Abstract: Mixture discriminant analysis (MDA) and subclass discriminant analysis (SDA) belong to the supervised classification approaches. They have advantage over the standard linear discriminant analysis (LDA) in large sample size problems, since both of them divide the samples in each class into subclasses which keep locality but LDA does not. However, since the current MDA and SDA algorithms perform subclass division in just one step in the original data space before solving the generalized eigenvalue problem, two problems are exposed: (1) they ignore the relation among classes since subclass division is performed in each isolated class; (2) they cannot guarantee good performance of classifiers in the transformed space, because locality in the original data space may not be kept in the transformed space. To address these problems, this paper presents a new approach for subclass division based on k-means clustering in the projected space, class by class using the iterative steps under EM-alike framework. Experiments are performed on the artificial data set, the UCI machine learning data sets, the CENPARMI handwritten numeral database, the NUST603 handwritten Chinese character database, and the terrain cover database. Extensive experimental results demonstrate the performance advantages of the proposed method. [Copyright &y& Elsevier]

Abstract: This paper presents a novel feature-selection algorithm for data regression with a lot of irrelevant features. The proposed method is based on well-established machine-learning technique without any assumption about the underlying data distribution. The key idea in this method is to decompose an arbitrarily complex nonlinear problem into a set of locally linear ones through local information, and to learn globally feature relevance within the least squares loss framework. In contrast to other feature-selection algorithms for data regression, the learning of this method is efficient since the solution can be readily found through gradient descent with a simple update rule. Experiments on some synthetic and real-world data sets demonstrate the viability of our formulation of the feature-selection problem and the effectiveness of our algorithm. [Copyright &y& Elsevier]

• A novel explainable neural network is proposed for modeling main effects and structured interactions. • The GAMI-Net is a disentangled feedforward network with multiple additive subnetworks. • GAMI-Net takes into account three interpretability constraints: sparsity, heredity, marginal clarity. • An adaptive training algorithm is developed for training GAMI-Net efficiently. • GAMI-Net enjoys superior interpretability and outperforms benchmark methods. The lack of interpretability is an inevitable problem when using neural network models in real applications. In this paper, an explainable neural network based on generalized additive models with structured interactions (GAMI-Net) is proposed to pursue a good balance between prediction accuracy and model interpretability. GAMI-Net is a disentangled feedforward network with multiple additive subnetworks; each subnetwork consists of multiple hidden layers and is designed for capturing one main effect or one pairwise interaction. Three interpretability aspects are further considered, including a) sparsity, to select the most significant effects for parsimonious representations; b) heredity, a pairwise interaction could only be included when at least one of its parent main effects exists; and c) marginal clarity, to make main effects and pairwise interactions mutually distinguishable. An adaptive training algorithm is developed, where main effects are first trained and then pairwise interactions are fitted to the residuals. Numerical experiments on both synthetic functions and real-world datasets show that the proposed model enjoys superior interpretability and it maintains competitive prediction accuracy in comparison to the explainable boosting machine and other classic machine learning models. [ABSTRACT FROM AUTHOR]

• we first show that the solution paths produced by various algorithms have the property of piecewise linearity. • We point out model function builds the bridge between solution path and error path, and show that the piecewise linearity of solution path leads to the piecewise linearity of model function. • Based on the piecewise linearity of model function, we prove that a large class of error (or loss) functions are piecewise constant, linear, or quadratic w.r.t. the regularization parameter. Finally, we propose our GEP for the generalized error (or loss) functions and solution path algorithms, which guarantees to find the models with the minimum CV error. • The experimental results on a variety of datasets not only confirm our theoretical findings, but also show that the best model with our GEP has better generalization error on the test data, compared to the grid search, manual search, and random search. Model selection with cross validation (CV) is very popular in machine learning. However, CV with grid and other common search strategies cannot guarantee to find the model with minimum CV error, which is often the ultimate goal of model selection. Recently, various solution path algorithms have been proposed for several important learning algorithms including support vector classification, Lasso, and so on. However, they still do not guarantee to find the model with minimum CV error. In this paper, we first show that the solution paths produced by various algorithms have the property of piecewise linearity. Then, we prove that a large class of error (or loss) functions are piecewise constant, linear, or quadratic w.r.t. the regularization parameter, based on the solution path. Finally, we propose a new generalized error path algorithm (GEP), and prove that it will find the model with minimum CV error in a finite number of steps for the entire range of the regularization parameter. The experimental results on a variety of datasets not only confirm our theoretical findings, but also show that the best model with our GEP has better generalization error on the test data, compared to the grid search, manual search, and random search. [ABSTRACT FROM AUTHOR]

Abstract: In recent years, semi-supervised clustering (SSC) has aroused considerable interests from the machine learning and data mining communities. In this paper we propose a novel SSC approach with enhanced spectral embedding (ESE), which not only considers the geometric structure information contained in data sets, but also can make use of the given side information such as pairwise constraints. Specially, we first construct a symmetry-favored k-NN graph, which is highly robust to noise and outliers, and can reflect the underlying manifold structures of data sets. Then we learn the enhanced spectral embedding towards an ideal data representation as consistent with the given pairwise constraints as possible. Finally, by using the regularization of spectral embedding we formulate learning the new data representation as a semidefinite-quadratic-linear programming (SQLP) problem, which can be efficiently solved. Experimental results on a variety of synthetic and real-world data sets show that our ESE approach outperforms the state-of-the-art SSC algorithms in terms of speed and quality on both vector-based and graph-based clustering. [Copyright &y& Elsevier]

Abstract: In this paper, we present a structured sparse representation appearance model for tracking an object in a video system. The mechanism behind our method is to model the appearance of an object as a sparse linear combination of structured union of subspaces in a basis library, which consists of a learned Eigen template set and a partitioned occlusion template set. We address this structured sparse representation framework that preferably matches the practical visual tracking problem by taking the contiguous spatial distribution of occlusion into account. To achieve a sparse solution and reduce the computational cost, Block Orthogonal Matching Pursuit (BOMP) is adopted to solve the structured sparse representation problem. Furthermore, aiming to update the Eigen templates over time, the incremental Principal Component Analysis (PCA) based learning scheme is applied to adapt the varying appearance of the target online. Then we build a probabilistic observation model based on the approximation error between the recovered image and the observed sample. Finally, this observation model is integrated with a stochastic affine motion model to form a particle filter framework for visual tracking. Experiments on some publicly available benchmark video sequences demonstrate the advantages of the proposed algorithm over other state-of-the-art approaches. [Copyright &y& Elsevier]

Abstract: This paper presents a supervised learning algorithm for image deblurring. The task is addressed into the conceptual framework of matrix regression and gradient evolution. Specifically, given pairs of blurred image patches and their corresponding clear ones, an optimization framework of matrix regression is proposed to learn a matrix mapping. For an image to be deblurred, the learned matrix mapping will be employed to map each of its image patches directly to be a new one with more sharp details. The mapped result is then analyzed in terms of edge profiles, and the image is finally deblurred in way of gradient evolution. The algorithm is fast, and easy to be implemented. Comparative experiments on diverse natural images and the applications to interactive deblurring of real-world out-of-focus images illustrate the validity of our method. [Copyright &y& Elsevier]

Abstract: Decoding perceptual or cognitive states based on brain activity measured using functional magnetic resonance imaging (fMRI) can be achieved using machine learning algorithms to train classifiers of specific stimuli. However, the high dimensionality and intrinsically low signal to noise ratio (SNR) of fMRI data poses great challenges to such techniques. The problem is aggravated in the case of multiple subject experiments because of the high inter-subject variability in brain function. To address these difficulties, the majority of current approaches uses a single classifier. Since, in many cases, different stimuli activate different brain areas, it makes sense to use a set of classifiers each specialized in a different stimulus. Therefore, we propose in this paper using an ensemble of classifiers for decoding fMRI data. Each classifier in the ensemble has a favorite class or stimulus and uses an optimized feature set for that particular stimulus. The output for each individual stimulus is therefore obtained from the corresponding classifier and the final classification is achieved by simply selecting the best score. The method was applied to three empirical fMRI datasets from multiple subjects performing visual tasks with four classes of stimuli. Ensembles of GNB and k-NN base classifiers were tested. The ensemble of classifiers systematically outperformed a single classifier for the two most challenging datasets. In the remaining dataset, a ceiling effect was observed which probably precluded a clear distinction between the two classification approaches. Our results may be explained by the fact that different visual stimuli elicit specific patterns of brain activation and indicate that an ensemble of classifiers provides an advantageous alternative to commonly used single classifiers, particularly when decoding stimuli associated with specific brain areas. [Copyright &y& Elsevier]

Abstract: Manifold-ranking is a powerful method in semi-supervised learning, and its performance heavily depends on the quality of the constructed graph. In this paper, we propose a novel graph structure named k-regular nearest neighbor (k-RNN) graph as well as its constructing algorithm, and apply the new graph structure in the framework of manifold-ranking based retrieval. We show that the manifold-ranking algorithm based on our proposed graph structure performs better than that of the existing graph structures such as k-nearest neighbor (k-NN) graph and connected graph in image retrieval, 2D data clustering as well as 3D model retrieval. In addition, the automatic sample reweighting and graph updating algorithms are presented for the relevance feedback of our algorithm. Experiments demonstrate that the proposed algorithm outperforms the state-of-the-art algorithms. [Copyright &y& Elsevier]

Abstract: Fukunaga–Koontz Transform (FKT) is a famous feature extraction method in statistical pattern recognition, which aims to find a set of vectors that have the best representative power for one class while the poorest representative power for the other class. Li and Savvides propose a one-against-all strategy to deal with multi-class problems, in which the two-class FKT method can be directly applied to find the presentative vectors of each class. Motivated by the FKT method, in this paper we propose a new discriminant subspace analysis (DSA) method for the multi-class feature extraction problems. To solve DSA, we propose an iterative algorithm for the joint diagonalization (JD) problem. Finally, we generalize the linear DSA method to handle nonlinear feature extraction problems via the kernel trick. To demonstrate the effectiveness of the proposed method for pattern recognition problems, we conduct extensive experiments on real data sets and show that the proposed method outperforms most commonly used feature extraction methods. [Copyright &y& Elsevier]

Abstract: Ensemble approaches to classification have attracted a great deal of interest recently. This paper presents a novel method for designing the neural network ensemble using coevolutionary algorithm. The bootstrap resampling procedure is employed to obtain different training subsets that are used to estimate different component networks of the ensemble. Then the cooperative coevolutionary algorithm is developed to optimize the ensemble model via the divide-and-cooperative mechanism. All component networks are coevolved in parallel in the scheme of interacting co-adapted subpopulations. The fitness of an individual from a particular subpopulation is assessed by associating it with the representatives from other subpopulations. In order to promote the cooperation of all component networks, the proposed method considers both the accuracy and the diversity among the component networks that are evaluated using the multi-objective Pareto optimality measure. A hybrid output-combination method is designed to determine the final ensemble output. Experimental results illustrate that the proposed method is able to obtain neural network ensemble models with better classification accuracy in comparison with currently popular ensemble algorithms. [Copyright &y& Elsevier]

Abstract: Topic modeling is a powerful tool for discovering the underlying or hidden structure in text corpora. Typical algorithms for topic modeling include probabilistic latent semantic analysis (PLSA) and latent Dirichlet allocation (LDA). Despite their different inspirations, both approaches are instances of generative model, whereas the discriminative structure of the documents is ignored. In this paper, we propose locally discriminative topic model (LDTM), a novel topic modeling approach which considers both generative and discriminative structures of the data space. Different from PLSA and LDA in which the topic distribution of a document is dependent on all the other documents, LDTM takes a local perspective that the topic distribution of each document is strongly dependent on its neighbors. By modeling the local relationships of documents within each neighborhood via a local linear model, we learn topic distributions that vary smoothly along the geodesics of the data manifold, and can better capture the discriminative structure in the data. The experimental results on text clustering and web page categorization demonstrate the effectiveness of our proposed approach. [Copyright &y& Elsevier]

Abstract: This paper introduces hard clustering algorithms that are able to partition objects taking into account simultaneously their relational descriptions given by multiple dissimilarity matrices. These matrices have been generated using different sets of variables and dissimilarity functions. These methods are designed to furnish a partition and a prototype for each cluster as well as to learn a relevance weight for each dissimilarity matrix by optimizing an adequacy criterion that measures the fitting between the clusters and their representatives. These relevance weights change at each algorithm iteration and can either be the same for all clusters or different from one cluster to another. Experiments with data sets (synthetic and from UCI machine learning repository) described by real-valued variables as well as with time trajectory data sets show the usefulness of the proposed algorithms. [Copyright &y& Elsevier]

Abstract: We introduce novel discriminative semi-supervised learning algorithms for dynamical systems, and apply them to the problem of 3D human motion estimation. Our recent work on discriminative learning of dynamical systems has been proven to achieve superior performance than traditional generative learning approaches. However, one of the main issues of learning the dynamical systems is to gather labeled output sequences which are typically obtained from precise motion capture tools, hence expensive. In this paper we utilize a large amount of unlabeled (input) video data to improve the prediction performance of the dynamical systems significantly. We suggest two discriminative semi-supervised learning approaches that extend the well-known algorithms in static domains to the sequential, real-valued multivariate output domains: (i) self-training which we derive as coordinate ascent optimization of a proper discriminative objective over both model parameters and the unlabeled state sequences, (ii) minimum entropy approach which maximally reduces the model''s uncertainty in state prediction for unlabeled data points. These approaches are shown to achieve significant improvement against the traditional generative semi-supervised learning methods. We demonstrate the benefits of our approaches on the 3D human motion estimation problems. [Copyright &y& Elsevier]

Abstract: Human action recognition is a challenging task due to significant intra-class variations, occlusion, and background clutter. Most of the existing work use the action models based on statistic learning algorithms for classification. To achieve good performance on recognition, a large amount of the labeled samples are therefore required to train the sophisticated action models. However, collecting labeled samples is labor-intensive. To tackle this problem, we propose a boosted multi-class semi-supervised learning algorithm in which the co-EM algorithm is adopted to leverage the information from unlabeled data. Three key issues are addressed in this paper. Firstly, we formulate the action recognition in a multi-class semi-supervised learning problem to deal with the insufficient labeled data and high computational expense. Secondly, boosted co-EM is employed for the semi-supervised model construction. To overcome the high dimensional feature space, weighted multiple discriminant analysis (WMDA) is used to project the features into low dimensional subspaces in which the Gaussian mixture models (GMM) are trained and boosting scheme is used to integrate the subspace models. Thirdly, we present the upper bound of the training error in multi-class framework, which is able to guide the novel classifier construction. In theory, the proposed solution is proved to minimize this upper error bound. Experimental results have shown good performance on public datasets. [Copyright &y& Elsevier]

Abstract: In this paper, we introduce a novel shape/object retrieval algorithm shortest path propagation (SSP). Given a query object q and a target database object p, we explicitly find the shortest path between them in the distance manifold of the database objects. Then a new distance measure between q and p is learned based on the database objects on the shortest path to replace the original distance measure. The promising results on both MEPG-7 shape dataset and a protein dataset demonstrate that our method can significantly improve the ranking of the object retrieval. [Copyright &y& Elsevier]

Abstract: This paper addresses the problem of optimal feature extraction from a wavelet representation. Our work aims at building features by selecting wavelet coefficients resulting from signal or image decomposition on an adapted wavelet basis. For this purpose, we jointly learn in a kernelized large-margin context the wavelet shape as well as the appropriate scale and translation of the wavelets, hence the name “wavelet kernel learning”. This problem is posed as a multiple kernel learning problem, where the number of kernels can be very large. For solving such a problem, we introduce a novel multiple kernel learning algorithm based on active constraints methods. We furthermore propose some variants of this algorithm that can produce approximate solutions more efficiently. Empirical analysis show that our active constraint MKL algorithm achieves state-of-the art efficiency. When used for wavelet kernel learning, our experimental results show that the approaches we propose are competitive with respect to the state-of-the-art on brain–computer interface and Brodatz texture datasets. [Copyright &y& Elsevier]

Abstract: This paper deals with the super-resolution (SR) problem based on a single low-resolution (LR) image. Inspired by the local tangent space alignment algorithm in for nonlinear dimensionality reduction of manifolds, we propose a novel patch-learning method using locally affine patch mapping (LAPM) to solve the SR problem. This approach maps the patch manifold of low-resolution image to the patch manifold of the corresponding high-resolution (HR) image. This patch mapping is learned by a training set of pairs of LR/HR images, utilizing the affine equivalence between the local low-dimensional coordinates of the two manifolds. The latent HR image of the input (an LR image) is estimated by the HR patches which are generated by the proposed patch mapping on the LR patches of the input. We also give a simple analysis of the reconstruction errors of the algorithm LAPM. Furthermore we propose a global refinement technique to improve the estimated HR image. Numerical results are given to show the efficiency of our proposed methods by comparing these methods with other existing algorithms. [Copyright &y& Elsevier]

Abstract: The k-means algorithm and its variations are known to be fast clustering algorithms. However, they are sensitive to the choice of starting points and are inefficient for solving clustering problems in large datasets. Recently, incremental approaches have been developed to resolve difficulties with the choice of starting points. The global k-means and the modified global k-means algorithms are based on such an approach. They iteratively add one cluster center at a time. Numerical experiments show that these algorithms considerably improve the k-means algorithm. However, they require storing the whole affinity matrix or computing this matrix at each iteration. This makes both algorithms time consuming and memory demanding for clustering even moderately large datasets. In this paper, a new version of the modified global k-means algorithm is proposed. We introduce an auxiliary cluster function to generate a set of starting points lying in different parts of the dataset. We exploit information gathered in previous iterations of the incremental algorithm to eliminate the need of computing or storing the whole affinity matrix and thereby to reduce computational effort and memory usage. Results of numerical experiments on six standard datasets demonstrate that the new algorithm is more efficient than the global and the modified global k-means algorithms. [Copyright &y& Elsevier]

Abstract: Feature selection is an important preprocessing step for building efficient, generalizable and interpretable classifiers on high dimensional data sets. Given the assumption on the sufficient labelled samples, the Markov Blanket provides a complete and sound solution to the selection of optimal features, by exploring the conditional independence relationships among the features. In real-world applications, unfortunately, it is usually easy to get unlabelled samples, but expensive to obtain the corresponding accurate labels on the samples. This leads to the potential waste of valuable classification information buried in unlabelled samples. In this paper, we propose a new BAyesian Semi-SUpervised Method, or BASSUM in short, to exploit the values of unlabelled samples on classification feature selection problem. Generally speaking, the inclusion of unlabelled samples helps the feature selection algorithm on (1) pinpointing more specific conditional independence tests involving fewer variable features and (2) improving the robustness of individual conditional independence tests with additional statistical information. Our experimental results show that BASSUM enhances the efficiency of traditional feature selection methods and overcomes the difficulties on redundant features in existing semi-supervised solutions. [Copyright &y& Elsevier]

Abstract: In instance-based machine learning, algorithms often suffer from storing large numbers of training instances. This results in large computer memory usage, long response time, and often oversensitivity to noise. In order to overcome such problems, various instance reduction algorithms have been developed to remove noisy and surplus instances. This paper discusses existing algorithms in the field of instance selection and abstraction, and introduces a new approach, the Class Boundary Preserving Algorithm (CBP), which is a multi-stage method for pruning the training set, based on a simple but very effective heuristic for instance removal. CBP is tested with a large number of datasets and comparatively evaluated against eight of the most successful instance-based condensation algorithms. Experiments showed that our algorithm achieved similar classification accuracies, with much improved storage reduction and competitive execution speeds. [ABSTRACT FROM AUTHOR]

Abstract: We tackle the structured output classification problem using the Conditional Random Fields (CRFs). Unlike the standard 0/1 loss case, we consider a cost-sensitive learning setting where we are given a non-0/1 misclassification cost matrix at the individual output level. Although the task of cost-sensitive classification has many interesting practical applications that retain domain-specific scales in the output space (e.g., hierarchical or ordinal scale), most CRF learning algorithms are unable to effectively deal with the cost-sensitive scenarios as they merely assume a nominal scale (hence 0/1 loss) in the output space. In this paper, we incorporate the cost-sensitive loss into the large margin learning framework. By large margin learning, the proposed algorithm inherits most benefits from the SVM-like margin-based classifiers, such as the provable generalization error bounds. Moreover, the soft-max approximation employed in our approach yields a convex optimization similar to the standard CRF learning with only slight modification in the potential functions. We also provide the theoretical cost-sensitive generalization error bound. We demonstrate the improved prediction performance of the proposed method over the existing approaches in a diverse set of sequence/image structured prediction problems that often arise in pattern recognition and computer vision domains. [Copyright &y& Elsevier]

Abstract: Support vector machine (SVM) is a widely used classification technique. However, it is difficult to use SVMs to deal with very large data sets efficiently. Although decomposed SVMs (DSVMs) and core vector machines (CVMs) have been proposed to overcome this difficulty, they cannot be applied to online classification (or classification with learning ability) because, when new coming samples are misclassified, the classifier has to be adjusted based on the new coming misclassified samples and all the training samples. The purpose of this paper is to address this issue by proposing an online CVM classifier with adaptive minimum-enclosing-ball (MEB) adjustment, called online CVMs (OCVMs). The OCVM algorithm has two features: (1) many training samples are permanently deleted during the training process, which would not influence the final trained classifier; (2) with a limited number of selected samples obtained in the training step, the adjustment of the classifier can be made online based on new coming misclassified samples. Experiments on both synthetic and real-world data have shown the validity and effectiveness of the OCVM algorithm. [Copyright &y& Elsevier]

Abstract: In this paper, we propose a general learning framework based on local and global regularization. In the local regularization part, our algorithm constructs a regularized classifier for each data point using its neighborhood, while the global regularization part adopts a Laplacian regularizer to smooth the data labels predicted by those local classifiers. We show that such a learning framework can easily be incorporated into either unsupervised learning, semi-supervised learning, and supervised learning paradigm. Moreover, many existing learning algorithms can be derived from our framework. Finally we present some experimental results to show the effectiveness of our method. [Copyright &y& Elsevier]