Search

The projection twin support vector machine (PTSVM) is a potential tool for classification problem. However the loss function of PTSVM is hinge loss function which is a unbounded loss and not robust enough to outliers. In this work, a robust PTSVM (termed RSHPTSVM) is proposed based on rescaled square hinge loss (RSH-loss) to handle classification problem. A close relationship between RSH-loss and correntropy is established theoretically. The RSH-loss can be viewed as a correntropy-induced loss by a reproducing piecewise kernel. As such a correntropy loss, it has vastly different properties from hinge loss such as boundedness, robustness and nonconvexity. Moreover, RSH-loss is with higher order statistical information from samples. However the nonconvexity of RSHPTSVM makes it difficult to optimize, so that an efficient iterative optimization algorithm based on semi-quadratic optimization theory is proposed to solve RSHPTSVM, which can quickly converge to the optimal solution. Furthermore, we extend our RSHPTSVM from binary classification to multi-classification and propose a robust projection multi-birth support vector machine model (termed RSHPMBSVM). The proposed method is implemented on various datasets including three artificial datasets, UCI datasets, and a practical application dataset. The experiment results under no noise and label noise circumstance confirm the feasibility and effectiveness of the proposed methods. [ABSTRACT FROM AUTHOR]

The content of free calcium oxide (f-CaO) in cement clinker is an important index for cement quality. Aiming at the characteristics of strong coupling, time-varying delay and highly non-linearity in cement clinker production, a soft sensor model based on multivariate time series analysis and convolutional neural network (MVTS–CNN) is proposed for the online f-CaO content monitoring. Based on the process industry characteristics, the MVTS–CNN modeling involves the detailed analysis of coupling relationship and time-varying delay in cement production and the application of neural network in multivariate time-series feature extraction. The main researches and contributions are fourfold: First, the strong coupling in the production system is further analyzed, and the proposed model is focused on the data coupling between specific processes, not the control coupling. Second, a multivariate time series analysis method is designed to select the time series that may have direct impacts on the f-CaO content in different production conditions, which is founded on the information on time delay range and longest active duration. Third, a multivariate time series feature extraction method is designed and adopted in the MVTS–CNN model to extract the multivariate time series features, such as active duration difference features, coupling features, nonlinear features and key time series features. Fourth, a new timing matching method, which is combined the rough timing matching of multivariate time series and the detailed timing matching of key features, is proposed to deal with the time-varying delay in various production conditions. Compared with traditional CNN, support vector machines (SVM) and long-short term memory networks (LSTM), the results demonstrate that the MVTS–CNN model has higher accuracy, better generalization ability and superior robustness. • A multivariate segment-to-point model is proposed for f-CaO content estimation. • The data coupling is the dominant factor in coupling, not the control coupling. • The time series analysis is on the basis of time delay range and active duration. • The active duration difference is considered in the feature extraction. • A novel staged timing matching method is proposed for time-varying delay. [ABSTRACT FROM AUTHOR]

Least squares twin support vector machine (LSTSVM) is an effective and efficient learning algorithm for pattern classification. However, the distance in LSTSVM is measured by squared L 2 -norm metric that may magnify the influence of outliers. In this paper, a novel robust least squares twin support vector machine framework is proposed for binary classification, termed as C L 2 , p -LSTSVM, which utilizes capped L 2 , p -norm distance metric to reduce the influence of noise and outliers. The goal of C L 2 , p -LSTSVM is to minimize the capped L 2 , p -norm intra-class distance dispersion, and eliminate the influence of outliers during training process, where the value of the metric is controlled by the capped parameter, which can ensure better robustness. The proposed metric includes and extends the traditional metrics by setting appropriate values of p and capped parameter. This strategy not only retains the advantages of LSTSVM, but also improves the robustness in solving a binary classification problem with outliers. However, the nonconvexity of metric makes it difficult to optimize. We design an effective iterative algorithm to solve the C L 2 , p -LSTSVM. In each iteration, two systems of linear equations are solved. Simultaneously, we present some insightful analyses on the computational complexity and convergence of algorithm. Moreover, we extend the C L 2 , p -LSTSVM to nonlinear classifier and semi-supervised classification. Experiments are conducted on artificial datasets, UCI benchmark datasets, and image datasets to evaluate our method. Under different noise settings and different evaluation criteria, the experiment results show that the C L 2 , p -LSTSVM has better robustness than state-of-the-art approaches in most cases, which demonstrates the feasibility and effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

The insufficiency and contamination of supervision information are the main factors affecting the performance of support vector machines (SVMs) in real-world applications. To address this issue, novel correntropy loss functions and Laplacian SVM (LapSVM) are utilized for robust semi-supervised classification. It is known that correntropy loss functions have been used in robust learning and achieved promising results. However, the potential for more diverse priors has not been extensively explored. In this paper, a correntropy loss function induced from Laplace kernel function, called LK-loss, is applied to LapSVM for the construction of robust semi-supervised classifier. Properties of LK-loss are demonstrated including robustness, symmetry, boundedness, Fisher consistency and asymptotic approximation behaviors. Moreover, the asymmetric version of LK-loss is introduced to further improve the performance. Concave-convex procedure (CCCP) technique is used to handle the non-convexity of Laplace kernel-induced correntropy loss functions iteratively. Experimental results show that in most cases, the proposed methods have better generalization performance than the comparing ones, which demonstrate the feasibility and effectiveness of the proposed semi-supervised classification framework. [ABSTRACT FROM AUTHOR]

We propose a new distribution-free Bayes optimal classifier, called the twin minimax probability machine (TWMPM), which combines the benefits of both minimax probability machine(MPM) and twin support vector machine (TWSVM). TWMPM tries to construct two nonparallel hyperplanes such that each hyperplane separates one class samples with maximal probability, and is distant from the other class samples simultaneously. Moreover, the proposed TWMPM can control the misclassification error of samples in a worst-case setting by minimizing the upper bound on misclassification probability. An efficient algorithm for TWMPM is first proposed, which transforms TWMPM into concave fractional programming by applying multivariate Chebyshev inequality. Then the proposed TWMPM is reformulated as a pair of convex quadric programming (QP) by proper mathematical transformations. This guarantees TWMPM to have global optimal solution and be solved simply and effectively. In addition, we develop also an iterative algorithm for the proposed TWMPM. By comparing the two proposed algorithms theoretically, it is easy to know that the convex quadric programming algorithm is with lower computation burden than iterative algorithm for the TWMPM. A linear TWMPM version is first built, and then we show how to exploit mercer kernel to obtain nonlinear TWMPM version. The computation complexity for QP algorithm of TWMPM is in the same order as the traditional twin support vector machine (TWSVM). Experiments are carried out on three databases: UCI benchmark database, a practical application database and an artificial database. With low computation complexity and fewer parameters, experiments show the feasibility and effectiveness of the proposed TWMPM and its QP algorithm. • Propose a twin minimax probability machine (TWMPM) classification framework. • TWMPM has advantages of both MPM and TWSVM. • With maximal probability, TWMPM separates two-class samples. • Reformulate TWMPM as concave maximization problems with global optimal solutions. • TWMPM is with low computation complexity and few parameters. [ABSTRACT FROM AUTHOR]

• A robust twin bounded SVM (RTBSVM) is proposed. • The least squares version (FRTBSVM) of RTBSVM is proposed. • Two effective algorithms are derived. • The convergence of algorithms are proved and computational complexity is analyzed. • Experiments show that RTBSVM and FRTBSVM are feasibility and effectiveness. In this paper, to improve the performance of capped L 1 -norm twin support vector machine (CTSVM), we first propose a new robust twin bounded support vector machine (RTBSVM) by introducing the regularization term. The significant advantage of our RTBSVM over CTSVM is that the structural risk minimization principle is implemented. This embodies the marrow of statistical learning theory, so this modification can improve the performance of classification. Furthermore, to accelerate the computation of RTBSVM and simultaneously inherit the merit of robustness, we construct a least squares version of RTBSVM (called RTBSVM). This formulation leads to a simple and fast algorithm for binary classifiers by solving just two systems of linear equations. Finally, we derive two simple and effective iterative optimization algorithms for solving RTBSVM and FRTBSVM, respectively. Simultaneously, we theoretically rigorously analyze and prove the computational complexity, local optimality and convergence of the algorithms. Experimental results on one synthetic dataset and nine UCI datasets demonstrate that our methods are competitive with other methods. Additionally, the FRTBSVM is directly applied to recognize the purity of hybrid maize seeds using near-infrared spectral data. Experiments show that our method achieves better performance than the traditional methods in most spectral regions. [ABSTRACT FROM AUTHOR]

As an important multi-classification learning tool, multi-birth support vector machine (MBSVM) has been widely studied and applied due to its low computational complexity and good generalization. In this paper, a new multi-birth support vector machine is proposed to handle multi-class classification problem, called projection multi-birth support vector machine (PMBSVM). Specifically, we intend to seek a projection direction wk for k-th class, so that the covariance of remaining samples (except the k-th class) is as small as possible, and the samples of k-th class are as far as possible from the mean of the remaining samples. The proposed PMBSVM not only inherits the advantages of MBSVM, but also can find a suitable projection direction for each class so that the sample is separable in the projection space. Additionally, a regularization term is introduced to maximize the margin of different classes in the projected space. Moreover, a recursive PMBSVM algorithm is proposed for generating multiple orthogonal projection directions for each class. Then we extend the proposed approaches to nonlinear situations through kernel technology. Simulation results on benchmark datasets show that the proposed algorithms improve the generalization in most cases. [ABSTRACT FROM AUTHOR]

In this work we propose a minimax probability extreme learning machine framework (MPME), which combines the benefits of minimax probability machine (MPM) with extreme learning machine (ELM). For binary classification problems, we illustrate that the proposed MPME can be interpreted geometrically by minimizing the maximum of Mahalanobis distances to the two classes. Then two variants of the MPME are presented based on the l 2 -norm loss and l 1 -norm loss functions (called LSEMPME and LADMPME) respectively. Without making specific assumption on the data distribution, the proposed methods can provide explicit upper-bounds for the generalization error, moreover the LSEMPME and LADMPME minimize empirical risk simultaneously. The decision hyperplanes of the proposed methods pass through the origin in ELM feature space with few decision variables. By using the multivariate Chebyshev–Cantelli inequality, all the proposed problems can be reformulated as second-order cone programming (SOCP) with global solutions. Furthermore, numerical experiments have been carried out on two databases that are drawn from UCI benchmark database and a practical application database. First, the proposed methods are evaluated for a practical application consisting on the analysis of licorice seeds using near-infrared spectral (NIR) data. Experiments in six different spectral regions illustrate that the proposed methods can improve generalization in most cases. Then the proposed methods are evaluated on benchmark datasets. In comparison with traditional methods including MPM, ELM and support vector machine (SVM), experiments show that the proposed methods achieve comparable results in generalization. With few decision variables, the proposed methods are easy to implement for nonlinear classification and to estimate a lower-bound on the prediction accuracy. • We propose a new minimax probability extreme learning machine (MPME). • We extend MPME based on 1-norm and 2-norm loss functions respectively. • All problems are posed as SOCP problems with global solutions. • MPME can provide an explicit lower bound on correct classification probability. • MPME has good interpretation geometrically in ELM feature space. [ABSTRACT FROM AUTHOR]

Abstract We propose a new homotopy loss, where practitioners can tune the parameter to derive different loss functions such as l 1 - n o r m loss, logarithmic loss, Geman–Reynolds loss, Geman–McClure loss and correntropy-based loss. Moreover, we illustrate that the proposed loss satisfies Fisher consistency, and we analyze the robustness of the proposed homotopy loss from different perspectives: M-estimation and adversarial perturbations. Then, we represent a new evaluation standard to measure robustness and demonstrate its upper bound to ensure the validity of this measure. Applied the proposed homotopy loss to least square support vector machine (LSSVM) and the extreme learning machine (ELM), two robust models are presented to enhance the robustness. Furthermore, re-weighted least square algorithm is used to solve the problems, and the resulting algorithms converge globally. In addition, the proposed methods are implemented on various datasets with different levels of noise. Compared with traditional methods, experiment results on real-world datasets show that the proposed methods have superior anti-interference ability to outliers in most cases. [ABSTRACT FROM AUTHOR]

Highlights • A new robust loss function is designed for imbalanced data sets. • The proposed model combines this loss with SVM. • The robustness of model is analyzed in theory. • The Bayes optimal solution is derived. • Alternative iterative algorithm is designed to reduce algorithm complexity. Abstract Based on minimizing misclassification cost, a new robust loss function is designed in this paper to deal with the imbalanced classification problem under noise environment. It is nonconvex but maintains Fisher consistency. Applying the proposed loss function into support vector machine (SVM), a robust SVM framework is presented which results in a Bayes optimal classifier. However, nonconvexity makes the model difficult to optimize. We develop an alternative iterative algorithm to solve the proposed model. What's more, we analyze the robustness of the proposed model theoretically from a re-weighted SVM viewpoint and the obtained optimal solution is consistent with Bayesian optimal decision rule. Furthermore, numerical experiments are carried out on databases that are drawn from UCI Machine Learning Repository and a practical application. With two different types of noise environments, one with label noise and one with feature noise, experiment results show that on these two databases the proposed method achieves better generalization results compared to other SVM methods. [ABSTRACT FROM AUTHOR]

Correntropy is a local similarity measure between two arbitrary variables, and it has been applied in a variety of learning algorithms to improve noise insensitivity. In this paper, based on the correntropy, a non-convex and bounded loss function is obtained which contains second and higher order moments of the classification margin. And the novel loss function is robust to noises and close to the 0–1 loss function. Then we introduce it into extreme learning machine (ELM), and propose a correntropy-based robust ELM framework for classification, trained by half quadratic optimization to cope with non-convexity of the algorithm. To evaluate robustness, feature noise and label noise are simulated to provide noisy environments. Experimental results on benchmark datasets demonstrate that the proposed algorithm is better than original algorithms and robust algorithms. Moreover, the superiority of proposed algorithm in noisy environment is more evident, which further proves its robustness to noises. [ABSTRACT FROM AUTHOR]

Support vector machine(SVM) with pinball loss(PINSVM) has been recently proposed and shown its advantages in pattern recognition. In this paper, we present a robust bounded loss function (called L t -loss) that truncates pinball loss function. Then a novel robust SVM formulation with L t -loss(called TPINSVM) is proposed to enhance noise robustness. Moreover, we demonstrate that the proposed TPINSVM satisfies Bayes rule and it has a certain sparseness. However, the non-convexity of the proposed TPINSVM makes it difficult to optimize. We develop a continuous optimization method, DC(difference of convex functions) programming method, to solve the proposed TPINSVM. The resulting DC optimization algorithm converges finitely. Furthermore, the proposed TPINSVM is directly applied to recognize the purity of hybrid maize seeds using near-infrared spectral data. Experiments show that the proposed method achieves better performance than the traditional methods in most spectral regions. Meanwhile we simulate the proposed TPINSVM in benchmark datasets in different situations. In noiseless setting, the proposed TPINSVM either improves or shows no significant difference in generalization compared to the traditional approaches. While in noise situations, TPINSVM improves generalization in most cases. [ABSTRACT FROM AUTHOR]

Semi-supervised learning has been an attractive research tool for using unlabeled data in pattern recognition. Applying a novel semi-definite programming (SDP) relaxation strategy to a class of continuous semi-supervised support vector machines (S3VMs), a new convex relaxation framework for the S3VMs is proposed based on SDP. Compared with other SDP relaxations for S3VMs, the proposed methods only require solving the primal problems and can implement L1-norm regularization. Furthermore, the proposed technique is applied directly to recognize the purity of hybrid maize seeds using near-infrared spectral data, from which we find that the proposed method achieves equivalent performance to the exact solution algorithm for solving the S3VM in different spectral regions. Experiments on several benchmark data sets demonstrate that the proposed convex technique is competitive with other SDP relaxation methods for solving semi-supervised SVMs in generalization. [ABSTRACT FROM AUTHOR]

Abstract: Owing to its wide applicability, semi-supervised learning is an attractive method for using unlabeled data in classification. Applying a new smoothing strategy to a class of continuous semi-supervised support vector machines (S3VMs), this paper proposes a class of smooth S3VMs (S4VMs) without adding new variables and constraints to the corresponding S3VMs. Moreover, a general framework for solving the S4VMs is constructed based on robust DC (difference of convex functions) programming. Furthermore, DC optimization algorithms (DCAs) for solving the S4VMs are investigated. The resulting DCAs converge and only require solving one linear or quadratic program at each iteration. Numerical experiments on some real-world databases demonstrate that the proposed smooth S3VMs are feasible and effective, and have comparable results as other S3VMs. [Copyright &y& Elsevier]

Abstract: The recognition of the hardness of licorice seeds is a challenging task. The purpose of this investigation is to identify the hardness of licorice seeds employing a semi-supervised learning method and near-infrared spectroscopy. An excellent semi-supervised learning model, the semi-supervised support vector machine (S3VM), is built using the small labeled samples and the large unlabeled samples. Moreover, the proposed model is solved by employing an effective method, the robust DC (difference of convex functions) programming. The resulting algorithm only requires the solving of a few linear programs. Furthermore, this model is used for the direct classification of licorice samples. Comparing with the supervised support vector machine (SVM), experimental results on different spectral regions show that incorporating unlabeled samples in training improves the generalization when insufficient training information is available. Moreover, our method outperforms the existing S3VM method by obtaining better performance in different spectral regions. These results show that it is possible to identify the hardness of licorice seeds using the proposed S3VM and near-infrared spectroscopic data. We hope that the results obtained in this study will help further investigations of the hardness of crop seeds. [Copyright &y& Elsevier]

In general, introducing robust distance metrics and loss functions in the learning process can improve the robustness of the algorithms. In this work, we first propose a new robust loss function called adaptive capped L θ ε -loss. For different problems, we can choose different loss functions through adaptive parameter θ during the learning process. Secondly, we propose a new robust distance metric induced by correntropy (CIM) that is based on Laplacian kernel. The CIM contains first and higher-order moments from samples. Further, we demonstrate some important and interesting properties of the L θ ε -loss and CIM, such as robustness, boundedness, nonconvexity, etc. Finally, we apply the to L θ ε -loss and CIM to twin support vector machine (TWSVM) and develop an adaptive robust learning framework, namely adaptive robust twin support vector machine (ARTSVM). The proposed ARTSVM not only inherits the advantages of TWSVM but also improves the robustness of classification problems. A non-convex optimization method, DC (difference of convex functions) programming algorithm (DCA) is used to solve the proposed ARTSVM, and the convergence of the algorithm is proved theoretically. Experiments on multiple datasets show that the proposed ARTSVM is competitive with existing methods. • A correntropy-based generalized robust distance metric is proposed, called correntropy induced metric (CIM). • A robust loss function is proposed, namely, adaptive capped L θ ε -loss. • Some important and interesting properties of the L θ ε -loss and CIM are demonstrated. • A adaptive robust twin support vector machine (ARTSVM) is proposed based on CIM and L θ ε -loss. • DC programming technique is used to solve the ARTSVM. • Numerical experiments under different noises show that the proposed ARTSVM is effective and more robust to outliers. [ABSTRACT FROM AUTHOR]

A new robust loss function (called L q -loss) is proposed based on the concept of quantile and correntropy, which can be seen as an improved version of quantile loss function. The proposed L q -loss has some important properties such as asymmetry, non-convexity and boundedness, which has received a lot of attention recently. The L q -loss includes and extends the traditional loss functions such as pinball loss, rescaled hinge loss, L 1 -norm loss and zero-norm loss. Additionally, we demonstrate that the L q -loss is a kernel-induced loss by reproducing piecewise kernel function. Further, two robust SVM frameworks are presented to handle robust classification and regression problems by applying L q -loss to support vector machine, respectively. Last but not least, we demonstrate that the proposed classification framework satisfies Bayes' optimal decision rule. However, the non-convexity of the proposed L q -loss makes it difficult to optimize. A non-convex optimization method, concave–convex procedure (CCCP) technique, is used to solve the proposed models, and the convergence of the algorithms is proved theoretically. For classification and regression tasks, experiments are carried out on three databases including UCI benchmark datasets, artificial datasets and a practical application dataset. Compared to some classical and advanced methods, numerical simulations under different noise setting and different evaluation criteria show that the proposed methods have good robustness to feature noise and outliers in both classification and regression applications. • Propose a generalized quantile loss (L q -loss) to handle robust learning. • Demonstrate important properties: asymmetry, non-convexity, approximability and boundedness. • Two robust models are proposed with L q -loss to enhance robustness. • The proposed classification framework satisfies Bayes' optimal decision rule. • concave–convex procedure (CCCP) is used to handle nonconvexity. [ABSTRACT FROM AUTHOR]