Search

Knowledge distillation (KD) builds a lightweight Student Model (SM) and trains it to approximate a large Teacher Model (TM) by exploring knowledge learned by the TM, which shows effectiveness to train lightweight CNN models. However, training a small SM to achieve better performance remains a challenging problem. Recent researches on human learning behaviors show that both the knowledge from teachers and the knowledge learning processes of teachers are significant for students. Inspired by this characteristic, in this paper, we propose a new perspective, called Progressive Mimic Learning (PML), to train lightweight CNN models by mimicking the learning trajectory of the TM. In order to obtain a more powerful SM, the useful hints in the learning process of the TM are explored. To start with, the TM learning process is divided into multiple stages, and the last state of the TM in each stage is recorded as a landmark. The learning trajectory of the TM is composed of these landmarks. Then, a landmark loss is defined to constrain the SM to progressively mimic the learning process of the TM, by employing landmarks in the learning trajectory as a training hint of the SM. Several experiments are conducted on four benchmark data sets, CIFAR-10, CIFAR-100, Fashion-MNIST, and ImageNet-10, to investigate the performance of the PML. The results show that the PML can make SMs generate more accurate predictions than SMs trained by its counterparts.

Recently, least squares twin multi-class support vector machine (LSTKSVC) was proposed as a least squares version of twin multi-class classification support vector machine (Twin-KSVC), both based on twin support vector machine (TWSVM). In this paper, we propose a novel multi-class classifier termed as Improvements on least squares twin multi-class classification support vector machine that is motivated by LSTKSVC and Twin-KSVC. Similarly to LSTKSVC that evaluates all the training data into a ` ` 1 − v e r s u s − 1 − v e r s u s − r e s t ″ structure, the algorithm here proposed generates ternary output { − 1 , 0 , + 1 } . Whereas Twin-KSVC needs to solve two quadratic programming problems (QPPs), the solution of the two modified primal problems for our algorithm is reduced to two systems of linear equations. Besides that, in our algorithm the structural risk minimization (SRM) principle is implemented by introducing a regularization term, along with minimizing the empirical risk. To test the efficacy and validity of the proposed method, numerical experiments on ten UCI benchmark data sets are performed. The results obtained further corroborate the effectiveness of the proposed algorithm.

Extreme learning machine (ELM) as an emergent technology has shown its good performance in classification applications. However, ELM algorithm needs to find the inversion of matrix in nature, which will limit its application on many occasions. This paper proposes an ELM speedup algorithm based on the analysis of ELM algorithm. By applying randomized approximation method, the proposed algorithm can approximate the key matrix (For example, the kernel matrix in the kernel-based ELM) with a low-rank matrix. By doing so, the complexity of the inversion can be reduced from O ( n 3 ) to O ( kn 2 + k 3 ) (n is the size of the data set, and k is the numerical rank of the approximated matrix). On the premise of not decreasing the accuracy too much, the training time can be cut down substantially, which has important significance in practical application of machine learning algorithms. The experimental results on benchmark data sets demonstrate the effectiveness of the proposed algorithm.

Recently, two parsimonious algorithms were proposed to sparsify extreme learning machine (ELM), i.e., constructive parsimonious ELM (CP-ELM) and destructive parsimonious ELM (DP-ELM). In this paper, the ideas behind CP-ELM and DP-ELM are extended to the regularized ELM (RELM), thus obtaining CP-RELM and DP-RELM. For CP-RELM(DP-RELM), there are two schemes to realize it, viz. CP-RELM-I and CP-RELM-II(DP-RELM-I and DP-RELM-II). Generally speaking, CP-RELM-II(DP-RELM-II) outperforms CP-RELM-I(DP-RELM-I) in terms of parsimoniousness. Under nearly the same generalization, compared with CP-ELM(DP-ELM), CP-RELM-II(DP-RELM-II) usually needs fewer hidden nodes. In addition, different from CP-ELM and DP-ELM, for CP-RELM and DP-RELM the number of candidate hidden nodes may be larger than the number of training samples, which assists the selection of much better hidden nodes for constructing more compact networks. Finally, eleven benchmark data sets divided into two groups are utilized to do experiments and the usefulness of the proposed algorithms is reported.

Active set methods are competitive alternatives of SVM optimizers to working set or decomposition techniques for moderate scale problems. While there are many works on applying the primal active set methods to SVM, few study the dual active set methods. Comparing with the primal active set method, the dual active set method is more efficient and numerically stable due to using an unconstrained minimum as a feasible start point and having choices of which constraints to add to the active set. However, since the conventional dual active set method can only add one violated constraint in one step, this paper proposes a novel dual active set method, which allows to add multiple constraints and accelerates the convergence. Moreover, the new method does not require to compute the matrix inversion when applied to SVM, and essentially reduces the training time. Experiment results on several benchmark data sets validate that the proposed method is indeed more efficient than the primal and the conventional dual active set methods.

In this paper, a novel feature extraction method based on regularized correntropy criterion (FEND-RCC) is proposed for novelty detection. In FEND-RCC, the presented criterion aims to maximize the difference between the correntropy of the normal data with their mean and the correntropy of the novel data with the mean of the normal data. Moreover, the optimal projection vectors in the objective function of FEND-RCC are iteratively obtained by the half-quadratic optimization technique. Experimental results on two synthetic data sets and thirteen benchmark data sets for novelty detection demonstrate that FEND-RCC is superior to its related approaches.

When performing the classification on the high dimensional, the sparse, or the noisy data, many approaches easily lead to the dramatic performance degradation. To deal with this issue from the different perspective, this paper proposes a cognitive gravitation model (CGM) based on both the law of gravitation in physics and the cognitive laws, where the self-information of each sample instead of mass is applied. Subsequently, a new classifier is designed which utilizes CGM to find k nearest neighbors from each class for the query sample and then classifies this query sample to the class whose cognitive gravitation is largest. The cognitive gravitation of the class is defined as the sum of the cognitive gravitation between its each nearest neighbor and the query sample. The advantage of our approach is that it has a firm and simple mathematical basis while it has good classification performance. The conducted experiments on challenging benchmark data sets validate the proposed model and the classification approach.

Manifold learning is a hot research topic in the field of computer science. A crucial issue with current manifold learning methods is that they lack a natural quantitative measure to assess the quality of learned embeddings, which greatly limits their applications to real-world problems. In this paper, a new embedding quality assessment method for manifold learning, named as Normalization Independent Embedding Quality Assessment (NIEQA), is proposed. Compared with current assessment methods which are limited to isometric embeddings, the NIEQA method has a much larger application range due to two features. First, it is based on a new measure which can effectively evaluate how well local neighborhood geometry is preserved under normalization, hence it can be applied to both isometric and normalized embeddings. Second, it can provide both local and global evaluations to output an overall assessment. Therefore, NIEQA can serve as a natural tool in model selection and evaluation tasks for manifold learning. Experimental results on benchmark data sets validate the effectiveness of the proposed method., Comment: 16 pages, 8 figures, 3 tables

In this paper, a fast method of selecting features for kernel minimum squared error (KMSE) is proposed to mitigate the computational burden in the case where the size of the training patterns is large. Compared with other existent algorithms of selecting features for KMSE, this iterative KMSE, viz. IKMSE, shows better property of enhancing the computational efficiency without sacrificing the generalization performance. Experimental reports on the benchmark data sets, nonlinear autoregressive model and real problem address the efficacy and feasibility of the proposed IKMSE. In addition, IKMSE can be easily extended to classification fields.

A new quality assessment criterion for evaluating the performance of the nonlinear dimensionality reduction (NLDR) methods is proposed in this paper. Differing from the current quality assessment criteria focusing on the local-neighborhood-preserving performance of the NLDR methods, the proposed criterion capitalizes on a new aspect, the global-structure-holding performance, of the NLDR methods. By taking both properties into consideration, the intrinsic capability of the NLDR methods can be more faithfully reflected, and hence more rational measurement for the proper selection of NLDR methods in real-life applications can be offered. The theoretical argument is supported by experiment results implemented on a series of benchmark data sets.

A novel optimum extreme learning machines (ELM) construction method was proposed. We define an extended covering matrix with smooth function, relax the objective and constraints to formulate a more general linear programming method for the minimum sphere set covering problem. We call this method linear programming minimum sphere set covering (LPMSSC). We also present a corresponding kernelized LPMSSC and extended LPMSSC with non-Euclidean L1 and L-infinity metric. We then propose to apply the LPMSSC method to ELM and propose a data dependent ELM (DDELM) algorithm. We can obtain compact ELM for pattern classification via LPMSSC. We investigate the performances of the proposed method through UCI benchmark data sets.

Piecewise linear networks (PLNs) are attractive because they can be trained quickly and provide good performance in many nonlinear approximation problems. Most existing design algorithms for piecewise linear networks are not convergent, non-optimal, or are not designed to handle noisy data. In this paper, four algorithms are presented which attack this problem. They are: (1) a convergent design algorithm which builds the PLN one module at a time using a branch and bound technique; (2) two pruning algorithms which eliminate less useful modules from the network; and (3) a sifting algorithm which picks the best networks out of the many designed. The performance of the PLN is compared with that of the multilayer perceptron (MLP) using several benchmark data sets. Numerical results demonstrate that piecewise linear networks are adequate for many approximation problems.