Search

An electroencephalogram (EEG)–based brain–computer interface (BCI) speller is a system that conveys thought to enable communication between humans and computers using brain conduction. This system is useful for patients with severe disabilities and provides a way for them to communicate with computers or with other people. The EEG-based BCI speller system is being studied in various ways, but it is vulnerable to adversarial examples. An adversarial example is an attack sample created by adding a little noise to an original sample in such a way that it appears normal to humans but will be misclassified by the model. Adversarial examples can be useful in situations such as military scenarios in which there is a mixture of friendly models and enemy models. When a friendly BCI speller and an enemy BCI speller coexist, an adversarial example may be mistakenly taken by the enemy BCI speller to signal an incorrect intention on the part of the person with a disability; in another scenario, an enemy BCI speller may leak personal information of the individual with a disability or initiate unwanted financial transactions. In this paper, we propose a method to create such a sample, called a "friend-guard" EEG adversarial example. In the proposed method, a very small EEG signal is added to an original sample, creating an adversarial example that the friendly model will classify correctly but the enemy model will classify incorrectly. A P300 dataset was used in the experimental evaluation of the method, and linear regression models were used as target models. In the experiment, the proposed method was able to generate friend-guard EEG adversarial examples that were incorrectly classified with success rates of 88.4% and 69.7% by the enemy model for subject A and subject B, respectively, while maintaining the accuracy of the friendly model at 85.9% and 74.4% for subjects A and B, respectively. [ABSTRACT FROM AUTHOR]

The expensive optimization problem, widespread in real-world applications, poses great challenges to traditional evolutionary algorithms (EAs) due to its costly evaluations. A lot of investigations have proved that it is a practical way to incorporate surrogate models into EAs to reduce the number of time-consuming evaluations. Although various types of surrogate models from the community of machine learning, such as regression models that approximate the fitness values of candidate solutions or classification models that predict candidate solutions' category, have been widely applied separately, none of them consistently outperforms the other. Considering the intrinsic consistency of alleviating computational burden to improve the optimization efficiency, this paper introduces a two-layer architecture to integrate regression and classification models and synergize global and local evolution for high-dimensional expensive many-objective optimization problem (EMaOP). To implement this idea, the Kriging model based on incremental learning is utilized to explore the potential promising space. Further, multi local support vector machine (SVM) classifiers assisted improved differential evolution is run by turn with the global search to accelerate convergence. Accordingly, a novel surrogate-assisted EA, named Classification- and Regression-assisted evolutionary algorithm (CR-SAEA), is proposed on this basis. Experimental studies on several benchmark functions with up to 11 objectives and 200 dimensions confirm the effectiveness of the proposed algorithm and its competitiveness against state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Uncertainty estimation is a critical component of building safe and reliable machine learning models. Accurate estimation of uncertainties is essential for identifying and mitigating potential risks and ensuring that machine learning systems operate reliably in real-world scenarios. Various approaches, such as ensemble and Bayesian neural networks have been developed by sampling probability predictions from submodels, which is computationally expensive. At present, these techniques are incapable of precisely delineating the boundary separating in-distribution (ID) and out-of-distribution (OOD) data. To fill up this research gap, this paper presents a normalizing flow based framework to directly predict parameters of prior distributions over the probability with a neural network, the proposed model is able to effectively differentiate between ID and OOD data in regression problems. The posterior distributions learned by the model precisely represent uncertainties for OOD data based solely on ID data, without the need for OOD data during training. This approach has shown promising results in a number of applications, including image depth estimation and image adversarial attacks. [ABSTRACT FROM AUTHOR]

Online multi-object tracking is a process of extending multi-object trajectories with only past information. In this process, tracking drift, missing detection, and occlusion among objects are common problems. To address the problem of tracking drift, in this paper, a detector–tracker integration framework is proposed and the framework includes the linear regression model (LRM) that can detect and track objects simultaneously. The two kinds of results are combined using LRM to suppress tracking drift. Moreover, to overcoming missing detection and the occlusion, a structural similarity calculation based on the attention mechanism is proposed. The attention mechanism is utilized to learn the discriminative features of each object, and a structural similarity calculation are used to improve the ability to extend the objects' trajectories. Based on these, we design a multi-object tracking strategy, enabling the trajectory's initialization, extension, and termination. Finally, experiments and analysis are executed on MOT16 and MOT17 benchmarks datasets, and the proposed method obtains multi-object tracking accuracy of 60%. [ABSTRACT FROM AUTHOR]

Extreme Learning Machine (ELM) is a learning algorithm based on generalized single-hidden-layer feed-forward neural network. Since ELM has an excellent performance on regression and classification problems, it has been paid more and more attention recently. The determination of structure of ELM plays a vital role in ELM applications. Essentially, determination of the structure of ELM is equivalent to the determination of the hidden layer structure. Utilizing a smaller scale of the hidden layer structure can promote faster running speed. In this paper, we propose algorithm PCI-ELM (Pruned-Convex Incremental Extreme Learning Machine) based on CI-ELM (Convex Incremental Extreme Learning Machine). Furthermore, we also present an improved PCI-ELM algorithm, EPCI-ELM (Enhanced Pruned-Convex Incremental Extreme Learning Machine), which introduces a filtering strategy for PCI-ELM during the neurons adding process. In order to adjust the single-hidden-layer feed-forward neural network more flexibly and achieve the most compact form of the hidden layer structure, in this paper, we propose an algorithm which can dynamically determine hidden layer structure, DCI-ELM (Dynamic Convex Incremental Extreme Learning Machine). At the end of this paper, we verify the performance of PCI-ELM, EPCI-ELM and DCI-ELM. The results show that PCI-ELM, EPCI-ELM and DCI-ELM control hidden layer structure very well and construct the more compact single-hidden-layer feed-forward neural network. [ABSTRACT FROM AUTHOR]

To compact the architecture of extreme learning machine (ELM), two incremental learning algorithms are proposed in this paper. The previous incremental learning algorithms for ELM recruit hidden nodes randomly, which is equivalent to implementing a random selection from a candidate set of infinite size. Hence, it is impossible to recruit good hidden nodes, and thus it usually requires more hidden nodes than traditional neural networks to achieve matched performance. To improve the quality of the hidden nodes recruited, an incremental learning algorithm for ELM is presented based on Gram--Schmidt process (GSI-ELM), which recruits the best hidden node from a random subset of fixed size via defining an evaluating criterion at each learning step. However, the “nesting effect” exists in the GSI-ELM, that is to say, the hidden nodes once recruited by GSI-ELM can not be later discarded. To treat this “nesting problem”, the improved GSI-ELM (IGSI-ELM) is generated with an elimination mechanism. At each learning step IGSI-ELM eliminates the worst hidden node from the already-recruited group if it is not the newly-recruited one. Finally, to verify the efficacy and feasibility of the proposed algorithms, i.e. GSI-ELM and IGSI-ELM, in this paper, experiments on regression and classification benchmark data sets are investigated. [ABSTRACT FROM AUTHOR]

Machine learning methods employing positive kernels have been developed and widely used for classification, regression, prediction and unsupervised learning applications, whereby the estimate function takes the form of a weighted-sum kernel expansion. Unacceptable computational burden with large datasets and difficulty in tuning hyperparameters are usually the drawbacks of kernel methods. In order to reduce the computational burden, this paper presents a modified version of the Feature Vector Selection (FVS) method, proposing an approximation of the estimate function as a weighted sum of the predicted values of the Feature Vectors (FVs), where the weights are computed as the oblique projections of the new data points on the FVs in the feature space. Such approximation is, then, obtained by optimizing only the predicted values of the FVs. By defining a least square error optimization problem with equal constraints, analytic solutions of the predicted values of the FVs can be obtained. The proposed method is named Feature Vector Regression (FVR). The tuning of hyperparameters in FVR is also explained in the paper and shown to be less complicated than for other kernel methods. Comparisons with some other popular kernel methods for regression on several public datasets show that FVR, with a small subset of the training dataset (i.e. selected FVs), gives results comparable with those of the methods which give best results in terms of the prediction accuracy. The main contribution of this paper is the new kernel method (i.e. FVR), capable of achieving satisfactory results with reduced efforts because of the small number of hyperparameters to be tuned and the reduced training dataset size used. [ABSTRACT FROM AUTHOR]

Abstract Transverse Flux Permanent Magnet Motor (TFPMM) has received extensive attention in the field of electric vehicles. The magnetic circuit of TFPMM is three-dimensional and non-linear, which leads to the high nonlinearity of electromagnetic torque. Although the three-dimensional finite element method (3DFEM) could be used to estimate torque, it is very time-consuming. Instead of it, this study adopts a kind of machine learning method—Gaussian process regression (GPR). For the hyper-parameters optimization of GPR, most previous studies used single-objective algorithms for improving the regression accuracy. However, GPR is an algorithm of probability prediction and it could not guarantee to have the satisfactory confidence interval characteristic simultaneously while the regression precision achieves optimal. Therefore this paper proposes a variable parameters fuzzy dominance genetic algorithm (VPFDGA) which is suitable for the multi-objective optimization, including the optimization of regression precision, confidence interval reliability, confidence interval width and skill score. By combining GPR with VPFDGA, the electromagnetic torque of a building-block transverse flux permanent magnet motor (B-TFPMM) is estimated by VPFDGA-GPR (GPR based on variable parameters fuzzy dominance genetic algorithm). Besides, two other GPRs based multi-objective optimization, three GPRs based on single-objective optimization and a GPR based on weighted sum method that is the classic multi-objective optimization algorithm are all implemented to compare with VPFDGA-GPR. The results of comparison show that VPFDGA-GPR has the better performances including the higher regression precision, more powerful ability of probability prediction, higher stability, less convergence time and so on. [ABSTRACT FROM AUTHOR]

Abstract For regression tasks, the existing extreme learning machine (ELM) and kernel extreme learning machine (KELM) algorithms exhibit singularity and over-fitting problems when the number of training samples is less than the number of hidden layer neurons. To overcome these shortcomings, this paper introduces random reduction kernel and regularization parameters, and the regularization incremental extreme learning machine with random reduced kernel (RKRIELM) algorithm is proposed. RKRIELM combines the kernel function and incremental extreme learning machine (I-ELM) to avoid randomness, thereby solving the singularity problem when the number of initial training samples of the ELM is less than the number of hidden layer neurons. Moreover, it uses the number of hidden layer neurons as the precondition for the loop ending of the training algorithm. Additionally, the regularization parameter is used to reduce the risk of over-fitting. Regression experiments were conducted for evaluating the proposed method, ELM, KELM, reduced kernel extreme learning machine, rotation forest selective ensemble extreme learning machine, reduced support vector regression, and gray wolf optimization support vector regression with standard data sets. The results indicate that the proposed method has a lower prediction error and better training efficiency than the other algorithms in most cases. [ABSTRACT FROM AUTHOR]

Highlights • We propose a multiobjective optimization-based sparse extreme learning machine (MO-SELM) for classification and regression tasks. • The sparse connecting structure of ELM is designed for learning a more compact network. • We improve MOEA/D to optimize the proposed multiobjective model and make decisions by ensemble learning. • Experimental results reveal the superior performance of the proposed MO-SELM. Abstract Extreme Learning Machine (ELM) is a popular machine learning method and has been widely applied to real-world problems due to its fast training speed and good generalization performance. However, in ELM, the randomly assigned input weights and hidden biases usually degrade the generalization performance. Furthermore, ELM is considered as an empirical risk minimization model and easily leads to overfitting when dataset exists some outliers. In this paper, we proposed a novel algorithm named Multiobjective Optimization-based Sparse Extreme Learning Machine (MO-SELM), where parameter optimization and structure learning are integrated into the learning process to simultaneously enhance the generalization performance and alleviate the overfitting problem. In MO-SELM, the training error and the connecting sparsity are taken as two conflicting objectives of the multiobjective model, which aims to find sparse connecting structures with optimal weights and biases. Then, a hybrid encoding-based MOEA/D is used to optimize the multiobjective model. In addition, ensemble learning is embedded into this algorithm to make decisions after multiobjective optimization. Experimental results of several classification and regression applications demonstrate the effectiveness of the proposed MO-SELM. [ABSTRACT FROM AUTHOR]

Abstract This work identifies an important, previously unaddressed issue for regression based on neural networks – learning to accurately approximate problems where the output is not a function of the input (i.e. where the number of outputs required varies across input space). Such non-functional regression problems arise in a number of applications, and can not be adequately handled by existing neural network algorithms. To demonstrate the benefits possible from directly addressing non-functional regression, this paper proposes the first neural algorithm to do so – an extension of the Resource Allocating Network (RAN) which adds additional output neurons to the network structure during training. This new algorithm, called the Resource Allocating Network with Varying Output Cardinality (RANVOC), is demonstrated to be capable of learning to perform non-functional regression, on both artificially constructed data and also on the real-world task of specifying parameter settings for a plasma-spray process. Importantly RANVOC is shown to outperform not just the original RAN algorithm, but also the best possible error rates achievable by any functional form of regression. [ABSTRACT FROM AUTHOR]

Ordinal regression (OR) is an important research topic in machine learning and has attracted extensive attention due to its wide applications. So far, a variety of methods have been proposed to perform OR, in which the class-center-induced threshold methods (like KDLOR and MOR) have received more attention, for their simplicity and promising performance. The class-center-induced ORs typically calculate the ordinal thresholds with class centers, which are typically derived from the l 2 -norm. Unfortunately, in such a way, the class means may be biased when the data is corrupted with outliers (i.e., non-i.i.d. noises) such that the resulting OR accuracy will be deteriorated. Motivated by the success of l p -norm in applications against noises, in this paper we propose a novel type of class centroid derived from the l p -norm (coined as l p -centroid) to overcome the drawbacks above, and provide an optimization algorithm and corresponding convergence analysis for computing the l p -centroid. To evaluate the effectiveness of l p -centroid in OR context against noises, we then combine the l p -centroid with two representative class-center-induced ORs, namely discriminant learning based and manifold learning based ORs. Finally, extensive OR experiments on synthetic and real-world datasets demonstrate the effectiveness and superiority of the proposed methods to related existing methods. [ABSTRACT FROM AUTHOR]

The privacy-preserving data analysis has been gained significant interest across several research communities. The current researches mainly focus on privacy-preserving classification and regression. On the other hand, feature selection is also one of the key problems in data mining and machine learning. However, for privacy-preserving feature selection, the relevant papers are few. In this paper, a local learning-based feature weighting framework is introduced. Moreover, in order to preserve the data privacy during local learning-based feature selection, the objective perturbation and output perturbation strategies are used to produce local learning-based feature selection algorithms with privacy preservation. Meanwhile, we give deep analysis about their privacy preserving property based on the differential privacy model. Some experiments are conducted on benchmark data sets. The experimental results show that our algorithms can preserve the data privacy to some extent and the objective perturbation always obtains higher classification performance than output perturbation when the privacy preserving degree is constant. [ABSTRACT FROM AUTHOR]

Semi-supervised dimensionality reduction is an important research topic in many pattern recognition and machine learning applications. Among all the methods for semi-supervised dimensionality reduction, SDA and LapRLS/L are two popular ones. Though the two methods are actually the extensions of different supervised methods, we show in this paper that both SDA and Lap-RLS/L can be unified under a regularized least square framework. In this paper, we propose a new effective semi-supervised dimensionality reduction method for better cope with data sampled from nonlinear manifold. In addition, the proposed method can both handle the regression as well as the subspace learning problem. Theoretical analysis and extensive simulations show the effectiveness of our algorithm. The results in simulations demonstrate that our proposed algorithm can achieve great superiority compared with other existing methods. [ABSTRACT FROM AUTHOR]

In this paper a polynomial fuzzy regression model with fuzzy independent variables and fuzzy parameters is discussed. Within this paper the fuzzy neural network model is used to obtain an estimate for the fuzzy parameters in a statistical sense. Based on the extension principle, a simple algorithm from the cost function of the fuzzy neural network is proposed, in order to find the approximate parameters. Finally, we illustrate our approach by some numerical examples. [ABSTRACT FROM AUTHOR]

Gene Expression Programming (GEP) is a powerful evolutionary method for knowledge discovery and model learning. Based on the basic GEP algorithm, this paper proposes an improved algorithm named S_GEP, which is especially suitable for dealing with symbolic regression problems. The major advantages for this S_GEP method include: (1) A new method for evaluating individual without expression tree; (2) a corresponding expression tree construction schema for the new evaluating individual method if required by some special complex problems; and (3) a new approach for manipulating numeric constants so as to improve the convergence. A thorough comparative study between our proposed S_GEP method with the primitive GEP, as well as other methods are included in this paper. The comparative results show that the proposed S_GEP method can significantly improve the GEP performance. Several well-studied benchmark test cases and real-world test cases demonstrate the efficiency and capability of our proposed S_GEP for symbolic regression problems. [ABSTRACT FROM AUTHOR]

Abstract: This paper investigates the construction of linear-in-the-parameters (LITP) models for multi-output regression problems. Most existing stepwise forward algorithms choose the regressor terms one by one, each time maximizing the model error reduction ratio. The drawback is that such procedures cannot guarantee a sparse model, especially under highly noisy learning conditions. The main objective of this paper is to improve the sparsity and generalization capability of a model for multi-output regression problems, while reducing the computational complexity. This is achieved by proposing a novel multi-output two-stage locally regularized model construction (MTLRMC) method using the extreme learning machine (ELM). In this new algorithm, the nonlinear parameters in each term, such as the width of the Gaussian function and the power of a polynomial term, are firstly determined by the ELM. An initial multi-output LITP model is then generated according to the termination criteria in the first stage. The significance of each selected regressor is checked and the insignificant ones are replaced at the second stage. The proposed method can produce an optimized compact model by using the regularized parameters. Further, to reduce the computational complexity, a proper regression context is used to allow fast implementation of the proposed method. Simulation results confirm the effectiveness of the proposed technique. [Copyright &y& Elsevier]

For robust visual tracking, appearance modeling should be able to well separate the object from its backgrounds, while accurately adapt to its appearance variations. However, most of the existing tracking methods mainly focus on one of the two aspects; or design two different modules to combine them with the price of double computational cost. In this paper, by using pairwise metric learning, we present a novel appearance model for robust visual tracking. Specifically, visual tracking is viewed as a pairwise regression problem, and extreme learning machine (ELM) is utilized to construct the pairwise regression framework. In ELM-based pairwise training, two constraints are enforced: the target observations must have different regression outputs from those background ones; while the various target observations during tracking should have approximate regression outputs. Thus, the discriminative and generative capabilities are fully considered in a single object tracking model. Moreover, online sequential ELM (OS-ELM) is used to update the resulting appearance model, thereby leading to a more robust tracking process. Extensive experimental evaluations on challenging video sequences demonstrate the effectiveness and efficiency of the proposed tracker. [ABSTRACT FROM AUTHOR]

Predicting the properties of materials like concrete has been proven a difficult task given the complex interactions among its components. Over the years, researchers have used Statistics, Machine Learning, and Evolutionary Computation to build models in an attempt to accurately predict such properties. High-quality models are often non-linear, justifying the study of nonlinear regression tools. In this paper, we employ a traditional multiple linear regression method by ordinary least squares to solve the task. However, the model is built upon nonlinear features automatically engineered by Kaizen Programming, a recently proposed hybrid method. Experimental results show that Kaizen Programming can find low-correlated features in an acceptable computational time. Such features build high-quality models with better predictive quality than results reported in the literature. [ABSTRACT FROM AUTHOR]

Abstract: The output weights computing of extreme learning machine (ELM) encounters two problems, the computational and outlier robustness problems. The computational problem occurs when the hidden layer output matrix is a not full column rank matrix or an ill-conditioned matrix because of randomly generated input weights and biases. An existing solution to this problem is Singular Value Decomposition (SVD) method. However, the training speed is still affected by the large complexity of SVD when computing the Moore–Penrose (MP) pseudo inverse. The outlier robustness problem may occur when the training data set contaminated with outliers then the accuracy rate of ELM is extremely affected. This paper proposes the Extended Complete Orthogonal Decomposition (ECOD) method to solve the computational problem in ELM weights computing via ECODLS algorithm. And the paper also proposes the other three algorithms, i.e. the iteratively reweighted least squares (IRWLS-ELM), ELM based on the multivariate least-trimmed squares (MLTS-ELM), and ELM based on the one-step reweighted MLTS (RMLTS-ELM) to solve the outlier robustness problem. However, they also encounter the computational problem. Therefore, the ECOD via ECODLS algorithm is also used successfully in the three proposed algorithms. The experiments of regression problems were conducted on both toy and real-world data sets. The outlier types are one-sided and two-sided outliers. Each experiment was randomly contaminated with outliers, of one type only, with 10%, 20%, 30%, 40%, and 50% of the total training data size. Meta-metrics evaluation was used to measure the outlier robustness of the proposed algorithms compared to the existing algorithms, i.e. the minimax probability machine regression (MPMR) and the ordinary ELM. The experimental results showed that ECOD can effectively replace SVD. The ECOD is robust to the not full column rank or the ill-conditional problem. The speed of the ELM training using ECOD is also faster than the ordinary training algorithm. Moreover, the meta-metrics measure showed that the proposed algorithms are less affected by the increasing number of outliers than the existing algorithms. [Copyright &y& Elsevier]

Abstract: This paper investigates the design of a linear-in-the-parameters (LITP) regression classifier for two-class problems. Most existing algorithms generally learn a classifier (model) from the available training data based on some stopping criterions, such as the Akaike''s final prediction error (FPE). The drawback here is that the classifier obtained is then not directly obtained based on its generalization capability. The main objective of this paper is to improve the sparsity and generalization capability of a classifier, while reducing the computational expense in producing it. This is achieved by proposing an automatic two-stage locally regularized classifier construction (TSLRCC) method using the extreme learning machine (ELM). In this new algorithm, the nonlinear parameters in each term, such as the width of the Gaussian function and the power of a polynomial term, are firstly determined by the ELM. An initial classifier is then generated by the direct evaluation of these candidates models according to the leave-one-out (LOO) misclassification rate in the first stage. The significance of each selected regressor term is also checked and insignificant ones are replaced in the second stage. To reduce the computational complexity, a proper regression context is defined which allows fast implementation of the proposed method. Simulation results confirm the effectiveness of the proposed technique. [Copyright &y& Elsevier]

Extreme learning machine (ELM) is a machine learning technique based on competitive single-hidden layer feedforward neural network (SLFN). However, traditional ELM and its variants are only based on random assignment of hidden weights using a uniform distribution, and then the calculation of the weights output using the least-squares method. This paper proposes a new architecture based on a non-linear layer in parallel by another non-linear layer and with entries of independent weights. We explore the use of a deterministic assignment of the hidden weight values using low-discrepancy sequences (LDSs). The simulations are performed with Halton and Sobol sequences. The results for regression and classification problems confirm the advantages of using the proposed method called PL-ELM algorithm with the deterministic assignment of hidden weights. Moreover, the PL-ELM algorithm with the deterministic generation using LDSs can be extended to other modified ELM algorithms. [ABSTRACT FROM AUTHOR]

In this paper, we propose and solve a new machine learning problem called the extensive semi-quantitative regression, where the information about some target values is incomplete; we only know their lower bounds and/or upper bounds instead of their exact values. To employ the information efficiently in extensive semi-quantitative regression, we introduce a local graph to capture the geometric structure for the samples with the exact target values and the target bounds, and construct a graph-based support vector regressor, called ESQ-SVR. The efficiency of our ESQ-SVR is supported by the results of preliminary experiments conducted on both the artificial and real world datasets. [ABSTRACT FROM AUTHOR]

This paper presents a distance metric learning method for k -nearest neighbors regression. We define the constraints based on triplets, which are built from the neighborhood of each training instance, to learn the distance metric. The resulting optimization problem can be formulated as a convex quadratic program. Quadratic programming has a disadvantage that it does not scale well in large-scale settings. To reduce the time complexity of training, we propose a novel dual coordinate descent method for this type of problem. Experimental results on several regression data sets show that our method obtains a competitive performance when compared with the state-of-the-art distance metric learning methods, while being an order of magnitude faster. [ABSTRACT FROM AUTHOR]

Abstract: The well-known sequential minimal optimization (SMO) algorithm is the most commonly used algorithm for numerical solutions of the support vector learning problems. At each iteration in the traditional SMO algorithm, also called 2PSMO algorithm in this paper, it jointly optimizes only two chosen parameters. The two parameters are selected either heuristically or randomly, whilst the optimization with respect to the two chosen parameters is performed analytically. The 2PSMO algorithm is naturally generalized to the three-parameter sequential minimal optimization (3PSMO) algorithm in this paper. At each iteration of this new algorithm, it jointly optimizes three chosen parameters. As in 2PSMO algorithm, the three parameters are selected either heuristically or randomly, whilst the optimization with respect to the three chosen parameters is performed analytically. Consequently, the main difference between these two algorithms is that the optimization is performed at each iteration of the 2PSMO algorithm on a line segment, whilst that of the 3PSMO algorithm on a two-dimensional region consisting of infinitely many line segments. This implies that the maximum can be attained more efficiently by 3PSMO algorithm. Main updating formulae of both algorithms for each support vector learning problem are presented. To assess the efficiency of the 3PSMO algorithm compared with the 2PSMO algorithm, 14 benchmark datasets, 7 for classification and 7 for regression, will be tested and numerical performances are compared. Simulation results demonstrate that the 3PSMO outperforms the 2PSMO algorithm significantly in both executing time and computation complexity. [Copyright &y& Elsevier]

Abstract: The paper presents an approach for performing regression on large data sets in reasonable time, using an ensemble of extreme learning machines (ELMs). The main purpose and contribution of this paper are to explore how the evaluation of this ensemble of ELMs can be accelerated in three distinct ways: (1) training and model structure selection of the individual ELMs are accelerated by performing these steps on the graphics processing unit (GPU), instead of the processor (CPU); (2) the training of ELM is performed in such a way that computed results can be reused in the model structure selection, making training plus model structure selection more efficient; (3) the modularity of the ensemble model is exploited and the process of model training and model structure selection is parallelized across multiple GPU and CPU cores, such that multiple models can be built at the same time. The experiments show that competitive performance is obtained on the regression tasks, and that the GPU-accelerated and parallelized ELM ensemble achieves attractive speedups over using a single CPU. Furthermore, the proposed approach is not limited to a specific type of ELM and can be employed for a large variety of ELMs. [Copyright &y& Elsevier]

Abstract: Extreme learning machine (ELM) is one of the most popular and important learning algorithms. It comes from single-hidden-layer feedforward neural networks. It has been proved that ELM can achieve better performance than support vector machine (SVM) in regression and classification. In this paper, mathematically, with regression problem, the step 3 of ELM is studied. First of all, the equation are reformulated as an optimal model. With the optimality, the necessary conditions of optimal solution are presented. The equation is replaced by . We can prove that the latter must have one solution at least. Second, optimal approximation solution is discussed in cases of is column full rank, row full rank, neither column nor row full rank. In the last case, the rank-1 and rank-2 methods are used to get optimal approximation solution. In theory, this paper present a better algorithm for ELM. [Copyright &y& Elsevier]

Abstract: Extreme learning machine (ELM) as an emergent technology has shown its good performance in regression applications as well as in large dataset (and/or multi-label) classification applications. The ELM theory shows that the hidden nodes of the “generalized” single-hidden layer feedforward networks (SLFNs), which need not be neuron alike, can be randomly generated and the universal approximation capability of such SLFNs can be guaranteed. This paper further studies ELM for classification in the aspect of the standard optimization method and extends ELM to a specific type of “generalized” SLFNs—support vector network. This paper shows that: (1) under the ELM learning framework, SVM''s maximal margin property and the minimal norm of weights theory of feedforward neural networks are actually consistent; (2) from the standard optimization method point of view ELM for classification and SVM are equivalent but ELM has less optimization constraints due to its special separability feature; (3) as analyzed in theory and further verified by the simulation results, ELM for classification tends to achieve better generalization performance than traditional SVM. ELM for classification is less sensitive to user specified parameters and can be implemented easily. [Copyright &y& Elsevier]

Abstract: Extreme learning machine (ELM) proposed by Huang et al. was developed for generalized single hidden layer feedforward networks (SLFNs) with a wide variety of hidden nodes. It proved to be very fast and effective especially for solving function approximation problems with a predetermined network structure. However, the method for determining the network structure of preliminary ELM may be tedious and may not lead to a parsimonious solution. In this paper, a systematic two-stage algorithm (named TS-ELM) is introduced to handle the problem. In the first stage, a forward recursive algorithm is applied to select the hidden nodes from the candidates randomly generated in each step and add them to the network until the stopping criterion achieves its minimum. The significance of each hidden node is then reviewed in the second stage and the insignificance ones are removed from the network, which drastically reduces the network complexity. The effectiveness of TS-ELM is verified by the empirical studies in this paper. [ABSTRACT FROM AUTHOR]

The online sequential extreme learning machine (OS-ELM) algorithm is an on-line and incremental learning method, which can learn data one-by-one or chunk-by-chunk with a fixed or varying chunk size. And OS-ELM achieves the same learning performance as ELM trained by the complete set of data. However, in on-line learning environments, the concepts to be learned may change with time, a feature referred to as concept drift. To use ELMs in such non-stationary environments, a forgetting parameters extreme learning machine (FP-ELM) is proposed in this paper. The proposed FP-ELM can achieve incremental and on-line learning, just like OS-ELM. Furthermore, FP-ELM will assign a forgetting parameter to the previous training data according to the current performance to adapt to possible changes after a new chunk comes. The regularized optimization method is used to avoid estimator windup. Performance comparisons between FP-ELM and two frequently used ensemble approaches are carried out on several regression and classification problems with concept drift. The experimental results show that FP-ELM produces comparable or better performance with lower training time. [ABSTRACT FROM AUTHOR]

Machine Learning has two central processes of interest that captivate the scientific community: classification and regression. Although instance selection for classification has shown its usefulness and has been researched in depth, instance selection for regression has not followed the same path and there are few published algorithms on the subject. In this paper, we propose that various adaptations of DROP, a well-known family of instance selection methods for classification, be applied to regression. Their behaviour is analysed using a broad range of datasets. The results are presented of the analysis of four new proposals for the reduction of dataset size, the effect on error when several classifiers are trained with the reduced dataset, and their robustness against noise. This last aspect is especially important, since in real life, it is frequent that the registered data be inexact and present distortions due to different causes: errors in the measurement tools, typos when writing results, existence of outliers and spurious readings, corruption in files, etc. When the datasets are small it is possible to manually correct these problems, but for big and huge datasets is better to have automatic methods to deal with these problems. In the experimental part, the proposed methods are found to be quite robust to noise. [ABSTRACT FROM AUTHOR]

The fact that the linear estimators using the rank-based Wilcoxon approach in linear regression problems are usually insensitive to outliers is known in statistics. Outliers are the data points that differ greatly from the pattern set by the bulk of the data. Inspired by this fact, Hsieh et al. introduced the Wilcoxon approach into the area of machine learning. They investigated four new learning machines, such as Wilcoxon neural network (WNN), and developed four gradient descent based backpropagation algorithms to train these learning machines. The performances of these machines are better than ordinary nonrobust neural networks in outliers exist tasks. However, it is hard to balance the learning speed and the stability of these algorithms which is inherently the drawback of gradient descent based algorithms. In this paper, a new algorithm is used to train the output weights of single-layer feedforward neural networks (SLFN) with input weights and biases being randomly chosen. This algorithm is called Wilcoxon-norm based robust extreme learning machine or WRELM for short. [ABSTRACT FROM AUTHOR]

LASSO is known to have a problem of excessive shrinkage at a sparse representation. To analyze this problem in detail, in this paper, we consider a positive scaling for soft-thresholding estimators that are LASSO estimators in an orthogonal regression problem. We especially consider a non-parametric orthogonal regression problem which includes wavelet denosing. We first gave a risk (generalization error) of LARS (least angle regression) based soft-thresholding with a single scaling parameter. We then showed that an optimal scaling value that minimizes the risk under a sparseness condition is 1 + O ( log n / n ) , where n is the number of samples. The important point is that the optimal value of scaling is larger than one. This implies that expanding soft-thresholding estimator shows a better generalization performance compared to a naive soft-thresholding. This also implies that a risk of LARS-based soft-thresholding with the optimal scaling is smaller than without scaling. We then showed their difference is O ( log n / n ) . This also shows an effectiveness of the introduction of scaling. Through simple numerical experiments, we found that LARS-based soft-thresholding with scaling can improve both of sparsity and generalization performance compared to a naive soft-thresholding. [ABSTRACT FROM AUTHOR]

As a single-hidden-layer feedforward neural network, an extreme learning machine (ELM) randomizes the weights between the input layer and the hidden layer as well as the bias of hidden neurons, and analytically determines the weights between the hidden layer and the output layer using the least-squares method. This paper proposes a two-hidden-layer ELM (denoted TELM) by introducing a novel method for obtaining the parameters of the second hidden layer (connection weights between the first and second hidden layer and the bias of the second hidden layer), hence bringing the actual hidden layer output closer to the expected hidden layer output in the two-hidden-layer feedforward network. Simultaneously, the TELM method inherits the randomness of the ELM technique for the first hidden layer (connection weights between the input weights and the first hidden layer and the bias of the first hidden layer). Experiments on several regression problems and some popular classification datasets demonstrate that the proposed TELM can consistently outperform the original ELM, as well as some existing multilayer ELM variants, in terms of average accuracy and the number of hidden neurons. [ABSTRACT FROM AUTHOR]

Extreme Learning Machine (ELM), which was initially proposed for training single-layer feed-forward networks (SLFNs), provides us a unified efficient and effective framework for regression and multiclass classification. Though various ELM variants were proposed in recent years, most of them focused on the supervised learning scenario while little effort was made to extend it into unsupervised learning paradigm. Therefore, it is of great significance to put ELM into learning tasks with only unlabeled data. One popular approach for mining knowledge from unlabeled data is based on the manifold assumption, which exploits the geometrical structure of data by assuming that nearby points will also be close to each other in transformation space. However, considering the manifold information only is insufficient for discriminative tasks. In this paper, we propose an improved unsupervised discriminative ELM (UDELM) model, whose main advantage is to combine the local manifold learning with global discriminative learning together. UDELM can be efficiently optimized by solving a generalized eigen-value decomposition problem. Extensive comparisons over several state-of-the-art models on clustering image and emotional EEG data demonstrate the efficacy of UDELM. [ABSTRACT FROM AUTHOR]

In the paper, we examine the general regression problem under the missing data scenario. In order to provide reliable estimates for the regression function (approximation), a novel methodology based on Gaussian Mixture Model and Extreme Learning Machine is developed. Gaussian Mixture Model is used to model the data distribution which is adapted to handle missing values, while Extreme Learning Machine enables to devise a multiple imputation strategy for final estimation. With multiple imputation and ensemble approach over many Extreme Learning Machines, final estimation is improved over the mean imputation performed only once to complete the data. The proposed methodology has longer running times compared to simple methods, but the overall increase in accuracy justifies this trade-off. [ABSTRACT FROM AUTHOR]

Extreme learning machine (ELM) is a very simple machine learning algorithm and it can achieve a good generalization performance with extremely fast speed. Therefore it has practical significance for data analysis in real-world applications. However, it is implemented normally under the empirical risk minimization scheme and it may tend to generate a large-scale and over-fitting model. In this paper, an ELM model based on L 1 -norm and L 2 -norm regularizations is proposed to handle regression and multiple-class classification problems in a unified framework. The proposed model called L 1 – L 2 -ELM combines the grouping effect benefits of L 2 penalty and the tendency towards sparse solution of L 1 penalty, thus it can control the complexity of the network and prevent over-fitting. To solve the mixed penalty problem, the separate elastic net algorithm and Bayesian information criterion (BIC) are adopted to find the optimal model for each response variable. We test the L 1 – L 2 -ELM algorithm on one artificial case and nine benchmark data sets to evaluate its performance. Simulation results have shown that the proposed algorithms outperform the original ELM as well as other advanced ELM algorithms in terms of prediction accuracy, and it is more robust in both regression and classification applications. [ABSTRACT FROM AUTHOR]

The random-hidden-node extreme learning machine (ELM) is a much more generalized cluster of single-hidden-layer feed-forward neural networks (SLFNs) which has three parts: random projection, non-linear transformation, and ridge regression (RR) model. Networks with deep architectures have demonstrated state-of-the-art performance in a variety of settings, especially with computer vision tasks. Deep learning algorithms such as stacked autoencoder (SAE) and deep belief network (DBN) are built on learning several levels of representation of the input. Beyond simply learning features by stacking autoencoders (AE), there is a need for increasing its robustness to noise and reinforcing the sparsity of weights to make it easier to discover interesting and prominent features. The sparse AE and denoising AE was hence developed for this purpose. This paper proposes an approach: SSDAE-RR (stacked sparse denoising autoencoder – ridge regression) that effectively integrates the advantages in SAE, sparse AE, denoising AE, and the RR implementation in ELM algorithm. We conducted experimental study on real-world classification (binary and multiclass) and regression problems with different scales among several relevant approaches: SSDAE-RR, ELM, DBN, neural network (NN), and SAE. The performance analysis shows that the SSDAE-RR tends to achieve a better generalization ability on relatively large datasets (large sample size and high dimension) that were not pre-processed for feature abstraction. For 16 out of 18 tested datasets, the performance of SSDAE-RR is more stable than other tested approaches. We also note that the sparsity regularization and denoising mechanism seem to be mandatory for constructing interpretable feature representations. The fact that a SSDAE-RR approach often has a comparable training time to ELM makes it useful in some real applications. [ABSTRACT FROM AUTHOR]

Extreme learning machine (ELM) uses a non-iterative method to train single-hidden-layer feed-forward networks (SLFNs), which has been proven to be an efficient and effective learning model for both classification and regression. The main advantage of ELM lies in that the input weights as well as the hidden layer biases can be randomly generated, which contributes to the analytical solution of output weights. In this paper, we propose a discriminative manifold ELM (DMELM) by simultaneously considering the discriminative information and geometric structure of data; specifically, we exploit the discriminative information in the local neighborhood around each data point. To this end, a graph regularizer based on a newly designed graph Laplacian to characterize both properties is formulated and incorporated into the ELM objective. In DMELM, the output weights can also be obtained in analytical form. Extensive experiments are conducted on image and EEG signal classification to evaluate the effectiveness of DMELM. The results show that DMELM consistently achieves better performance than original ELM and yields promising results in comparison with several state-of-the-art algorithms, which suggests that both the discriminative as well as manifold information are beneficial to classification. [ABSTRACT FROM AUTHOR]

Extreme Learning Machine (ELM), a competitive machine learning technique for single-hidden-layer feedforward neural networks (SLFNNs), has proven to be efficient and effective algorithm for regression and classification problems. However, traditional ELM involves a large number of hidden nodes for complex real world regression and classification problems which increasing the computation burden. In this paper, a decomposition based fast ELM (DFELM) algorithm is proposed to effectively reduce the computational burden for large number of hidden nodes condition. Compared with ELM algorithm, DFELM algorithm has faster training time with a large number of hidden nodes maintaining the same accuracy performance. Experiment on three regression problems, six classification problems and a complex blast furnace modeling problem are carried out to verify the performance of DFELM algorithm. Moreover, the decomposition method can be extended to other modified ELM algorithms to further reduce the training time. [ABSTRACT FROM AUTHOR]

Electric load forecasting is an important issue for power utility, associated with the management of daily operations such as energy transfer scheduling, unit commitment, and load dispatch. Inspired by strong non-linear learning capability of support vector regression (SVR), this paper presents a SVR model hybridized with the differential empirical mode decomposition (DEMD) method and auto regression (AR) for electric load forecasting. The differential EMD method is used to decompose the electric load into several detail parts associated with high frequencies (intrinsic mode function (IMF)) and an approximate part associated with low frequencies. The electric load data from the New South Wales (NSW, Australia) market and the New York Independent System Operator (NYISO, USA) are employed for comparing the forecasting performances of different alternative models. The results illustrate the validity of the idea that the proposed model can simultaneously provide forecasting with good accuracy and interpretability. [ABSTRACT FROM AUTHOR]

In this paper, we propose a visual saliency detection algorithm which incorporates both generative and discriminative saliency models into a unified framework. First, we develop a generative model by defining image saliency as the sparse coding residual based on a learned background dictionary. Second, we introduce a discriminative model by solving an optimization problem that exploits the intrinsic relevance of similar regions for regressing region-based saliency to the smooth state. Third, a weighted sum of multi-scale region-level saliency is computed as the pixel-level saliency, which generates a more continuous and smooth result. Furthermore, object location is also utilized to suppress background noise, which acts as a vital prior for saliency detection. Experimental results show that the proposed algorithm generates more accurate saliency maps and performs favorably against the state-of-the-art saliency detection methods on three publicly available datasets. [ABSTRACT FROM AUTHOR]

In recent years, there has been a great interest in prostate segmentation, which is an important and challenging task for CT image guided radiotherapy. In this paper, a learning-based segmentation method via joint transductive feature selection and transductive regression is presented, which incorporates the physician׳s simple manual specification (only taking a few seconds), to aid accurate segmentation, especially for the case with large irregular prostate motion. More specifically, for the current treatment image, experienced physician is first allowed to manually assign the labels for a small subset of prostate and non-prostate voxels, especially in the first and last slices of the prostate regions. Then, the proposed method follows the two step: in prostate-likelihood estimation step, two novel algorithms, tLasso and wLapRLS, will be sequentially employed for transductive feature selection and transductive regression, respectively, aiming to generate the prostate-likelihood map. In multi-atlases based label fusion step, the final segmentation result will be obtained according to the corresponding prostate-likelihood map and the previous images of the same patient. The proposed method has been substantially evaluated on a real prostate CT dataset including 24 patients with 330 CT images, and compared with several state-of-the-art methods. Experimental results show that the proposed method outperforms the state-of-the-arts in terms of higher Dice ratio, higher true positive fraction, and lower centroid distances. Also, the results demonstrate that simple manual specification can help improve the segmentation performance, which is clinically feasible in real practice. [ABSTRACT FROM AUTHOR]

Most of the existing Artificial Intelligence (AI) models for data regression commonly assume that the data samples are completely clean without noise or worst yet, only the symmetrical noise is in considerations. However in the real world applications, this is often not the case. This paper addresses a significant note of inefficiency in methods for regression when dealing with outliers, especially for cases with polarity of noise involved (i.e., one sided noise with either only positive noise or negative noise). Using soft margin loss function concept, we propose Constrained Optimization method based Extreme Learning Machine for Regression, hereafter denoted as CO-ELM-R. The proposed method incorporates the two Lagrange multipliers that mimic Support Vector Regression (SVR) into the basis of ELM to cope with infeasible constraints of the regression optimization problem. Thus, CO-ELM-R will complement the recursive iterations of SVR in the training phase due to the fact that ELM is much simpler in structure and faster in implementation. The proposed CO-ELM-R is evaluated empirically on a few benchmark data sets and a real world application of NO x gas emission data set collected from one of the power plant in Malaysia. The obtained results have demonstrated its validity and efficacy in handling noisy data regression problems. [ABSTRACT FROM AUTHOR]

In this paper, we show how to learn a hyperplane regressor by minimizing a tight or Θ bound on its VC dimension. While minimizing the VC dimension with respect to the defining variables is an ill posed and intractable problem, we propose a smooth, continuous, and differentiable function for a tight bound. Minimizing a tight bound yields the Minimal Complexity Machine (MCM) Regressor, and involves solving a simple linear programming problem. Experimental results show that on a number of benchmark datasets, the proposed approach yields regressors with error rates much lower than those obtained with conventional SVM regresssors, while often using fewer support vectors. On some benchmark datasets, the number of support vectors is less than one-tenth the number used by SVMs, indicating that the MCM does indeed learn simpler representations. [ABSTRACT FROM AUTHOR]

Feature selection plays an important role in many machine learning applications. By extracting meaningful features and eliminating both redundancies and noises, it effectively improves the accuracy and efficiency of the learning algorithm. In this paper, an unsupervised feature selection method called GSFS-llc (group sparse feature selection on local learning based clustering) is proposed. GSFS-llc first retrieves the cluster structure in a dataset using LLC and then selects important features that best preserve the cluster structure by L 2,1 -norm regularized regression. By combining group sparsity regularization with locality based learning algorithm, GSFS-llc leads a more robust result than other methods and selects features that respect the underlying geometric structure in the dataset. Extensive experimental results over real world handwritten digits, human faces, voices and object images datasets demonstrate the superiority of the GSFS-llc algorithm. [ABSTRACT FROM AUTHOR]

This paper concerns the universal consistency of extreme learning machine (ELM) for radial basis function networks (RBFNs). That is, the estimator constructed by ELM for RBFNs learning system can approximate an arbitrary regression function to any accuracy, as long as the number of the training samples is sufficiently large. Furthermore, we also give the conditions for the kernel functions, with which the corresponding ELM-RBFNs estimator is strongly universal consistency. These results not only underlie the feasibility of ELM for RBFNs case, but also provide guidance of practical selection for kernel functions in ELM application. [ABSTRACT FROM AUTHOR]

Light Detection and Ranging (LiDAR) is a remote sensor able to extract three-dimensional information. Environmental models in forest areas have been benefited by the use of LiDAR-derived information in the last years. A multiple linear regression (MLR) with previous stepwise feature selection is the most common method in the literature to develop those models. MLR defines the relation between the set of field measurements and the statistics extracted from a LiDAR flight. Machine learning has emerged as a suitable tool to improve classic stepwise MLR results on LiDAR. Unfortunately, few studies have been proposed to compare the quality of the multiple machine learning approaches. This paper presents a comparison between the classic MLR-based methodology and regression techniques in machine learning (neural networks, support vector machines, nearest neighbour, ensembles such as random forests) with special emphasis on regression trees. The selected techniques are applied to real LiDAR data from two areas in the province of Lugo (Galizia, Spain). The results confirm that classic MLR is outperformed by machine learning techniques and concretely, our experiments suggest that Support Vector Regression with Gaussian kernels statistically outperforms the rest of the techniques. [ABSTRACT FROM AUTHOR]

The increase of population causes the raise of security threat in crowed environment, which makes crowd counting becoming more and more important. For common complexity scenes, existing crowd counting approaches are mainly based on regression models which learn a mapping between low-level features and class labels. One of the major challenges for generating a good regression function is the insufficient and imbalanced training data. Observationally, the problem of crowd counting has the characteristic that crowd images with adjacent class labels contain similar features, which can be utilized to reduce the effect of insufficiency and imbalance by the strategy of information reuse. Consequently, this paper introduces a label distribution learning (LDL) strategy into crowd counting, where crowd images are labelled with label distributions instead of the conventional single labels. In this way, one crowd instance can contribute to not only the learning of its real class label, but also the learning of neighboring class labels. Hence the training data are increased significantly, and the classes with insufficient training data are supplied with more training data, which belongs to their adjacent classes. Experimental results prove the effectiveness and robustness of the LDL method for crowd counting. We have also shown the outstanding performance of the approach in different dataset. [ABSTRACT FROM AUTHOR]

Least square regression (LSR) and its variants have been widely used for classification tasks. However, LSR-based methods ignore the local geometry structure of the data and the transformation matrix is not sparse or robust. In this paper, a novel linear regression (LR) framework is proposed for image classification. Two concrete algorithms are proposed under the framework, which are named manifold discriminant regression learning (MDRL) and robust manifold discriminant regression learning (RMDRL). MDRL introduces different norms for different purposes in the learning steps. MDRL introduces a within-class graph and between-class graph to compute an optimal subspace that can separate data points belonging to different class as far as possible and keep the data points from the same class closely. MDRL joints different norms constraints to generate sparse projections for feature extraction and classification. To enhance the robustness of discriminative LSR (DLSR), RMDRL uses the nuclear norm as a regularization term to learn a robust projection matrix. Extensive experiments are conducted on many databases to evaluate the performance of the proposed methods and the states-of-the-art algorithms. The experimental results indicate that our proposed methods outperform the related algorithms. [ABSTRACT FROM AUTHOR]