Search

Broad learning system (BLS) is a novel and efficient model, which facilitates representation learning and classification by concatenating feature nodes and enhancement nodes. In spite of the efficient properties, BLS is still suboptimal when facing with imbalance problem. Besides, outliers and noises in imbalanced data remain a challenge for BLS. To address the above issues, in this paper we first propose a weighted BLS, which assigns a weight to each training sample, and adopt a general weighting scheme, which augments the weight of samples from the minority class. To further explore the prior distribution of original data, we design a density based weight generation mechanism to guide the specific weight matrix generation and propose the adaptive weighted broad learning system (AWBLS). This mechanism considers the inter-class and intra-class distance simultaneously in the density calculation. Finally, we propose the incremental weighted ensemble broad learning system (IWEB) by utilizing a progressive mechanism to further improve the stability and robustness of AWBLS. Extensive comparative experiments on 38 real-world data sets verfy that IWEB outperforms most of the imbalance ensemble classification methods. [ABSTRACT FROM AUTHOR]

Applications challenged by the joint problem of concept drift and class imbalance are attracting increasing research interest. This paper proposes a novel Reinforcement Online Active Learning Ensemble for Drifting Imbalanced data stream (ROALE-DI). The ensemble classifier has a long-term stable classifier and a dynamic classifier group which applies a reinforcement mechanism to increase the weight of the dynamic classifiers, which perform better on the minority class, and decreases the weight of the opposite. When the data stream is class imbalanced, the classifiers will lack the training samples of the minority class. To supply training samples, when creating a new classifier, the labeled instances buffer is used to provide instances of the minority class. Then, a hybrid labeling strategy that combines the uncertainty strategy and imbalance strategy is proposed to define whether to obtain the real label of an instance. An experimental evaluation compares the classification performance of the proposed method with semi-supervised and supervised algorithms on both real-world and synthetic data streams. The results show that the ROALE-DI achieves higher Area Under the ROC Curve (AUC) and accuracy values with even fewer real labels, and the labeling cost dynamically adjusts according to the concept drift and class imbalance ratio. [ABSTRACT FROM AUTHOR]

Though the convex optimization has been widely used in power systems, it still cannot guarantee to yield a tight (accurate) solution to some problems. To mitigate this issue, this paper proposes an ensemble learning based convex approximation for alternating current (AC) power flow equations that differs from the existing convex relaxations. The proposed approach is based on three-phase quadratic power flow equations in rectangular coordinates. To develop this data-driven convex approximation of power flows, the polynomial regression (PR) is first deployed as a basic learner to fit convex relationships between the independent and dependent variables. Then, ensemble learning algorithms such as gradient boosting (GB) and bagging are introduced to combine learners to boost model performance. Based on the learned convex approximation of power flow, optimal power flow (OPF) is formulated as a convex quadratic programming problem. The simulation results on IEEE standard cases of both balanced and unbalanced systems show that, in the context of solving OPF, the proposed data-driven convex approximation outperforms the conventional semi-definite programming (SDP) relaxation in both accuracy and computational efficiency, especially in the cases that the conventional SDP relaxation fails. [ABSTRACT FROM AUTHOR]

Automated inspection systems play an important role in manufacturing to guarantee higher quality and reduce production costs. In the semiconductor manufacturing industry, assembly and testing processes are getting more complex, resulting in a greater tendency of defects to impact the production process. These defects can cause field failures and can result in customer dissatisfactions and returns. Currently available defect detection and classification systems are customized and hard-wired to the detection of particular classes of defects and cannot deal with new unknown classes of defects. This issue is aggravated by the very small sample size of available anomalies for learning, by the data imbalance problem, since the number of defective samples is significantly much smaller than the number of normal samples, and by the presence of noise. This paper presents a novel multifeature, sparse-based defect detection and classification approach that uses the stacking concept to enhance the classification accuracy. The stacking-based classifier is augmented with a novel adaptive over/downsampling technique to deal with the data imbalance problem. A new pruning technique is proposed to eliminate bad base learners. Shortage of defective units, similarities within different classes of defects, wide variation within the same defect class, and data imbalance are the basic challenges to deal with. Experimental results on real-world data from Intel show that the proposed approach results in a high classification accuracy as compared with the existing methods.

Note to Practitioners—The basic motivation of this paper is to design an automated cost-effective, adaptive, and intelligent defect detection and classification system that is easy to train using a small-size sample set of defects and that is robust to noise. The system is scalable in terms of the numbers and types of defects and features, which leads to a shorter development cycle. The presented system is immediately applicable to different types of defects. Inputs of the system are grayscale images. These images are processed to perform defects detection, features extraction, and classification. [ABSTRACT FROM PUBLISHER]

For the fault classification of chemical industries, the typical Fisher discriminant analysis (FDA) model requires that all the training samples should be correctly labeled. Actually, training samples tend to be polluted by mislabeled samples for some unavoidable reasons, and thus the performance of the FDA may be severely affected. However, in the chemical process monitoring community, there is little attention to fault classification with label-noise in training samples. To settle this open issue, a manifold-preserving sparse graph (MPSG)-based ensemble FDA model is initially proposed in this paper. First, under the condition of maintaining the underlying manifold structure of training samples, an MPSG is utilized to filter some mislabeled samples. Second, to improve model robustness to the left mislabeled samples, Bagging based on FDA is used to construct several subclassifiers, which are combined to form a robust classifier. Experiments on the benchmark Tennessee Eastman process and a real industrial air separation unit demonstrate the effectiveness and the superiority of the proposed model. [ABSTRACT FROM AUTHOR]

Accurate and reliable low-voltage load forecasting is critical to optimal operation and control of distribution network and smart grid. However, compared to traditional regional load forecasting at high-voltage level, it faces tough challenges due to the inherent high uncertainty of the low-capacity load and distributed renewable energy integrated in the demand side. This paper proposes a novel hybrid ensemble deep learning (HEDL) approach for deterministic and probabilistic low-voltage load forecasting. The deep belief network (DBN) is applied to low-voltage load point prediction with the strong ability of approximating nonlinear mapping. A series of ensemble learning methods including bagging and boosting variants are introduced to improve the regression ability of DBN. In addition, the differencing transformation technique is utilized to ensure the stationarity of load time series for the application bagging and boosting methods. On the basis of the integrated thought of ensemble learning, a new hybrid ensemble algorithm is developed via integrating multiple separate ensemble methods. Considering the diversity in various ensemble algorithms, an effective K nearest neighbor classification method is utilized to adaptively determine the weights of sub-models. Furthermore, HEDL based probabilistic forecasting is proposed by taking advantage of the inherent resample idea in bagging and boosting. The effectiveness of the HEDL method for both deterministic and probabilistic forecasting has been systematically verified based on realistic load data from East China and Australia, indicating its promising prospective for practical applications in distribution networks. [ABSTRACT FROM AUTHOR]

Traditional learning from crowdsourced labeled data consists of two stages: inferring true labels for instances from their multiple noisy labels and building a learning model using these instances with the inferred labels. This straightforward two-stage learning scheme suffers from two weaknesses: (1) the accuracy of inference may be very low; (2) useful information may be lost during inference. In this paper, we proposed a novel ensemble method for learning from crowds. Our proposed method is a meta-learning scheme. It first uses a bootstrapping process to create $M$M sub-datasets from an original crowdsourced labeled dataset. For each sub-dataset, each instance is duplicated with different weights according to the distribution and class memberships of its multiple noisy labels. A base classifier is then trained from this extended sub-dataset. Finally, unlabeled instances are predicted by aggregating the outputs of these $M$M base classifiers. Because the proposed method gets rid of the inference procedure and uses the full dataset to train learning models, it preserves the useful information for learning as much as possible. Experimental results on nine simulated and two real-world crowdsourcing datasets consistently show that the proposed ensemble learning method significantly outperforms five state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In one-class classification problems, only the data for the target class is available, whereas the data for the non-target class may be completely absent. In this paper, we study one-class nearest neighbor (OCNN) classifiers and their different variants. We present a theoretical analysis to show the relationships among different variants of OCNN that may use different neighbors or thresholds to identify unseen examples of the non-target class. We also present a method based on inter-quartile range for optimizing parameters used in OCNN in the absence of non-target data during training. Then, we propose two ensemble approaches based on random subspace and random projection methods to create accurate OCNN ensembles. We tested the proposed methods on 15 benchmark and real world domain-specific datasets and show that random-projection ensembles of OCNN perform best. [ABSTRACT FROM AUTHOR]

An open question in ensemble-based active learning is how to choose one classifier type, or appropriate combinations of multiple classifier types, to construct ensembles for a given task. While existing approaches typically choose one classifier type, this paper presents a method that trains and adapts multiple instances of multiple classifier types toward an appropriate ensemble during active learning. The method is termed adaptive heterogeneous ensembles (henceforth referred to as AHE). Experimental evaluations show that AHE constructs heterogeneous ensembles that outperform homogeneous ensembles composed of any one of the classifier types, as well as bagging, boosting and the random subspace method with random sampling. We also show in this paper that the advantage of AHE over other methods is increased if (1) the overall size of the ensemble also adapts during learning; and (2) the target data set is composed of more than two class labels. Through analysis we show that the AHE outperforms other methods because it automatically discovers complementary classifiers: for each data instance in the data set, instances of the classifier type best suited for that data point vote together, while instances of the other, inappropriate classifier types disagree, thereby producing a correct overall majority vote. [ABSTRACT FROM AUTHOR]

In the semiconductor packaging system composed of complex assembly processes, it is necessary to detect faults early to minimize the time and cost required to produce semiconductor chip scraps and to improve production yield. Early detection of faults is possible by performing a failure prediction analysis during the semiconductor packaging process. In this paper, we consider a statistical process control model for predicting the final failure of a printed circuit board lot based on observed event sequences in the wire bonding process step. To estimate the parameters efficiently in the predictive model, we propose a two-stage process. Here, irrelevant subsequences of events are deleted by a sequential mining approach at the first step, and a predictive model is constructed with the remaining subsequences of events using the logistic regression and bagged least absolute shrinkage and selection operator (LASSO) estimate, which we call the B-LASSO method. In particular, to resolve problems caused by unbalanced data, the B-LASSO uses case-control sampling rather than simple random sampling with replacement. The performance of the B-LASSO is compared with other competing methods by analyzing a work-site dataset to confirm that the B-LASSO is superior. [ABSTRACT FROM PUBLISHER]

While both cost-sensitive learning and online learning have been studied separately, these two issues have seldom been addressed simultaneously. Yet, there are many applications where both aspects are important. This paper investigates a class of algorithmic approaches suitable for online cost-sensitive learning, designed for such problems. The basic idea is to leverage existing methods for online ensemble algorithms, and combine these with batch mode methods for cost-sensitive bagging/boosting algorithms. Within this framework, we describe several theoretically sound online cost-sensitive bagging and online cost-sensitive boosting algorithms, and show that the convergence of the proposed algorithms is guaranteed under certain conditions. We then present extensive experimental results on benchmark datasets to compare the performance of the various proposed approaches. [ABSTRACT FROM PUBLISHER]

This paper compares the performance of several boosting and bagging techniques in the context of learning from imbalanced and noisy binary-class data. Noise and class imbalance are two well-established data characteristics encountered in a wide range of data mining and machine learning initiatives. The learning algorithms studied in this paper, which include SMOTEBoost, RUSBoost, Exactly Balanced Bagging, and Roughly Balanced Bagging, combine boosting or bagging with data sampling to make them more effective when data are imbalanced. These techniques are evaluated in a comprehensive suite of experiments, for which nearly four million classification models were trained. All classifiers are assessed using seven different performance metrics, providing a complete perspective on the performance of these techniques, and results are tested for statistical significance via analysis-of-variance modeling. The experiments show that the bagging techniques generally outperform boosting, and hence in noisy data environments, bagging is the preferred method for handling class imbalance. [ABSTRACT FROM PUBLISHER]

Short-term load forecasting is always a popular topic in the electric power industry because of its essentiality in energy system planning and operation. In the deregulated power system, an improvement of a few percentages in the prediction accuracy would bring benefits worth of millions of dollars, which makes load forecasting become more important than ever before. This paper focuses on the short-term load forecasting for a power system in the U.S., where several alternative meteorological forecasts are available from different commercial weather services. To effectively take advantage of the alternative meteorological predictions in the load forecasting system, a new comprehensive forecasting methodology has been proposed in this paper. Specifically, combining forecasting using adaptive coefficients is applied to share the strength of the different temperature forecasts in the first stage, and then, ensemble neural networks have been used to improve the model's generalization performance based on bagging. The proposed load forecasting system has been verified by using the real data from the utility. A range of comparisons with different forecasting models have been conducted. The forecasting results demonstrate the superiority of the proposed methodology. [ABSTRACT FROM AUTHOR]

This paper focuses on the measure of recommendation stability, which reflects the consistency of recommender system predictions. Stability is a desired property of recommendation algorithms and has important implications on users’ trust and acceptance of recommendations. Prior research has reported that some popular recommendation algorithms can suffer from a high degree of instability. In this study, we explore two scalable, general-purpose meta-algorithmic approaches—based on bagging and iterative smoothing—that can be used in conjunction with different traditional recommendation algorithms to improve their stability. Our experimental results on real-world rating data demonstrate that both approaches can achieve substantially higher stability as compared to the original recommendation algorithms. Furthermore, perhaps as importantly, the proposed approaches not only do not sacrifice the predictive accuracy in order to improve recommendation stability, but are actually able to provide additional accuracy improvements. [ABSTRACT FROM AUTHOR]

Online class imbalance learning is a new learning problem that combines the challenges of both online learning and class imbalance learning. It deals with data streams having very skewed class distributions. This type of problems commonly exists in real-world applications, such as fault diagnosis of real-time control monitoring systems and intrusion detection in computer networks. In our earlier work, we defined class imbalance online, and proposed two learning algorithms OOB and UOB that build an ensemble model overcoming class imbalance in real time through resampling and time-decayed metrics. In this paper, we further improve the resampling strategy inside OOB and UOB, and look into their performance in both static and dynamic data streams. We give the first comprehensive analysis of class imbalance in data streams, in terms of data distributions, imbalance rates and changes in class imbalance status. We find that UOB is better at recognizing minority-class examples in static data streams, and OOB is more robust against dynamic changes in class imbalance status. The data distribution is a major factor affecting their performance. Based on the insight gained, we then propose two new ensemble methods that maintain both OOB and UOB with adaptive weights for final predictions, called WEOB1 and WEOB2. They are shown to possess the strength of OOB and UOB with good accuracy and robustness. [ABSTRACT FROM AUTHOR]

In this paper, we present a novel ensemble method random projection random discretization ensembles(RPRDE) to create ensembles of linear multivariate decision trees by using a univariate decision tree algorithm. The present method combines the better computational complexity of a univariate decision tree algorithm with the better representational power of linear multivariate decision trees. We develop random discretization (RD) method that creates random discretized features from continuous features. Random projection (RP) is used to create new features that are linear combinations of original features. A new dataset is created by augmenting discretized features (created by using RD) with features created by using RP. Each decision tree of a RPRD ensemble is trained on one dataset from the pool of these datasets by using a univariate decision tree algorithm. As these multivariate decision trees (because of features created by RP) have more representational power than univariate decision trees, we expect accurate decision trees in the ensemble. Diverse training datasets ensure diverse decision trees in the ensemble. We study the performance of RPRDE against other popular ensemble techniques using C4.5 tree as the base classifier. RPRDE matches or outperforms other popular ensemble methods. Experiments results also suggest that the proposed method is quite robust to the class noise. [ABSTRACT FROM AUTHOR]

Classifier learning with data-sets that suffer from imbalanced class distributions is a challenging problem in data mining community. This issue occurs when the number of examples that represent one class is much lower than the ones of the other classes. Its presence in many real-world applications has brought along a growth of attention from researchers. In machine learning, the ensemble of classifiers are known to increase the accuracy of single classifiers by combining several of them, but neither of these learning techniques alone solve the class imbalance problem, to deal with this issue the ensemble learning algorithms have to be designed specifically. In this paper, our aim is to review the state of the art on ensemble techniques in the framework of imbalanced data-sets, with focus on two-class problems. We propose a taxonomy for ensemble-based methods to address the class imbalance where each proposal can be categorized depending on the inner ensemble methodology in which it is based. In addition, we develop a thorough empirical comparison by the consideration of the most significant published approaches, within the families of the taxonomy proposed, to show whether any of them makes a difference. This comparison has shown the good behavior of the simplest approaches which combine random undersampling techniques with bagging or boosting ensembles. In addition, the positive synergy between sampling techniques and bagging has stood out. Furthermore, our results show empirically that ensemble-based algorithms are worthwhile since they outperform the mere use of preprocessing techniques before learning the classifier, therefore justifying the increase of complexity by means of a significant enhancement of the results. [ABSTRACT FROM PUBLISHER]

This paper introduces a novel concept for creating an ensemble of classifiers. The concept is based on generating an ensemble of classifiers through clustering of data at multiple layers. The ensemble classifier model generates a set of alternative clustering of a dataset at different layers by randomly initializing the clustering parameters and trains a set of base classifiers on the patterns at different clusters in different layers. A test pattern is classified by first finding the appropriate cluster at each layer and then using the corresponding base classifier. The decisions obtained at different layers are fused into a final verdict using majority voting. As the base classifiers are trained on overlapping patterns at different layers, the proposed approach achieves diversity among the individual classifiers. Identification of difficult-to-classify patterns through clustering as well as achievement of diversity through layering leads to better classification results as evidenced from the experimental results. [ABSTRACT FROM AUTHOR]

This paper presents a SNCCDBAGG-based neural network (NN) ensemble approach for quality prediction in injection molding process. Bagging is used to create NNs for the ensemble by independently training these NNs on different training sets. Negative correlation learning via correlation-corrected data (NCCD) is used to achieve negative correlation of each network's error against errors for the rest of the ensemble by training transformed target data for NN in the ensemble as the desired network output for some epochs. A selection-based strategy is proposed to improve generalization ability when combining Bagging and NCCD. Experimental results show its good performance on quality predicting in injection molding process compared with single NN predictor and NCCD predictor. [ABSTRACT FROM AUTHOR]

In this paper, we propose two cooperative ensemble learning algorithms, i.e., NegBagg and NegBoost, for designing neural network (NN) ensembles. The proposed algorithms incrementally train different individual NNs in an ensemble using the negative correlation learning algorithm. Bagging and boosting algorithms are used in NegBagg and NegBoost, respectively, to create different training sets for different NNs in the ensemble. The idea behind using negative correlation learning in conjunction with the bagging/boosting algorithm is to facilitate interaction and cooperation among NNs during their training. Both NegBagg and NegBoost use a constructive approach to automatically determine the number of hidden neurons for NNs. NegBoost also uses the constructive approach to automatically determine the number of NNs for the ensemble. The two algorithms have been tested on a number of benchmark problems in machine learning and NNs, including Australian credit card assessment, breast cancer, diabetes, glass, heart disease, letter recognition, satellite, soybean, and waveform problems. The experimental results show that NegBagg and NegBoost require a small number of training epochs to produce compact NN ensembles with good generalization. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of the optimal design of numerical experiments for the construction of nonlinear surrogate models. We describe a new method, called learner disagreement from experiment resampling (LDR), which borrows ideas from active learning and from resampling methods: the analysis of the divergence of the predictions provided by a population of models, constructed by resampling, allows an iterative determination of the point of input space, where a numerical experiment should be performed in order to improve the accuracy of the predictor. The LDR method is illustrated on neural network models with bootstrap resampling, and on orthogonal polynomials with leave-one-out resampling. Other methods of experimental design such as random selection and D-optimal selection are investigated on the same benchmark problems. [ABSTRACT FROM AUTHOR]

Drug-related problems, particularly those that result from sub- or overtherapeutic doses of high-alert medications, have become a growing concern in clinical medicine. In this paper, we use a model-tree-based regression technique (namely, M5) and support vector machine (SVM) for regression to develop learning-based systems for predicting the adequacy of a vancomycin regimen. We empirically evaluate each system's accuracy in predicting patients' peak and trough concentrations in different clinical scenarios characterized by renal functions and regimen types. Our data consist of 1099 clinical cases that were collected from a major tertiary medical center in southern Taiwan. We also examine the use of Bagging for enhancing the prediction power of the respective systems and include in our evaluation a salient one-compartment model for performance benchmark purposes. Overall, our evaluation results suggest that both M5 and SVM are significantly more accurate than the benchmark one-compartment model in predicting patients' peak and trough concentrations across all investigated clinical scenarios. MS appears to benefit considerably from Bagging, which has a positive but seemingly smaller effect on SVM. Taken together, our findings indicate supervised learning techniques that are capable of effectively supporting clinicians' use of vancomycin or similar high-alert drugs in their patient care and management. [ABSTRACT FROM AUTHOR]

Ensemble learning strategies, especially Boosting and Bagging decision trees, have demonstrated impressive capacities to improve the prediction accuracy of base learning algorithms. Further gains have been demonstrated by strategies that combine simple ensemble formation approaches. In this paper, we investigate the hypothesis that the improvement in accuracy of multistrategy approaches to ensemble learning is due to an increase in the diversity of ensemble members that are formed. In addition, guided by this hypothesis, we develop three new multistrategy ensemble learning techniques. Experimental results in a wide variety of natural domains suggest that these multistrategy ensemble learning techniques are, on average, more accurate than their component ensemble learning techniques. [ABSTRACT FROM AUTHOR]

This paper presents a novel cluster-oriented ensemble classifier. The proposed ensemble classifier is based on original concepts such as learning of cluster boundaries by the base classifiers and mapping of cluster confidences to class decision using a fusion classifier. The categorized data set is characterized into multiple clusters and fed to a number of distinctive base classifiers. The base classifiers learn cluster boundaries and produce cluster confidence vectors. A second level fusion classifier combines the cluster confidences and maps to class decisions. The proposed ensemble classifier modifies the learning domain for the base classifiers and facilitates efficient learning. The proposed approach is evaluated on benchmark data sets from UCI machine learning repository to identify the impact of multicluster boundaries on classifier learning and classification accuracy. The experimental results and two-tailed sign test demonstrate the superiority of the proposed cluster-oriented ensemble classifier over existing ensemble classifiers published in the literature. [ABSTRACT FROM AUTHOR]

Recently, the classification study is accelerated, especially in machine learning expertise. Although the decision tree was still recommended as a classification tool in diagnosing electric power apparatus because of the property having the visible if-then rule, the recent development in classification methods, especially those using the ensemble methods, suggests us to apply these methods to condition diagnosis area. In this paper, we report that the new ensemble methods show extremely high accuracy in classification of the electric power apparatus diagnosis, although rule visibility is sacrificed. [ABSTRACT FROM AUTHOR]