Search

Due to that multivariate time series, multistep forecasting technology has a guiding role in many fields, such as electricity consumption, traffic flow detection, and stock price prediction, many approaches have been proposed, seeking to realize accurate prediction based on historical data. However, multivariate time series in real-world applications often contain complex and non-linear interdependencies between time steps and series, for which traditional approaches that just model dependencies in time dimension may fail to make forecasting accurately. In addition, the traditional cyclic multistep prediction strategy leads to error accumulation and consequently results in the reduction of prediction performance. To address these limitations, we propose a novel Deep Multivariate Time Series Multistep Forecasting Network based on the Encoder-Decoder model, abbreviated as DualMNet. DualMNet uses the temporal patterns module to capture long-term patterns between time steps, and, simultaneously, employs the spatial patterns module to discover the interdependencies between time series. Furthermore, the decoder utilizes the historical data of the target sequence to predict the long time-series sequences at the final forward operation, significantly ameliorating the issue of error accumulation. Finally, the conducted t-test results, based upon the extensive experimental results on the three benchmark multivariate time series datasets, they demonstrate that the multistep predictive performance of DualMNet is significantly superior to those of the comparison models, and the ablation study shows that integrating all the components of DualMNet together results in robust forecasting performance. [ABSTRACT FROM AUTHOR]

With the rapid development of deep learning, in recent years, many excellent deep learning models have been developed to solve the problem of video frame prediction. Among them, most models directly generate predicted target frames. However, the predicted frames obtained in this way are often fuzzy and not realistic enough. In order to solve this problem, this paper first attempts to integrate the attention mechanism with Convolutional Long Short-Term Memory, and correspondingly proposes a new deep learning model, abbreviated as AttConvLSTM. One prominent and original characteristic of this newly constructed model is that, each of its layer calculates the attention weight of the obtained information, focusing on the information key part. Although the proposed AttConvLSTM model effectively improves the prediction accuracy, it still does not solve the problem that the prediction frames directly generated by classical deep learning models are often fuzzy and not realistic. Therefore, inspired by the concept of optical flow, this work further develops a novel Deep Pixel Restoration AttConvLSTM (DPRAConvLSTM) model. This model cleverly uses the input frames and the end-to-end characteristics of deep learning. We innovatively restore the pixels of the input frames to the predicted frames, thereby avoiding the defects that typical deep learning models can easily cause, when directly generating the predicted frames. The experimental results effectively confirm that the finally formed DPRAConvLSTM model can not only improve the accuracy of prediction, but also obtain clearer and more realistic prediction frames. [ABSTRACT FROM AUTHOR]

In this work, an Incremental Learning Algorithm via Dynamically Weighting Ensemble Learning (DWE-IL) is proposed to solve the problem of Non-Stationary Time Series Prediction (NS-TSP). The basic principle of DWE-IL is to track real-time data changes by dynamically establishing and maintaining a knowledge base composed of multiple basic models. It trains the base model for each non-stationary time series subset, and finally combine each base model with dynamically weighting rules. The emphasis of the DWE-IL algorithm lies in the update of data weights and base model weights and the training of the base model. Finally, the experimental results of the DWE-IL algorithm on six non-stationary time series datasets are presented and compared with those of several other excellent algorithms. It can be concluded from the experimental results that the DWE-IL algorithm provides a good solution to the challenges of the NS-TSP tasks and has significantly superior performance over other comparative algorithms. [ABSTRACT FROM AUTHOR]

In Time Series Forecasting (TSF), researchers usually assume that there is enough training data can be obtained, with the old a'nd new data satisfying the same distribution. However, time series data always produces some time-varying characteristics over time, which will lead to relatively large differences between old and new data. As we all know, single-source TSF Transfer Learning (TL) faces the problem of negative transfer. Addressing this issue, this paper proposes a new Multi-Source TL algorithm, abbreviated as the MultiSrcTL algorithm, and a novel Active Multi-Source Transfer Learning, abbreviated as the AcMultiSrcTL algorithm, with the latter one integrating Multi-Source TL with Active Learning (AL), and taking the former one as its sub-algorithm. We introduce domain adaptation theory into this work, and analyze the expected target risk of TSF under the multi-source setting, accordingly. For the development of MultiSrcTL, we make full use of source similarity and domain dependability, using the Maximum Mean Discrepancy statistical indicator to measure the similarity between domains, so as to promote better transfer. A domain relation matrix is constructed to describe the relationship between source domains, so that the source-source and source-target relations are adequately considered. In the design of AcMultiSrcTL, Kullback-Leibler divergence is used to measure the similarity of related indicators to select the appropriate source domain. The uncertainty sampling method and the distribution match weighting technique are integrated, obtaining a new sample selection scheme. The empirical results on six benchmark datasets demonstrate the applicability and effectiveness of the two proposed algorithms for multi-source TSF TL. [ABSTRACT FROM AUTHOR]

Ensemble pruning can effectively overcome several shortcomings of the classical ensemble learning paradigm, such as the relatively high time and space complexity. However, each predictor has its own unique ability. One predictor may not perform well on some samples, but it will perform very well on other samples. Blindly underestimating the power of specific predictors is unreasonable. Choosing the best predictor set for each query sample is exactly what dynamic ensemble pruning techniques address. This paper proposes a hybrid Time Series Prediction (TSP) algorithm to implement one-step-ahead prediction task, integrating Dynamic Ensemble Pruning (DEP), Incremental Learning (IL), and Kernel Density Estimation (KDE), abbreviated as the EnsPKDE&IncLKDE algorithm. It dynamically selects proper predictor sets based on the kernel density distribution of all base learners' prediction values. Due to the characteristic of TSP problems that samples arrive in chronological order, the idea of IL is naturally introduced into EnsPKDE&IncLKDE, while DEP is a common technology to address the concept drift issue inherent in IL. The algorithm is divided into three subprocesses: 1) Overproduction, which generates the original ensemble learning system; 2) Dynamic Ensemble Pruning (DEP), achieved by one subalgorithm called EnsPKDE; 3) Incremental Learning (IL), realized by one subalgorithm termed IncLKDE. Benefited from the advantages of integrating Dynamic Ensemble Pruning scheme, Incremental Learning paradigm and Kernel Density Estimation, in the experimental results, EnsPKDE&IncLKDE demonstrates superior prediction performance to several other state-of-the-art algorithms in fulfilling time series forecasting tasks. [ABSTRACT FROM AUTHOR]

Extreme learning machine (ELM) and its variants have been widely used in the field of object recognition and other complex classification tasks. Traditional deep learning architectures like Convolutional Neural Network (CNN) are capable of extracting high-level features, which are the key for the models to make right decisions. However, traditional deep architectures are confronted with solving a tough, non-convex optimization problem, which is a time-consuming process. In this paper, we propose two hierarchical models, i.e., Random Recursive Constrained ELM (R2CELM) and Random Recursive Local- Receptive-Fields-Based ELM (R2ELM-LRF), which are constructed by stacking with CELM or ELM-LRF, respectively. Besides, inspired by the stacking generalization philosophy, random projection and kernelization are incorporated as their constitutive elements. R2CELM and R2ELM-LRF not only fully inherit the merits of ELM, but also take advantage of the superiority of CELM and ELM-LRF in the field of image recognition, respectively. The essence of CELM is to constrain the weight vectors from the input layer to the hidden layer to be consistent with the directions from one class to another class, while ELM-LRF is adept at exploiting the local structures in images through many local receptive fields. In the empirical results, R2CELM and R2ELM-LRF demonstrate their better performance in testing accuracy on the six benchmark image recognition datasets, compared with their basic learners and other state-of-the-art algorithms. Moreover, the proposed two deep ELM models need less training time when compared with traditional Deep Neural Network (DNN) based models. [ABSTRACT FROM AUTHOR]

A major assumption in traditional machine leaning is that the training and testing data must be in the same feature space and have the same distribution. However, in many real-world applications, this assumption may not hold. In recent years, transfer learning has emerged as a new learning paradigm to cope with this considerable challenge. It focuses on exploiting previously learnt knowledge by leveraging information from an old source domain to help learning in a new target domain. In this work, we integrate the knowledge-leverage-based Transfer Learning mechanism with a Rank-based Reduce Error ensemble selection approach to fulfill the transfer learning task, called RankRE-TL. Ensemble selection is important for improving both efficiency and predictive accuracy of an ensemble system. It aims to select a proper subset of the whole ensemble, which usually outperforms the whole one. Therefore, we appropriately modify the Reduce Error (RE) pruning technique and design a new Rank-based Reduce Error ensemble selection method (RankRE) to deal with the transfer learning task. The design idea of RankRE is to find the candidate classifier which is expected to improve the classification performance of the extended subensemble the most. In the RankRE-TL algorithm, the initial Support Vector Machine (SVM) ensemble is learnt based upon dynamic training dataset regrouping. And simultaneously, a new construction method of validation set is designed for RankRE-TL, which differs from the method used in conventional ensemble selection paradigm. [ABSTRACT FROM AUTHOR]

Deep multi-layered neural networks have nonlinear levels that allow them to represent highly varying nonlinear functions compactly. In this paper, we propose a new deep architecture with enhanced model discrimination ability that we refer to as mlpconv-wise supervised pre-training network in network (MPNIN). The process of information abstraction is facilitated within the receptive fields for MPNIN. The proposed architecture uses the framework of the recently developed NIN structure, which slides a universal approximator, such as a multilayer perceptron with rectifier units, across an image to extract features. However, the random initialization of NIN can produce poor solutions to gradient-based optimization. We use mlpconv-wise supervised pre-training to remedy this defect because this pre-training technique may contribute to overcoming the difficulties of training deep networks by better initializing the weights in all the layers. Moreover, batch normalization is applied to reduce internal covariate shift by pre-conditioning the model. Empirical investigations are conducted on the Mixed National Institute of Standards and Technology (MNIST), the Canadian Institute for Advanced Research (CIFAR-10), CIFAR-100, the Street View House Numbers (SVHN), the US Postal (USPS), Columbia University Image Library (COIL20), COIL100 and Olivetti Research Ltd (ORL) datasets, and the results verify the effectiveness of the proposed MPNIN architecture. [ABSTRACT FROM AUTHOR]

This paper describes the development of an effective and efficient Hierarchical and Parallel Branch-and-Bound Ensemble Selection (H&PB&BEnS) algorithm. Using the proposed H&PB&BEnS, ensemble selection is accomplished in a divisional, parallel, and hierarchical way. H&PB&BEnS uses the superior performance of the Branch-and-Bound (B&B) algorithm in relation to small-scale combinational optimization problems, whilst also managing to avoid 'the curse of dimensionality' that can result from the direct application of B&B to ensemble selection problems. The B&B algorithm is used to select each partitioned subensemble, which enhances the predictive accuracy of each pruned subsolution, and then the working mechanism of H&PB&BEnS improves the diversity of the ensemble selection results. H&PB&BEnS realizes layer-wise refinement of the selected ensemble solutions, which enables the classification performance of the selected ensembles to be improved in a layer-by-layer manner. Empirical investigations are conducted using five benchmark classification datasets, and the results verify the effectiveness and efficiency of the proposed H&PB&BEnS algorithm. [ABSTRACT FROM AUTHOR]

Although ordering-based pruning algorithms possess relatively high efficiency, there remains room for further improvement. To this end, this paper describes the combination of a dynamic programming technique with the ensemble-pruning problem. We incorporate dynamic programming into the classical ordering-based ensemble-pruning algorithm with complementariness measure (ComEP), and, with the help of two auxiliary tables, propose a reasonably efficient dynamic form, which we refer to as ComDPEP. To examine the performance of the proposed algorithm, we conduct a series of simulations on four benchmark classification datasets. The experimental results demonstrate the significantly higher efficiency of ComDPEP over the classic ComEP algorithm. The proposed ComDPEP algorithm also outperforms two other state-of-the-art ordering-based ensemble-pruning algorithms, which use uncertainty weighted accuracy and reduce-error pruning, respectively, as their measures. It is noteworthy that, the effectiveness of ComDPEP is just the same with that of the classical ComEP algorithm. [ABSTRACT FROM AUTHOR]

The greedy randomized adaptive search procedure (GRASP) is an iterative two-phase multi-start metaheuristic procedure for a combination optimization problem, while path relinking is an intensification procedure applied to the solutions generated by GRASP. In this paper, a hybrid ensemble selection algorithm incorporating GRASP with path relinking (PRelinkGraspEnS) is proposed for credit scoring. The base learner of the proposed method is an extreme learning machine (ELM). Bootstrap aggregation (bagging) is used to produce multiple diversified ELMs, while GRASP with path relinking is the approach for ensemble selection. The advantages of the ELM are inherited by the new algorithm, including fast learning speed, good generalization performance, and easy implementation. The PRelinkGraspEnS algorithm is able to escape from local optima and realizes a multi-start search. By incorporating path relinking into GRASP and using it as the ensemble selection method for the PRelinkGraspEnS the proposed algorithm becomes a procedure with a memory and high convergence speed. Three credit datasets are used to verify the efficiency of our proposed PRelinkGraspEnS algorithm. Experimental results demonstrate that PRelinkGraspEnS achieves significantly better generalization performance than the classical directed hill climbing ensemble pruning algorithm, support vector machines, multi-layer perceptrons, and a baseline method, the best single model. The experimental results further illustrate that by decreasing the average time needed to find a good-quality subensemble for the credit scoring problem, GRASP with path relinking outperforms pure GRASP (i.e., without path relinking). [ABSTRACT FROM AUTHOR]

As is well known, the Greedy Ensemble Pruning (GEP) algorithm, also called the Directed Hill Climbing Ensemble Pruning (DHCEP) algorithm, possesses relatively good performance and high speed. However, because the algorithm only explores a relatively small subspace within the whole solution space, it often produces suboptimal solutions of the ensemble pruning problem. Aiming to address this drawback, in this work, we propose a novel Randomized GEP (RandomGEP) algorithm, also called the Randomized DHCEP (RandomDHCEP) algorithm, that effectively enlarges the search space of the classical DHCEP while maintaining the same level of time complexity with the help of a randomization technique. The randomization of the classical DHCEP algorithm achieves a good tradeoff between the effectiveness and efficiency of ensemble pruning. Besides, the RandomDHCEP algorithm naturally inherits the two intrinsic advantages that a randomized algorithm usually possesses. First, in most cases, its running time or space requirements are smaller than well-behaved deterministic ensemble pruning algorithms. Second, it is easy to comprehend and implement. Experimental results on three benchmark classification datasets verify the practicality and effectiveness of the RandomGEP algorithm. [ABSTRACT FROM AUTHOR]