Search

Feedforward neural network (FNN) is the most popular network model, and the appropriate structure and learning algorithms are the key of its performance. This paper proposes an online self-organizing algorithm for feedforward neural network (OSNN) with a single hidden layer. The proposed OSNN optimizes the structure of FNN for time-varying system including structure design and parameter learning. In structure design, this paper measures the contribution ratios of hidden nodes by local sensitivity analysis based on differentiation method. OSNN merges hidden nodes with the others that have the highest correlation when their contribution ratios are almost zero and adds new hidden nodes by error reparation. For parameter learning, an improved online gradient method (OGM), called online gradient method with fixed memory (FMOGM), is proposed to improve the convergence speed and accuracy of OGM. In addition, this paper calculates the contribution ratios and the network error and estimates the local minima by using the fixed-sized training set of FMOGM instead of one sample at the current time, which can obtain more effective local information and a compact network structure. Finally, the proposed OSNN is verified using a number of benchmark problems and a practical problem for biochemical oxygen demand prediction in wastewater treatment. The experimental results show that OSNN has better convergence speed and accuracy than other algorithms. [ABSTRACT FROM AUTHOR]

Industrial process data are naturally in the form of complex time-series with high nonlinearities and dynamics. Long short-term memory (LSTM) networks are suitable for developing prediction models to handle nonlinear and dynamic process. However, LSTM neural networks have typically large and predefined structures, which may result in overfitting, and an optimal hidden neurons for a given problem cannot be automatically obtained. For this reason, a regularized self-organizing LSTM (RSO-LSTM) is proposed to optimize both the structure and the parameters of the network. First, an adaptive learning algorithm based on l 2-norm regularization is introduced for parameter adjustment. Thereafter, both the prediction accuracy and weight dispersion are considered to avoid overfitting. Second, a growing strategy is designed based on hidden neuronal sensitivity. The structure of the LSTM can then be determined automatically with improved compactness. Finally, a convergence analysis is performed to ensure the feasibility of RSO-LSTM. To demonstrate the merits of the proposed RSO-LSTM for time-series​ prediction, its results for three benchmark experiments and real industrial data of a municipal solid waste incineration process were examined and compared with those of other methods. The results indicated the superiority and potential of RSO-LSTM for industrial applications. • An adaptive learning algorithm based on l2-norm regularization is introduced for parameter adjustment. • The structure of the LSTM can be determined automatically with improved compactness via growth strategy. • The importance of the hidden neurons is measured by neuronal sensitivity. • The convergence of RSO-LSTM can be guaranteed. [ABSTRACT FROM AUTHOR]

This paper investigates how to construct a recurrent radial basis function neural network (RRBFNN) by an information-oriented algorithm (IOA) and how to adjust the parameters by a gradient algorithm simultaneously. In this IOA-based RRBFNN (IOA-RRBFNN), the proposed IOA is used to calculate the information processing strength (IPS) of hidden neurons, such that the independent component contributions between the hidden neurons and output neurons can be extracted. Then, a novel self-organizing strategy is proposed to optimize the structure of RRBFNN based on the input IPS and output IPS of hidden neurons. Meanwhile, a gradient algorithm is developed to update the parameters of IOA-RRBFNN. The proposed IOA-RRBFNN can be used to organize the network structure and adjust the parameters to improve its performance. Finally, several examples are presented to illustrate the effectiveness of IOA-RRBFNN. The results demonstrate that the proposed IOA-RRBFNN is more competitive in solving the nonlinear system modeling problems compared with some existing methods. [ABSTRACT FROM AUTHOR]

In this paper, an adaptive second order algorithm (ASOA) has been developed to train the fuzzy neural network (FNN) to achieve fast and robust convergence for nonlinear system modeling. Different from recent studies, this ASOA-based FNN (ASOA-FNN) has the quasi Hessian matrix and gradient vector which are accumulated as the sum of related sub matrices and vectors, respectively. Meanwhile, the learning rate of ASOA-FNN is designed to accelerate the learning speed. In addition, the convergence of the proposed ASOA-FNN has been proved both in the fixed learning rate phase and the adaptive learning rate phase. Finally, several comparisons have been realized and they have shown that the proposed ASOA-FNN has faster convergence speed and more accurate results than that of some existing methods. [ABSTRACT FROM AUTHOR]

Gradient-based algorithms are commonly used for training radial basis function neural network (RBFNN). However, it is still difficult to avoid vanishing gradient to improve the learning performance in the training process. For this reason, in this paper, an accelerated second-order learning (ASOL) algorithm is developed to train RBFNN. First, an adaptive expansion and pruning mechanism (AEPM) of gradient space, based on the integrity and orthogonality of hidden neurons, is designed. Then, the effective gradient information is constantly added to gradient space and the redundant gradient information is eliminated from gradient space. Second, with AEPM, the neurons are generated or pruned accordingly. In this way, a self-organizing RBFNN (SORBFNN) which reduces the structure complexity and improves the generalization ability is obtained. Then, the structure and parameters in the learning process can be optimized by the proposed ASOL-based SORBFNN (ASOL-SORBFNN). Third, some theoretical analyses including the efficiency of the proposed AEPM on avoiding the vanishing gradient and the stability of SORBFNN in the process of structural adjustment are given, then the successful application of the proposed ASOL-SORBFNN is guaranteed. Finally, to illustrate the advantages of the proposed ASOL-SORBFNN, several experimental studies are examined. By comparing with other existing approaches, the results show that ASOL-SORBFNN performs well in terms of both learning speed and prediction accuracy. [ABSTRACT FROM AUTHOR]