Search

Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural networks have been introduced to deal with different types of problems. However, the main goal of any neural network is to transform the non-linearly separable input data into more linearly separable abstract features using a hierarchy of layers. These layers are combinations of linear and nonlinear functions. The most popular and common non-linearity layers are activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish and Mish. In this paper, a comprehensive overview and survey is presented for AFs in neural networks for deep learning. Different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based are covered. Several characteristics of AFs such as output range, monotonicity, and smoothness are also pointed out. A performance comparison is also performed among 18 state-of-the-art AFs with different networks on different types of data. The insights of AFs are presented to benefit the researchers for doing further research and practitioners to select among different choices. The code used for experimental comparison is released at: https://github.com/shivram1987/ActivationFunctions. [ABSTRACT FROM AUTHOR]

This article investigates the coexistence of multistable equilibrium points (EPs) for n -neuron delayed quaternion-valued neural networks (DQVNNs) in which the continuous and piecewise nonlinear functions are used as the activation functions. Using the state space decomposition technique, Brouwer's fixed point theorem, and Lagrange's mean value theorem, some novel sufficient conditions have been derived to ensure the coexistence of 5 4 n EPs in which 3 4 n of them are locally stable. Furthermore, in this scientific contribution, positively invariant sets have been estimated which actually turned out to be quite challenging for the activation functions of the designed DQVNNs. Finally, some comparisons and convincing simulations with the application to associative memory of DQVNNs are given to verify the theoretical results at the end of this article. [ABSTRACT FROM AUTHOR]

• We propose fuzzy reinforced polynomial neural networks based on PNN architecture. • We propose fuzzy reinforced polynomial neurons (FRPNs) in the design of FRPNNs. • The proposed FRPNNs is a generalization of hybrid predictors (HPs). In the field of dynamic system identification and prediction, linear models (e.g., autoregressive models), nonlinear models (namely, neural networks models), and hybrid predictors (HPs) that are a hybridization of linear and nonlinear models have been proposed in the past. However, they are not completely free from limitations: they exhibit difficulties to describe high-order nonlinear relations between input and output variables. In this study, we propose fuzzy reinforced polynomial neural networks (FRPNNs), which are polynomial neural network architecture-based on fuzzy reinforced polynomial neurons (FRPNs) to overcome this limitation. The proposed FRPNs that consist of approximation part (AP) and compensation part (CP) arise as novel HPs. Here the CP for modeling nonlinear patterns can be regarded as forming the reinforced part for the AP that aims at capturing linear patterns, while AP is the linear polynomial neuron used in the conventional polynomial neural networks. In some sense, the overall FRPNNs are essentially generalized polynomial neural network architecture with novel HPs. The parameters considered in the design of the proposed fuzzy reinforced polynomial neural networks are optimized with the aid of the particle swarm optimization (PSO). The performance of FRPNNs is discussed involving time series and system identification datasets. Experimental results demonstrate that the proposed FRPNNs achieve at most the accuracy of 43.6% higher in comparison with the accuracy produced by some classical models reported in the literature. [ABSTRACT FROM AUTHOR]

As one of standard approaches to train deep neural networks, dropout has been applied to regularize large models to avoid overfitting, and the improvement in performance by dropout has been explained as avoiding co-adaptation between nodes. However, when correlations between nodes are compared after training the networks with or without dropout, one question arises if co-adaptation avoidance explains the dropout effect completely. In this paper, we propose an additional explanation of why dropout works and propose a new technique to design better activation functions. First, we show that dropout can be explained as an optimization technique to push the input towards the saturation area of nonlinear activation function by accelerating gradient information flowing even in the saturation area in backpropagation. Based on this explanation, we propose a new technique for activation functions, gradient acceleration in activation function (GAAF) , that accelerates gradients to flow even in the saturation area. Then, input to the activation function can climb onto the saturation area which makes the network more robust because the model converges on a flat region. Experiment results support our explanation of dropout and confirm that the proposed GAAF technique improves image classification performance with expected properties. [ABSTRACT FROM AUTHOR]

• Deep neural networks (DNN) with polynomials activation functions, or polynomial activation neural networks (PANN), are introduced for the first time. Nonlinear difference equation modeling is contrived. Relations with common DNN are examined by Taylor series. • Stability analysis for feed-forward and recurrent PANN's are considered under small parametrization assumption, through trajectory bounds estimation and linear approximation. • The coverage back-propagation parameterization of PANN's is suggested, and pre-training algorithms with polynomial activation optimization or without are worked out, respectively. The pre-training algorithms are computationally efficient. • As application of PANN, nonlinear function fitting is numerically investigated. The results show high capability of PANN in functions fitting or features retrieving. In this paper, firstly, a class of deep (or multi-layer) neural networks with polynomial activation functions (or polynomial activation neural networks: PANN) are created, while feed-forward and recurrent architectures and their nonlinear difference modeling are explicated. PANN's relationships with conventional deep neural networks with sigmoid activation functions are discussed briefly by means of Taylor series. Secondly, numerical stability and stabilization of PANN's are examined, and the stability conditions are derived with bounded-state trajectory inequalities and small-state linear approximation, under small parametrization assumption; stability analysis implication coincides with what we already learnt about neural network pre-training. Thirdly, based on what we term the coverage back-propagation parametrization, pre-training algorithms for PANN with or without activation functions optimization are constructed; particularly, activation function optimization is a new concept of this study, which brings us with more learning flexibility in general neural networks. Finally, nonlinear function fitting is numerically illustrated to show application of PANN, which reveals high generalization capability of linear parameter-varying neural algorithms. [ABSTRACT FROM AUTHOR]

This paper studies the controller design for an autonomous underwater vehicle (AUV) with the target tracking task. Considering the uncertainty the nonlinear longitudinal model, a sliding mode controller is designed. Meanwhile the neural networks (NNs) are used to approximate the unknown nonlinear function in the model. To improve the NNs learning rapidity, the prediction error which reflect the learning performance is constructed, further the updating law is designed utilizing the composite learning technique. The system stability is guaranteed through the Lyapunov approach. The simulation results verify that the designed method could force the AUV to track the target until rendezvous, and the model uncertainty is addressed better via the composite learning algorithm. [ABSTRACT FROM AUTHOR]

Nonlinear activation functions play an important role in zeroing neural network (ZNN), and it has be proved that ZNN can achieve finite-time convergence when the sign-bi-power (SBP) activation function is explored. However, its upper bound depends on initial states of ZNN seriously, which will restrict some practical applications since the knowledge of initial conditions is generally unavailable in advance. Besides, SBP activation function does not make ZNN reject external disturbances simultaneously. To address the above two issues encountered by ZNN, by suggesting a new nonlinear activation function, a robust and fixed-time zeroing neural dynamics (RaFT-ZND) model is proposed and analyzed for time-variant nonlinear equation (TVNE). As compared to the previous ZNN model with SBP activation function, the RaFT-ZND model not only converges to the theoretical solution of TVNE within a fixed time, but also rejects external disturbances to show good robustness. In addition, the upper bound of the fixed-time convergence is theoretically computed in mathematics, which is independent of initial states of the RaFT-ZND model. At last, computer simulations are conducted under external disturbances, and comparative results demonstrate the effectiveness, robustness, and advantage of the RaFT-ZND model for solving TVNE. [ABSTRACT FROM AUTHOR]

Global asymptotic stability is an important issue for wide applications of recurrent neural networks with time-varying delays. The Lyapunov–Krasovskii functional method is a powerful tool to check the global asymptotic stability of a delayed recurrent neural network. When the Lyapunov–Krasovskii functional method is employed, three steps are necessary in order to derive a global asymptotic stability criterion: (i) constructing a Lyapunov–Krasovskii functional, (ii) estimating the derivative of the Lyapunov–Krasovskii functional, and (iii) formulating a global asymptotic stability criterion. This paper provides an overview of recent developments in each step with insightful understanding. In the first step, some existing Lyapunov–Krasovskii functionals for stability of delayed recurrent neural networks are anatomized. In the second step, a free-weighting matrix approach, an integral inequality approach and its recent developments, reciprocally convex inequalities and S -procedure are analyzed in detail. In the third step, linear convex and quadratic convex approaches, together with the refinement of allowable delay sets are reviewed. Finally, some challenging issues are presented to guide the future research. [ABSTRACT FROM AUTHOR]

This paper addresses the adaptive prescribed performance tracking control problem for a class of nonlinear time-delay systems with actuator fault, input quantization, unknown control directions and disturbances. Neural networks (NNs) are employed to approximate the unknown nonlinear functions and the Nussbaum function is used to deal with the unknown control directions. Then, a novel adaptive prescribed performance controller is designed to reduce the effects of actuator fault, input quantization, NNs approximation errors and disturbances. Compared with the existing results, a new error transformation method is presented, and the knowledge of the quantization parameters and the control directions are unknown in the control design. Furthermore, the proposed control scheme can guarantee the semi-global boundedness of all the closed-loop signals and the prescribed time-varying tracking performance. Finally, simulation results are given to demonstrate the effectiveness of the proposed control method. [ABSTRACT FROM AUTHOR]

In this paper, the problem of fault-tolerant control (FTC) is investigated for a class of multivariable stochastic nonlinear systems in the strict-feedback form. The considered systems possess unknown nonlinear functions, unmeasured states, virtual control coefficients and actuator faults (bias and gain faults). Neural networks (NNs) are adopted to approximate the unknown nonlinear functions. Then, a state observer is constructed to solve the problem of unmeasured states. The problem of “explosion of complexity” is solved by using dynamic surface control (DSC) method. Based on the combination of Nussbaum gain function with Lyapunov function stability theory, an adaptive NNs output feedback FTC method is developed in the frame of adaptive backstepping design technique. It is shown that all signals in the closed-loop system are proved to be bounded in probability, and the system output can follow the given reference signal well. The effectiveness of the proposed approach is verified by a simulation example. [ABSTRACT FROM AUTHOR]

Recently, nonlinear expectile regression becomes popular because it can not only explore nonlinear relationships among variables, but also describe the complete distribution of a response variable conditional on covariates information. In contrast, the traditional nonlinear expectile regression mainly confronts two shortcomings. First, it is difficult to select an appropriate form of nonlinear function. Second, it ignores the interaction effects among covariates. In this paper, we develop a new expectile regression neural network (ERNN) model by adding a neural network structure to expectile regression approach. The proposed ERNN model is flexible and can be used to explore potential nonlinear effects of covariates on expectiles of the response. The ERNN model can be easily estimated through standard gradient based optimization algorithms and output conditional expectile functions directly. The advantage of ERNN model is illustrated by Monte Carlo simulation studies. The numerical results show that the ERNN model outperforms the conventional expectile regression and support vector machine models in terms of predictive ability with both in-sample and out-of-sample test. We also apply the ERNN model to the predictions of concrete compressive strength and housing price. It turns out the marginal contribution of each predictor to the conditional expectile of a response is useful for decision-making. [ABSTRACT FROM AUTHOR]

This paper addresses the trajectory-tracking control problem of mobile robot systems with nonholonomic constraints, in the presence of time-varying parametric uncertainties and external disturbances. This necessitates an accurate controller to meet as much control objectives, as possible. Therefore, this paper deals with an artificial intelligence control technique to design a robust controller to meet the control objectives. The design of the intelligent controller is based on the optimal control theory, the adaptive neural network system, and the robust control technique. The trajectory-tracking problem is solved using the optimal control methodology. Since the nonholonomic wheeled mobile robot is strongly nonlinear, the neural network system is applied to approximate the nonlinear function in the optimal control law. The robust controller, for his part, is then applied to adaptively estimate an unknown upper bound of the time-varying parametric uncertainties, external disturbances and approximation error of the neural network system. The stability of the closed-loop robot system is proven using the optimal control theory and Lyapunov stability analysis. The results of the simulation studies on three typical nonholonomic mobile robots are provided to demonstrate the effectiveness of the proposed controller. In addition, a comparative study with a recent robust adaptive controller shows that our proposed intelligent controller gives better results, in the sense that the output trajectory converges to the steady state faster with smaller tracking error. [ABSTRACT FROM AUTHOR]

In this paper, adaptive neural control (ANC) is investigated for a class of strict-feedback nonlinear stochastic systems with unknown parameters, unknown nonlinear functions and stochastic disturbances. The new controller of adaptive neural network with state feedback is presented by using a universal approximation of radial basis function neural network and backstepping. An adaptive neural network state-feedback controller is designed by constructing a suitable Lyapunov function. Adaptive bounding design technique is used to deal with the unknown nonlinear functions and unknown parameters. It is shown that the global asymptotically stable in probability can be achieved for the closed-loop system. The simulation results are presented to demonstrate the effectiveness of the proposed control strategy in the presence of unknown parameters, unknown nonlinear functions and stochastic disturbances. [ABSTRACT FROM AUTHOR]

This paper investigates the adaptive actuator fault compensation control for a class of uncertain multi-input single-out discrete-time systems with triangular forms. The considered actuator faults contain both loss of effectiveness and lock-in-place. With the help of radial basis function neural networks to approximate the unknown nonlinear functions, an adaptive fault tolerant control scheme is designed. Compared with some existing methods, one of features of the proposed method is that we introduce the backstepping technique to achieve the fault-tolerant control task. It is proved that the proposed control approach can guarantee that all the signals of the closed-loop systems are uniformly ultimately bounded and that the output can track a reference signal in the presence of the actuator faults. Finally, three simulation results are provided to confirm the effectiveness of the fault-tolerant control approach. [ABSTRACT FROM AUTHOR]

This paper proposes a novel robust adaptive consensus tracking control approach for a class of nonlinear multi-agent systems with modeling uncertainties and external disturbances. Radial Basis Function Neural Networks (RBFNNs) are used to approximate the unknown nonlinear function of agent׳s dynamic. Compared with existing NN consensus algorithms of nonlinear multi-agent systems, the proposed consensus control method only needs a small number of adjustable parameters, thus the online computation burden is greatly alleviated. In addition, by online updating the estimation of the NN approximation errors, the proposed consensus control approach can enhance the system robustness against modeling uncertainties. It is proven that all the signals of the multi-agent system are uniformly bounded and the consensus tracking errors converge to a small neighborhood of zero. A simulation is carried out to demonstrate the effectiveness of the proposed control method. [ABSTRACT FROM AUTHOR]

In this paper, a new fuzzy neural network model with correlated fuzzy rules (CFNN) based on the Levenberg–Marquardt ( LM ) optimization method is proposed. The proposed method is a new fuzzy network structure that is presented to approximate nonlinear functions especially the functions with high correlation between input variables with less number of fuzzy rules. A multivariable Gaussian fuzzy membership function is introduced that can consider the correlation between input variables and consequently it can model non-separable relations for interactive variables. The LM optimization method is used to learn parameters of both premise and consequent parts of the fuzzy rules. The suggested algorithm is successfully applied to seven tested examples including static function approximation, time-series prediction, non-linear dynamic system identification and a real-world complex regression problem. According to test observations; it can approximate nonlinear functions better than the other past algorithms with more compact structure and less number of fuzzy rules. [ABSTRACT FROM AUTHOR]

In this paper, a novel algorithm is developed for identifying Hammerstein model. The static nonlinear function is characterized by function link artificial neural network (FLANN) and the linear dynamic subsystem by an ARMA model. The utilization of FLANN can not only result in a simple and effective representation of static nonlinearity but also simplify the learning algorithm. A two-step procedure is adopted to identify Hammerstein model by using a specially designed input signal, which separates the identification of linear part from that of nonlinear part. Levenberg-Marquart algorithm is used to learn the weights of FLANN. Simulation examples demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]