Search

A major progress in deep multilayer neural networks (DNNs) is the invention of various unsupervised pretraining methods to initialize network parameters which lead to good prediction accuracy. This paper presents the sparseness analysis on the hidden unit in the pretraining process. In particular, we use the L1 -norm to measure sparseness and provide some sufficient conditions for that pretraining leads to sparseness with respect to the popular pretraining models—such as denoising autoencoders (DAEs) and restricted Boltzmann machines (RBMs). Our experimental results demonstrate that when the sufficient conditions are satisfied, the pretraining models lead to sparseness. Our experiments also reveal that when using the sigmoid activation functions, pretraining plays an important sparseness role in DNNs with sigmoid (Dsigm), and when using the rectifier linear unit (ReLU) activation functions, pretraining becomes less effective for DNNs with ReLU (Drelu). Luckily, Drelu can reach a higher recognition accuracy than DNNs with pretraining (DAEs and RBMs), as it can capture the main benefit (such as sparseness-encouraging) of pretraining in Dsigm. However, ReLU is not adapted to the different firing rates in biological neurons, because the firing rate actually changes along with the varying membrane resistances. To address this problem, we further propose a family of rectifier piecewise linear units (RePLUs) to fit the different firing rates. The experimental results show that the performance of RePLU is better than ReLU, and is comparable with those with some pretraining techniques, such as RBMs and DAEs. [ABSTRACT FROM PUBLISHER]

The multivalued neuron with periodic activation function (MVN-P) was proposed by Aizenberg for solving classification problems. The boundaries between two distinct categories are crisply specified in MVN-P, which may result in slow convergence or being unable to converge at all in the learning process. In this paper, we propose a revised model of MVN-P based on the idea of unsharp boundaries. In this revised model, a fuzzy buffer is provided around a boundary between two distinct categories, allowing incorrect assignments with membership degree less than a threshold to be tolerated in the training phase. Genetic algorithms are applied to derive optimal values for the parameters involved in this model, alleviating the burden of setting them manually by the user. Besides, MVN-P has difficulties solving the classification problems having a large number of categories. A tree structure is developed to overcome these difficulties. Simulation results demonstrate the effectiveness of our proposed ideas. [ABSTRACT FROM AUTHOR]

This brief presents a new guaranteed H\infty performance state estimation criterion for delayed static neural networks. To facilitate the use of the slope information about activation function, the estimation error of activation function is separated into two parts for the first time. Then, a novel Lyapunov-Krasovskii functional (LKF) is constructed, which has fully captured the slope information of the activation. Based on the new LKF, a less conservative design criterion of estimator is derived to ensure the asymptotic stability of estimation error system with H\infty performance. The desired estimator gain matrices and the performance index are obtained by solving a convex optimization problem. The simulation results show that the proposed method has much better performance than the most recent results. [ABSTRACT FROM PUBLISHER]

This brief considers the guaranteed $H_\infty $ performance state estimation problem of delayed static neural networks. An Arcak-type state estimator, which is more general than the widely adopted Luenberger-type one, is chosen to tackle this issue. A delay-dependent criterion is derived under which the estimation error system is globally asymptotically stable with a prescribed $H_\infty $ performance. It is shown that the design of suitable gain matrices and the optimal performance index are accomplished by solving a convex optimization problem subject to two linear matrix inequalities. Compared with some previous results, much better performance is achieved by our approach, which is greatly benefited from introducing an additional gain matrix in the domain of activation function. An example is finally given to demonstrate the advantage of the developed result. [ABSTRACT FROM AUTHOR]