Search

We are guided by the fact that zeroing neural networks (ZNN) are proven tool in online solving the time-varying (TV) matrix Moore–Penrose (M–P) inverse. This paper focuses on online computing TV full-row rank or full-column rank matrix M–P inverse using a novel ZNN model with an optimized activation function (AF) and improved error function (Zhangian). ZNN dynamical systems accelerated by the optimized class of AFs converge in a finite-time to the TV theoretical M–P inverse. The upper bounds of the estimated convergence time are obtained analytically using the Lyapunov stability theory. The simulation experiments support the theoretical analysis and demonstrate the effectiveness of the proposed ZNN dynamics. [ABSTRACT FROM AUTHOR]

This paper proposes two complex-valued zeroing neural network (Cv-ZNN) models for solving dynamic complex time-variant linear equations. The models involve two complex-valued nonlinear processing methods and adapt two real-valued activation functions. The convergence and robustness of the two Cv-ZNN models are analyzed comprehensively. Firstly, the convergence discussion shows that the corresponding stable error can rapidly converge to zero in finite time, with the upper limit of constriction time calculated. The upper limit of stable error is successfully obtained when the model implementation error is injected into the Cv-ZNN models, which displays their superior robustness. Besides, the results of numerical simulations reveal that the Cv-ZNN models effectively address the time-variant linear system of equations over complex field, even in noisy environments. The simulation results also demonstrate that two novel nonlinear activation functions perform much better than the Linear, Power, Bipolar Sigmoid Power-Sigmoid and Sign-bi-power activation functions. [ABSTRACT FROM AUTHOR]

In this paper, we firstly extend the Fourier neural operator (FNO) to discovery the mapping between two infinite-dimensional function spaces, where one is the fractional-order index space { ϵ | ϵ ∈ (0 , 1) } in the fractional integrable nonlinear wave equations while another denotes the soliton solution in the spatio-temporal function space. In other words, once the FNO network is trained, for any given ϵ ∈ (0 , 1) , the corresponding soliton solution can be quickly obtained. To be specific, the soliton solutions are learned for the fractional nonlinear Schrödinger (fNLS), fractional Korteweg–de Vries (fKdV), fractional modified Korteweg–de Vries (fmKdV) and fractional sine-Gordon (fsineG) equations. The FNO architecture is utilized to learn the soliton mappings of the above four equations. The data-driven solitons are also compared with exact solutions to illustrate the powerful approximation capability of the FNO. Moreover, we study the influences of several critical factors (e.g., activation functions containing Relu (x) , Sigmoid (x) , Swish (x) and the new one x tanh (x) , channels of fully connected layer) on the performance of the FNO algorithm. As a result, we find that the x tanh (x) and Swish (x) functions perform better than the Relu (x) and Sigmoid (x) functions in the FNO, and the FNO network with a more-channel fully-connected layer performs better as we expect. These results obtained in this paper will be useful to further understand the neural networks for fractional integrable nonlinear wave equations and the mappings between two infinite-dimensional function spaces. • Data-driven soliton mappings are investigated for integrable fractional nonlinear wave equations via FNO. • Data-driven solitons are also compared to exact solutions to illustrate the accuracies. • The influences of activation functions and channels of fully connected layer are analyzed on the performance of the FNO. [ABSTRACT FROM AUTHOR]

Machine Learning as a Service (MLaaS) provides clients with well-trained neural networks for predicting private data. Conventional prediction processes of MLaaS require clients to send sensitive inputs to the server, or proprietary models must be stored on the client-side device. The former reveals client privacy, while the latter harms the interests of model providers. Existing works on privacy-preserving MLaaS introduce cryptographic primitives to allow two parties to perform neural network inference without revealing either party's data. However, nonlinear activation functions bring high computational overhead and response delays to the inference process of these schemes. In this paper, we analyze the mechanism by which activation functions enhance model expressivity, and design an activation function S - cos that is friendly to secure neural network inference. Our proposed S - cos can be re-parameterized into a linear layer during the inference phase. Further, we propose an inference-time linear model called Beyond Linear Neural Network (B-LNN) equipped with S - cos , which exhibits promising performance on several benchmark datasets. • We argue that the activation functions introduce an inductive bias to the learning process, and the design toward cosine similarity makes ReLU and its variants superior to value-oriented polynomial activation functions. We further propose a novel cosine similarity-oriented activation function, called S - cos. • We analyze the advantages of deep features over shallow ones and introduce a random feature extraction module to improve the performance of models containing a single activation layer. • We design a Beyond Linear Neural Network (B-LNN), which is equipped with our proposed S - cos and random feature extraction module. • We design a secure neural network inference framework with Arithmetic Secret Sharing (A-SS) by taking advantage of the inference-time linearity of B-LNN. [ABSTRACT FROM AUTHOR]

A major progress in deep multilayer neural networks (DNNs) is the invention of various unsupervised pretraining methods to initialize network parameters which lead to good prediction accuracy. This paper presents the sparseness analysis on the hidden unit in the pretraining process. In particular, we use the L1 -norm to measure sparseness and provide some sufficient conditions for that pretraining leads to sparseness with respect to the popular pretraining models—such as denoising autoencoders (DAEs) and restricted Boltzmann machines (RBMs). Our experimental results demonstrate that when the sufficient conditions are satisfied, the pretraining models lead to sparseness. Our experiments also reveal that when using the sigmoid activation functions, pretraining plays an important sparseness role in DNNs with sigmoid (Dsigm), and when using the rectifier linear unit (ReLU) activation functions, pretraining becomes less effective for DNNs with ReLU (Drelu). Luckily, Drelu can reach a higher recognition accuracy than DNNs with pretraining (DAEs and RBMs), as it can capture the main benefit (such as sparseness-encouraging) of pretraining in Dsigm. However, ReLU is not adapted to the different firing rates in biological neurons, because the firing rate actually changes along with the varying membrane resistances. To address this problem, we further propose a family of rectifier piecewise linear units (RePLUs) to fit the different firing rates. The experimental results show that the performance of RePLU is better than ReLU, and is comparable with those with some pretraining techniques, such as RBMs and DAEs. [ABSTRACT FROM PUBLISHER]

Electromagnetic induction current is generated between the adjacent neurons in neural network caused by the existence of membrane potential difference. Memristor is the fourth fundamental electric element, which can mimic the behavior of neural synapses and simulate the electromagnetic induction effect. In this paper, in order to simplify practical implementation, the commonly used hyperbolic tangent activation function of Hopfield neural network (HNN) is replaced by a piecewise-linear function, and a simple bi-neuron-based memristor synapse-coupled HNN is proposed. Theoretical analysis and numerical simulation results illustrate that the memsirstive HNN possesses five equilibria including one unstable saddle-focus, two unstable saddle points, and two stable node points (or node-foci). Local attraction basins and phase plane plots show that the memristive HNN model behaves multistability of coexisting chaos, periodic limit cycles, and stable point attractors. Various system dynamical behaviors affected by parameters of the piecewise-linear activation function, memristor coupling strength, and initial conditions of the neurons are investigated by numerical simulations. Furthermore, an analog circuit of the memristive HNN model is simply designed, which is easier for hardware implementation as the piecewise-linear activation function can be implemented by simple op-amp-based module. Finally, the experimental results verify the correctness of the design and analyses. • A memristor synapse-coupled Hopfield neural network with piecewise-linear activation function is constructed. • Various coexisting chaos, periodic attractors, and stable point attractors are coined. • Hardware simplification of the memristive Hopfield neural network is implemented. [ABSTRACT FROM AUTHOR]

Different from gradient-based neural networks, a special kind of recurrent neural network (RNN) has recently been proposed by Zhang et al. for online matrix inversion. Such an RNN is designed based on a matrix-valued error function instead of a scalar-valued error function. In addition, it was depicted in an implicit dynamics instead of an explicit dynamics. In this paper, we develop and investigate a discrete-time model of Zhang neural network (termed as such and abbreviated as ZNN for presentation convenience), which is depicted by a system of difference equations. Comparing with Newton iteration for matrix inversion, we find that the discrete-time ZNN model incorporates Newton iteration as its special case. Noticing this relation, we perform numerical comparisons on different situations of using ZNN and Newton iteration for matrix inversion. Different kinds of activation functions and different step-size values are examined for superior convergence and better stability of ZNN. Numerical examples demonstrate the efficacy of both ZNN and Newton iteration for online matrix inversion. [ABSTRACT FROM AUTHOR]

Option pricing in the financial derivatives market is worth studying. When the volatility and the rate of return are not fixed in Blacks-Scholes (BS) equation, the analytical form of the solution is difficult to be obtained. In this paper, we propose deep learning based on blocked residual connection method (DLBR) for the numerical solving of BS equations. The validity period and inventory price are the inputs of the network, while the option price is the solution. Neurons on the multiple hidden layers are processed in blocks and each block is connected to the residuals. The boundary conditions are constructed as a part of the numerical solution and the loss function is minimized to obtain the optimal network parameters. The DLBR method has strong applicability and overcomes the shortcomings of extreme learning machines. Three experiment calculated the numerical solution of the BS equation about European options and promoted the option pricing model. Compared with the existing algorithms such as finite element method and physics-informed neural networks (PINN) method, the obtained numerical solution has higher precision and smaller error, which shows the feasibility and superiority of the proposed method. [ABSTRACT FROM AUTHOR]

The multivalued neuron with periodic activation function (MVN-P) was proposed by Aizenberg for solving classification problems. The boundaries between two distinct categories are crisply specified in MVN-P, which may result in slow convergence or being unable to converge at all in the learning process. In this paper, we propose a revised model of MVN-P based on the idea of unsharp boundaries. In this revised model, a fuzzy buffer is provided around a boundary between two distinct categories, allowing incorrect assignments with membership degree less than a threshold to be tolerated in the training phase. Genetic algorithms are applied to derive optimal values for the parameters involved in this model, alleviating the burden of setting them manually by the user. Besides, MVN-P has difficulties solving the classification problems having a large number of categories. A tree structure is developed to overcome these difficulties. Simulation results demonstrate the effectiveness of our proposed ideas. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, the multistability is studied for two-dimensional neural networks with multilevel activation functions. And it is showed that the system has n 2 isolated equilibrium points which are locally exponentially stable, where the activation function has n segments. Furthermore, evoked by periodic external input, n 2 periodic orbits which are locally exponentially attractive, can be found. And these results are extended to k-neuron networks, which is really enlarge the capacity of the associative memories. Examples and simulation results are used to illustrate the theory. [Copyright &y& Elsevier]

In this paper, we propose a novel differentiable activation function for convolutional neural networks. This function is proposed using an orthogonal Legendre polynomial based linear-in-parameter model. First, we present mathematical modeling for max pooling operation in convolutional neural networks and derive the relationship between max pooling and average pooling. The representation of max pooling using signal processing elements is presented. Further, we study approximations of various nonlinearities using the proposed linear-in-parameter model. The proposed activation function can be used as an alternative in place of existing activation functions in convolutional neural networks. Finally, the effectiveness of the proposed activation function is shown with empirical evaluations on benchmark image classification datasets using convolutional neural networks. [ABSTRACT FROM AUTHOR]