Search

In the present paper, we examine and analyze main paradigms of learning of multilayer neural networks starting with a single layer perceptron and ending with deep neural networks, which are considered regarded as a breakthrough in the field of the intelligent data processing. The baselessness of some ideas about the capacity of multilayer neural networks is shown and transition to deep neural networks is justified. We discuss the principal learning models of deep neural networks based on the restricted Boltzmann machine (RBM), an autoassociative approach and a stochastic gradient method with a Rectified Linear Unit (ReLU) activation function of neural elements.

This paper focuses on a new kind of artificial neural networks - the Z-transform artificial neural networks (ZTANNs). It is proposed to use the Z-transform instead of ordinary weights and a linear activation function of an artificial neuron. This extension allows to use artificial neural networks in new areas. The ordinary description of artificial neural networks is a special case of the description proposed in this paper. It also contains a description of the use of the ZTANN for automatic identification of objects in digital control system.

This paper investigates the application of a new form of neuron activation functions that are based on the fuzzy membership functions derived from the theory of fuzzy systems. On the basis of the results regarding neuron models with fuzzy activation functions, we created the models of fuzzy-neural networks. These fuzzy-neural network models differ from conventional networks that employ the fuzzy inference systems using the methods of neural networks. While conventional fuzzy-neural networks belong to the first type, fuzzy-neural networks proposed here are defined as the second-type models. The simulation results show that the proposed second-type model can successfully solve the problem of the property prediction for time – dependent signals. Neural networks with fuzzy impulse activation functions can be widely applied in many fields of science, technology and mechanical engineering to solve the problems of classification, prediction, approximation, etc.

Fixed-point optimization of deep neural networks plays an important role in hardware based design and low-power implementations. Many deep neural networks show fairly good performance even with 2- or 3-bit precision when quantized weights are fine-tuned by retraining. We propose an improved fixedpoint optimization algorithm that estimates the quantization step size dynamically during the retraining. In addition, a gradual quantization scheme is also tested, which sequentially applies fixed-point optimizations from high- to low-precision. The experiments are conducted for feed-forward deep neural networks (FFDNNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs)., Comment: This paper is accepted in ICASSP 2017

In order to study the approximation performance of general regression neural networks, the structure and algorithm of general regression neural networks are first introduced. Then general regression neural networks and back propagation neural networks improved by Levenberg-Marquardt algorithm are established through programming using MATLAB language. A certain nonlinear function is taken as an example to be approximated by the two kinds of neural networks. The simulation results indicate that compared with back propagation neural networks, general regression neural networks has better approximation precision and faster convergence speed, which means it has much better approximation ability than back propagation neural networks. Therefore, for more complex function approximation, general regression neural networks is recommended. It can reduce the complexity of neural networks and it is also easier to design.

BP neural network is a kind of widely used feed-forward network. However its innate shortcomings are gradually giving rise to the study of other networks. Currently one of the research focuses in the area of feed-forward networks is radial basis function neural network. To test the radial basis function neural network for nonlinear function approximation capability, this paper first introduces the theories of RBF networks, as well as the structure, function approximation and learning algorithm of radial basis function neural network. Then a simulation test is carried out to compare BPNN and RBFNN. The simulation results indicate that RBFNN is simpler in structure, faster in speed and better in approximation performance. That is to say RBFNN is superior to BPNN in many aspects. But when solving the same problem, the structure of radial basis networks is more complicated than that of BP neural networks. Keywords: Radial basis function; Neural network; Function approximation; Simulation; MATLAB

In this work, some ubiquitous neural networks are applied to model the landscape of a known problem function approximation. The performance of the various neural networks is analyzed and validated via some well-known benchmark problems as target functions, such as Sphere, Rastrigin, and Griewank functions. The experimental results show that among the three neural networks tested, Radial Basis Function (RBF) neural network is superior in terms of speed and accuracy for function approximation in comparison with Back Propagation (BP) and Generalized Regression Neural Network (GRNN).

We present a theoretical analysis of singular points of artificial deep neural networks, resulting in providing deep neural network models having no critical points introduced by a hierarchical structure. It is considered that such deep neural network models have good nature for gradient-based optimization. First, we show that there exist a large number of critical points introduced by a hierarchical structure in deep neural networks as straight lines, depending on the number of hidden layers and the number of hidden neurons. Second, we derive a sufficient condition for deep neural networks having no critical points introduced by a hierarchical structure, which can be applied to general deep neural networks. It is also shown that the existence of critical points introduced by a hierarchical structure is determined by the rank and the regularity of weight matrices for a specific class of deep neural networks. Finally, two kinds of implementation methods of the sufficient conditions to have no critical points are provided. One is a learning algorithm that can avoid critical points introduced by the hierarchical structure during learning (called avoidant learning algorithm). The other is a neural network that does not have some critical points introduced by the hierarchical structure as an inherent property (called avoidant neural network).

Feedforward neural networks are neural networks with (possibly) multiple layers of neurons such that each layer is fully connected to the next one. They have been widely studied in the past partially due to their universal approximation capabilities and empirical effectiveness on a variety of application domains for both regression and classification tasks. In this paper, we provide an overview on feedforward neural networks, focusing on weight initialization methods.

Complex neural networks, such as those found in the human brain, are able to very accurately discriminate and classify external stimuli. Some of their topological and computational properties have been extracted and used to great effect by the artificial intelligence community. However, even our best simulated neural networks are very pale abstractions of reality, partly because (in general) they fail to account for the temporal dynamics and recurrence inherent in natural neural networks, and instead employ feed-forward architecture and discrete, simultaneous activity. In this paper we begin to develop an intuitive, geometric framework to explore the ways in which different inputs could be discriminated in recurrent linear dynamical networks, with the eventual goal of being able to facilitate a transition to more realistic and effective artificial networks. We first establish a useful, closed-form measure on the space of minimum-energy inputs to a linear system, which allows an elucidation of how discrepancies between inputs impact output trajectories in the state space. We characterize, to an extent, the relationship between input and output difference as it relates to system dynamics as manifest in the geometry of the reachable output space. We draw from this characterization principles which may be employed in the design of dynamic, recurrent artificial networks for input discrimination.

The rapid development of computing machines led to renewed interest in deep neural networks. For years it is known that they have a great possibilities, but to use them new training algorithms are required. The paper shows benefits for deep neural networks usage by analysis of the Fourier series approximation of the activation function for shallow and deep neural network architectures. The proposed approach has been confirmed by experiments.

We propose to represent the shape of an organ using a neural network classifier. The shape is represented by a function learned by a neural network. Radial Basis Function (RBF) is used as the activation function for each perceptron. The learned implicit function is a combination of radial basis functions, which can represent complex shapes. The organ shape representation is learned using classification methods. Our testing results show that the neural network shape provides the best representation accuracy. The use of RBF provides a rotation, translation and scaling invariant feature to represent the shape. Experiments show that our method can accurately represent the organ shape.

WOS: 000374578800011, Multiplicative neuron model-based artificial neural networks are one of the artificial neural network types which have been proposed recently and have produced successful forecasting results. Sigmoid activation function was used in multiplicative neuron model-based artificial neural networks in the previous studies. Although artificial neural networks which involve the use of radial basis activation function produce more successful forecasting results, Gaussian activation function has not been used for multiplicative neuron model yet. In this study, rather than using a sigmoid activation function, Gaussian activation function was used in multiplicative neuron model artificial neural network. The weights of artificial neural network and parameters of activation functions were optimized by guaranteed convergence particle swarm optimization. Two major contributions of this study are as follows: the use of Gaussian activation function in multiplicative neuron model for the first time and the optimizing of central and propagation parameters of activation function with the weights of artificial neural network in a single optimization process. The superior forecasting performance of the proposed Gaussian activation function-based multiplicative neuron model artificial neural network was proved by applying it to real-life time series.

Due to its ability to solve nonlinear problems, Artificial Neural Network (ANN) could be applied in several areas of life. However, defining its architecture for solving a given problem is not formalized and remains an open research problem. On the other hand the complexity of such a technique due to its “black box” aspect, makes its interpretation more tedious. Since optimal factors completely cover the data and therefore give an explanation to these data, we propose in this paper to build feedforward ANNs using the optimal factors obtained from the boolean context representing a data. We show through experiments and comparisons on the use datasets that this approach provides relatively better results than those existing in the literature.

An information processing algorithm which simulates the way biological neural systems process information, and one of the most popular machine learning algorithms, ANN (Artificial Neural Network) has been extensively used for Data Mining, which extracts hidden patterns and valuable information from large databases. Data Mining is commonly used in a wide range of practices such as accounting, marketing, fraud detection, scientific discovery, etc. This paper introduces a new adaptive Higher Order Neural Network (HONN) model and applies it in data mining tasks such as determining liver disorders and predicting breast cancer recurrences. A new activation function which is a combination of sine and sigmoid functions is used as the neuron activation function for the new HONN model. There are free parameters in the new activation function. The paper compares the new HONN model against a Multi-Layer Perceptron (MLP) with the sigmoid activation function, an RBF Neural Network with the gaussian activation function, and a Recurrent Neural Network (RNN) with the sigmoid activation function. Experimental results show that the new HONN model offers several advantages over conventional ANN models such as improved generalisation capabilities as well as abilities in handling missing values in a dataset.

A new structure of wavelet neural networks (WNN) with extreme learning machine (ELM) is introduced in this paper. In the proposed wavelet neural networks, composite functions are applied at the hidden nodes and the learning is done using ELM. The input information is first processed by wavelet functions and then passed through a type of bounded nonconstant piecewise continuous activation functions g:R->R. A selection method that takes into account the domain of input space where the wavelets are not zero is used to initialize the translation and dilation parameters. The formed wavelet neural network is then trained with the computationally efficient ELM algorithm. Experimental results on the regression of some nonlinear functions and real-world data, the prediction of a chaotic signal and classifications on serval benchmark real-world data sets show that the proposed neural networks can achieve better performances in most cases than some relevant neural networks and learn much faster than neural networks training with the traditional back-propagation (BP) algorithm.

This paper discusses a class of discrete time recurrent neural networks with multivalued neurons (MVN), which have complex-valued weights and an activation function defined as a function of the argument of a weighted sum. Complementing state-of-the-art of such networks, our research focuses on the convergence analysis of the networks in synchronous update mode. Two related theorems are presented and simulation results are used to illustrate the theory.

This article presents a neural network model that permits to build a conceptual hierarchy to approximate functions over a given interval. Bio-inspired axo-axonic connections are used. In these connections the signal weight between two neurons is computed by the output of other neuron. Such arquitecture can generate polynomial expressions with lineal activation functions. This network can approximate any pattern set with a polynomial equation. This neural system classifies an input pattern as an element belonging to a category that the system has, until an exhaustive classification is obtained. The proposed model is not a hierarchy of neural networks, it establishes relationships among all the different neural networks in order to propagate the activation. Each neural network is in charge of the input pattern recognition to any prototyped category, and also in charge of transmitting the activation to other neural networks to be able to continue with the approximation. [Article copies are available for purchase from InfoSci-on-Demand.com]

Function simulation, which is called virtual reality too, is popularly applied to solve uncertain problems. Good performance of hidden layers and perfect capability of function simulation make artificial neural networks one of the best choices to simulate functions with form unknown. Inputs and outputs were used to train the structure of the artificial neural network to make the outputs of network vary with the given inputs and keep consistent with the original data within tolerance. However, we couldn’t get expected results by using samples of a simple two-variable-model for the cause of dimensional difference. The way of artificial neural networks to fit functions, which uses “multi-dimensional surface” of high dimension to fit “multi-dimensional line” of low dimension, was proved; the conclusion that good effects of fitting don’t mean good function modeling when a dimensional difference exists was provided, and a suggestion of “surface collecting” in practical engineering application was proposed when collecting useful data.

Radial basis function neural networks are the most widely used networks due to their rapid training, generality, and simplicity. The nature of these networks necessitates some types of errors which can never be removed by traditional training algorithms. This paper is an attempt to introduce the natural error sources of neural networks such as bias error, iteration-restricted error, and Gibbs error. Moreover, a new method is introduced, called post-training, to reduce these errors as far as desired.

Neural networks have become very useful tools for input-output knowledge discovery. However, some of the most powerful schemes require very complex machines and, thus, a large amount of calculation. This paper presents a general technique to reduce the computational burden associated with the operational phase of most neural networks that calculate their output as a weighted sum of terms, which comprises a wide variety of schemes, such as Multi-Net or Radial Basis Function networks. Basically, the idea consists on sequentially evaluating the sum terms, using a series of thresholds which are associated with the confidence that a partial output will coincide with the overall network classification criterion. Furthermore, we design some procedures for conveniently sorting out the network units, so that the most important ones are evaluated first. The possibilities of this strategy are illustrated with some experiments on a benchmark of binary classification problems, using RealAdaboost and RBF networks, which show that important computational savings can be achieved without significant degradation in terms of recognition accuracy.

This paper presents sufficient conditions for implementation of the Boolean functions by a single neural element with a threshold activation function. Based on these conditions, algorithms of neural elements synthesis are developed. They can be successfully used for the construction of neural networks designed for representation and recognition of discrete signals and images. Further suggested methods for the synthesis of a single neural element and for the construction of logical circuits out of them can be used in the cases where the use of other methods to solve these problems is difficult, because in these methods the number of operations depends exponentially on the number of inputs of neural elements.

This paper introduces Lie algebra-valued Hopfield neural networks, for which the states, outputs, weights and thresholds are all from a Lie algebra. This type of networks represents an alternative generalization of the real-valued neural networks besides the complex-, hyperbolic-, quaternion-, and Clifford-valued neural networks that have been intensively studied over the last few years. The dynamics of these networks from the energy function point of view is studied by giving the expression of such a function and proving that it is indeed an energy function for the proposed network.

Neural networks (NN) are architectures and algorithms for machine learning. They are quite powerful for tasks like classification, clustering, and pattern recognition. Large neural networks can be considered a universal function that can approximate any function, and hence are effective at learning from the training data, not only the useful information but also the noise in the training data. However, as the number of neurons and the number of hidden layers grow, the number of connections in the network increases exponentially, and the over fitting problem becomes more severe biased towards noise. Various methods have been proposed to address this problem such as AutoEncoder, Dropout, DropConnect, and Factored Mean training. In this paper, we propose a denoising autoencoder approach using a modified Elliott activation function and a cost function that favors sparsity in the input data. Preliminary experiments using the modified algorithm on several real data sets showed that the proposed approach performed well.

This paper introduces Lie algebra-valued feedforward neural networks, for which the inputs, outputs, weights and biases are all from a Lie algebra. This type of networks represents an alternative generalization of the real-valued neural networks besides the complex-, hyperbolic-, quaternion-, and Clifford-valued neural networks that have been intensively studied over the last few years. The full deduction of the gradient descent algorithm for training such networks is presented. The proposed networks are tested on two synthetic function approximation problems and on geometric transformations, the results being promising for the future of Lie algebra-valued neural networks.

A novel algorithm, bases on the adaptive resonance theory (ART), is proposed to design the structure of radial basis function (RBF) neural networks in this paper. Based on the concept of “similarity”, this proposed ART-like algorithm can be utilized to construct the RBF neural network. The number of the hidden nodes is able to be adjusted in the learning process. Meanwhile, the activity of each hidden node can be owned through the initial width design to make the structure compact. Finally, three examples are employed to test the effectiveness of the proposed ART-like RBF (ART-RBF) neural network. The results indicate that this ART-RBF neural network has better comparable generalization performance with compact structure and fast training time.

We consider a model of so-called hybrid recurrent neural networks composed with Boolean input and output cells as well as sigmoid internal cells. When subjected to some infinite binary input stream, the Boolean output cells necessarily exhibit some attractor dynamics, which is assumed to be of two possible kinds, namely either meaningful or spurious, and which underlies the arising of spatiotemporal patterns of output discharges. In this context, we show that rational-weighted neural networks are computationally equivalent to deterministic Muller Turing machines, whereas all other models of real-weighted or evolving neural networks are equivalent to each other, and strictly more powerful than deterministic Muller Turing machines. In this precise sense, the analog and evolving neural networks are super-Turing. We further provide some precise mathematical characterization of the expressive powers of all these neural models. These results constitute a generalization to the current computational context of those obtained in the cases of classical as well as interactive computations. They support the idea that recurrent neural networks represent a natural model of computation beyond the Turing limits.

Neural Network is said to emulate the brain, though, its processing is not quite how the biological brain really works. The Neural Network has witnessed significant improvement since 1943 to date. However, modifications on the Neural Network mainly focus on the structure itself, not the activation function despite the critical role of activation function in the performance of the Neural Network. In this paper, we present the modification of Neural Network activation function to improve the performance of the Neural Network. The theoretical background of the modification, including mathematical proof is fully described in the paper. The modified activation function is code name as SigHyper. The performance of SigHyper was evaluated against state of the art activation function on the crude oil price dataset. Results suggested that the proposed SigHyper was found to improved accuracy of the Neural Network. Analysis of variance showed that the accuracy of the SigHyper is significant. It was established that the SigHyper require further improvement. The activation function proposed in this research has added to the activation functions already discussed in the literature. The study may motivate researchers to further modify activation functions, hence, improve the performance of the Neural Network.

Hybrid feedback Group Method of Data Handling (GMDH)-type neural network using principal component-regression analysis is applied to the medical image recognition of the heart regions. In the GMDH-type neural network, the multi-layered deep neural networks are automatically organized so as to fit the complexity of the nonlinear system and, in general, the architectures of the GMDH-type neural network have many hidden layers and become complex for the nonlinear systems. In the multi-layered deep GMDH-type neural network with many hidden layers, the multi-colinearity occurs and the perdition accuracy become worse and the prediction values become unstable. In the GMDH-type neural network used in this study, the principal component-regression analysis is used as the learning algorithm of the neural network and the multi-colinearity do not occur and accurate and stable GMDH-type neural network architectures are automatically organized so as to fit the complexity of the nonlinear system. Furthermore, in this algorithm, three types of neural networks, such as sigmoid function neural network, radial basis function (RBF) neural network and polynomial neural network, can be generated using three types of neuron architectures, and the neural network architecture which fits the complexity of medical images, is selected from these three neural network architectures. This GMDH-type neural network is applied to the medical image recognition of the heart regions and it is shown that this GMDH-type neural network is useful for the medical image recognition of the heart regions.

In this paper, the revised Group Method of Data Handling (GMDH)-type neural networks with radial basis functions are proposed. This algorithm has been developed based on the basic GMDH-type neural networks which have both characteristics of the GMDH, the conventional multi-layered neural networks and the multi-layered radial basis function networks. The GMDH-type neural networks are automatically organized by using the heuristic self-organization method proposed by A.G. Ivakhnenko. In the GMDH-type neural networks, many types of neurons such as the sigmoid function type, the radial basis function type, the high order Polynomial type and the linear function type, are used and these neurons are automatically selected so as to fit the complexity of the nonlinear system by using an error criterion defined as Akaike's Information Criterion (AIC). The revised GMDH-type neural networks with radial basis functions proposed in this paper can identify a complex nonlinear system more accurately compared with the conventional radial basis function networks. Revised GMDH-type neural networks with radial basis functions are applied to the medical image recognition of the stomach and it is shown that this algorithm is very useful method for the medical image recognition.

The paper is concerned with continuously operating optimization neural networks with lossy dynamics. As the main feature of the neural model, the time-varying nature of neuron activation functions is introduced. The model presented is general in the sense that it covers the cases of neural networks for combinatorial optimization (Hopfield-like networks) and neural models for optimization problems with continuous decision variables (i.e., the Kennedy and Chua neural network). Besides the rigorous stability analysis of the proposed neural network, we also show how to derive lossy versions of improved Hopfield neural models from it and explore the relations to other optimization neural systems.

It is shown that modifying the sigmoidal basis function of a multi-layer feedforward artificial neural network using a control parameter improves the network's ability to learn. The modification is rendered by a gradient descent algorithm similar to the back-propagation. In doing so, the method retains all the goodies of the sigmoidal function and causes the ANN to approximate the decision function faster and also with better accuracy.

In this study, we propose hopfield neural networks with problem perturbing noise. In the proposed system, external noises are not induced to state values of neurons in order to avoid problems caused by the noise injection into the state values. We investigate the solving ability of the proposed system for quadratic assignment problems.

Goal of this paper is to highlight the most common problems and constraints which accompany the implementation of artificial neural networks on current digital computers. We focus on feed-forward multilayer neural networks, i.e. multilayer perceptrons, in role of function approximators. Multiple constraints of approximation by neural networks are discussed within the paper, taking into account research from the previous two decades. We address the issues of structural construction of feed-forward neural networks, learning and data pretreatment. Conclusions stated by universal approximation theorem cannot be blindly applied to implementations on real hardware without considering the limitations such as finite accuracy of floating point operations and data type overflow issues. This fact is emphasized in the paper.

Learning process is an important part in two-layer networks. It is imperative to search for an optimal learning rate to get a maximum error reduction in each learning step. Related literature has proposed various kinds of methods to find such an optimal learning rate in the past decades. In this paper, we proposed an improved dynamic optimal learning rate by adding an optimal ratio k. It is found that our improved dynamic optimal learning rate can generate a better result in learning processes. Meanwhile, we have proved the existence of the ratio kby giving it a proper math expression. Furthermore, we also applied the improved learning rate to solve inverse problem and compared the difference of the improved learning rate with the previous approach. It is observed that our proposed method performs better. Therefore, it can be concluded that our new method to search for dynamic optimal learning rate is valuable in the intelligence learning applications of neural networks, or it is effective in the aspect of tested problem at least.

In this work we show a study about which processes are related to chaotic and synchronized neural states based on the study of in-silico implementation of Stochastic Spiking Neural Networks (SSNN). Chaotic neural ensembles are excellent transmission and convolution systems. At the same time, synchronized cells (that can be understood as ordered states of the brain) are associated to more complex non-linear computations. We experimentally show that complex and quick pattern recognition processes arise when both synchronized and chaotic states are mixed. These measurements are in accordance with in-vivo observations related to the role of neural synchrony in pattern recognition and to the speed of the real biological process. The measurements obtained from the hardware implementation of different types of neural systems suggest that the brain processing can be governed by the superposition of these two complementary states with complementary functionalities (non-linear processing for synchronized states and information convolution and parallelization for chaotic).

Recent advances in neural network training provide a way to efficiently learn representations from raw data. Good representations are an important requirement for Music Information Retrieval (MIR) tasks to be performed successfully. However, a major problem with neural networks is that training time becomes prohibitive for very large datasets and the learning algorithm can get stuck in local minima for very deep and wide network architectures. In this paper we examine 3 ways to improve feature learning for audio data using neural networks: 1.using Rectified Linear Units (ReLUs) instead of standard sigmoid units; 2.using a powerful regularisation technique called Dropout; 3.using Hessian-Free (HF) optimisation to improve training of sigmoid nets. We show that these methods provide significant improvements in training time and the features learnt are better than state of the art handcrafted features, with a genre classification accuracy of 83 ± 1.1% on the Tzanetakis (GTZAN) dataset. We found that the rectifier networks learnt better features than the sigmoid networks. We also demonstrate the capacity of the features to capture relevant information from audio data by applying them to genre classification on the ISMIR 2004 dataset.

We describe a simple modification of neural networks which consists in extending the commonly used linear layer structure to an arbitrary graph structure. This allows us to combine the benefits of convolutional neural networks with the benefits of regular networks. The joint model has only a small increase in parameter size and training and decoding time are virtually unaffected. We report significant improvements over very strong baselines on two LVCSR tasks and one speech activity detection task.

In this paper, we propose a new technique to test the Satisfiability of propositional logic programming and quantified Boolean formula problem in radial basis function neural networks. For this purpose, we built radial basis function neural networks to represent the proportional logic which has exactly three variables in each clause. We used the Prey-predator algorithm to calculate the output weights of the neural networks, while the K-means clustering algorithm is used to determine the hidden parameters (the centers and the widths). Mean of the sum squared error function is used to measure the activity of the two algorithms. We applied the developed technique with the recurrent radial basis function neural networks to represent the quantified Boolean formulas. The new technique can be applied to solve many applications such as electronic circuits and NP-complete problems.

radial basis function ( RBF ) neural network depends mainly upon an adequate choice of the number and positions of its basis function centers. In this paper we have proposed an algorithm for RBF neural network and the results may be reduced for artificial neural networks as particular cases.

A model representing neural networks is proposed. It uses the functional graphs notion defined by R. Jakubowski (1977). This is a system of nodes connected with functional edges between which binary relations can be defined. Multilayer artificial neural networks can easily be defined using functional edges to model neurons, and parametrized binary relations to model synaptic connections. Learning is also defined in terms of functional graphs. The proposed description can produce descriptions of whole classes of networks.

Neural networks are developed for nonlinear mapping of desired control moments into achievable force commands for a distributed control subsystem. Fixed step and random component oracle data is calculated using iterative static function inversion methods. Unified and decoupled multi-layer neural structures are trained using gradient descent, batch mode gradient descent, and node-decoupled extended Kalman filtering methods. Exact function inversion, linear interpolation, and neural network mapping results are compared.

Feedforward multi-layer neural networks have very strong mapping capability that is based on the non-linearity of the activation function, however, the non-linearity of the activation function can cause the multiple local minima on the learning error-surfaces, which affect the learning rate and solving optimal weights. This paper proposes a learning method linearizing non-linearity of the activation function and discusses its merits and demerits theoretically.

Most artificial neural networks use a nonlinear activation function that includes sigmoid and hyperbolic tangent functions. Most artificial networks employ nonlinear functions such as these sigmoid and hyperbolic tangent functions, which incur high complexity costs, particularly during hardware implementation. In this paper, we propose new polynomial approximation methods for nonlinear activation functions that can substantially reduce complexity without sacrificing performance. The proposed approximation methods were applied to pattern classification problems. Experimental results show that the processing time was reduced by up to 50% without any performance degradations in terms of computer simulation.

Double sigmoid activation function is adopted to improve the BP neural networks to build the piezo-stage's on-line identification model. Then based on the on-line identification model, a neural networks model reference adaptive controller is designed to realize the piezo-stage's high accuracy tracking for micro/nanopositioning. Some experiment has been done to indicate the validity of the control scheme.