Search

Artificial neural networks (ANNs), a popular path towards artificial intelligence, have experienced remarkable success via mature models, various benchmarks, open-source datasets, and powerful computing platforms. Spiking neural networks (SNNs), a category of promising models to mimic the neuronal dynamics of the brain, have gained much attention for brain inspired computing and been widely deployed on neuromorphic devices. However, for a long time, there are ongoing debates and skepticisms about the value of SNNs in practical applications. Except for the low power attribute benefit from the spike-driven processing, SNNs usually perform worse than ANNs especially in terms of the application accuracy. Recently, researchers attempt to address this issue by borrowing learning methodologies from ANNs, such as backpropagation, to train high-accuracy SNN models. The rapid progress in this domain continuously produces amazing results with ever-increasing network size, whose growing path seems similar to the development of deep learning. Although these ways endow SNNs the capability to approach the accuracy of ANNs, the natural superiorities of SNNs and the way to outperform ANNs are potentially lost due to the use of ANN-oriented workloads and simplistic evaluation metrics. In this paper, we take the visual recognition task as a case study to answer the questions of " what workloads are ideal for SNNs and how to evaluate SNNs makes sense ". We design a series of contrast tests using different types of datasets (ANN-oriented and SNN-oriented), diverse processing models, signal conversion methods, and learning algorithms. We propose comprehensive metrics on the application accuracy and the cost of memory & compute to evaluate these models, and conduct extensive experiments. We evidence the fact that on ANN-oriented workloads, SNNs fail to beat their ANN counterparts; while on SNN-oriented workloads, SNNs can fully perform better. We further demonstrate that in SNNs there exists a trade-off between the application accuracy and the execution cost, which will be affected by the simulation time window and firing threshold. Based on these abundant analyses, we recommend the most suitable model for each scenario. To the best of our knowledge, this is the first work using systematical comparisons to explicitly reveal that the straightforward workload porting from ANNs to SNNs is unwise although many works are doing so and a comprehensive evaluation indeed matters. Finally, we highlight the urgent need to build a benchmarking framework for SNNs with broader tasks, datasets, and metrics. [ABSTRACT FROM AUTHOR]

Abstract Biological neural networks are systems of extraordinary computational capabilities shaped by evolution, development, and lifelong learning. The interplay of these elements leads to the emergence of biological intelligence. Inspired by such intricate natural phenomena, Evolved Plastic Artificial Neural Networks (EPANNs) employ simulated evolution in-silico to breed plastic neural networks with the aim to autonomously design and create learning systems. EPANN experiments evolve networks that include both innate properties and the ability to change and learn in response to experiences in different environments and problem domains. EPANNs' aims include autonomously creating learning systems, bootstrapping learning from scratch, recovering performance in unseen conditions, testing the computational advantages of particular neural components, and deriving hypotheses on the emergence of biological learning. Thus, EPANNs may include a large variety of different neuron types and dynamics, network architectures, plasticity rules, and other factors. While EPANNs have seen considerable progress over the last two decades, current scientific and technological advances in artificial neural networks are setting the conditions for radically new approaches and results. Exploiting the increased availability of computational resources and of simulation environments, the often challenging task of hand-designing learning neural networks could be replaced by more autonomous and creative processes. This paper brings together a variety of inspiring ideas that define the field of EPANNs. The main methods and results are reviewed. Finally, new opportunities and possible developments are presented. [ABSTRACT FROM AUTHOR]

The ability to train a network without memorizing the input/output data, thereby allowing a good predictive performance when applied to unseen data, is paramount in ANN applications. In this paper, we propose a frequency-domain approach to evaluate the network initialization in terms of quality of training, i.e., generalization capabilities. As an alternative to the conventional time-domain methods, the proposed approach eliminates the approximate nature of network validation using an excess of unseen data. The benefits of the proposed approach are demonstrated using two numerical examples, where two trained networks performed similarly on the training and the validation data sets, yet they revealed a significant difference in prediction accuracy when tested using a different data set. This observation is of utmost importance in modeling applications requiring a high degree of accuracy. The efficiency of the proposed approach is further demonstrated on a real-world problem, where unlike other initialization methods, a more conclusive assessment of generalization is achieved. On the practical front, subtle methodological and implementational facets are addressed to ensure reproducibility and pinpoint the limitations of the proposed approach. [ABSTRACT FROM AUTHOR]

Memristor describes the relationship between charge and flux. Although several window functions for memristors based on the HP linear and nonlinear dopant drift models have been studied, most of them are inadequate to capture the full characteristics of memristors. To address this issue, this paper proposes a unified window function to describe a general memristor with restrictions of its parameters given. Compared with other window functions, the proposed function demonstrates high validity and accuracy. In order to make the simulation results have high consistency with the results of actual circuit, we apply the new window function to the simulation of a memristor-based multilayer neural network (MNN) circuit. The overall accuracy will vary with the change of control parameters in the window function. It implies that the proposed model can guide the design of actual memristor-based circuits. [ABSTRACT FROM AUTHOR]

Unlike the first and the second generation artificial neural networks, spiking neural networks (SNNs) model the human brain by incorporating not only synaptic state but also a temporal component into their operating model. However, their intrinsic properties require expensive computation during training. This paper presents a novel algorithm to SpikeProp for SNN by introducing smoothing L 1 ∕ 2 regularization term into the error function. This algorithm makes the network structure sparse, with some smaller weights that can be eventually removed. Meanwhile, the convergence of this algorithm is proved under some reasonable conditions. The proposed algorithms have been tested for the convergence speed, the convergence rate and the generalization on the classical XOR-problem, Iris problem and Wisconsin Breast Cancer classification. [ABSTRACT FROM AUTHOR]

This paper is devoted to studying the fixed-time synchronization of memristor-based BAM neural networks (MBAMNNs) with discrete delay. Fixed-time synchronization means that synchronization can be achieved in a fixed time for any initial values of the considered systems. In the light of the double-layer structure of MBAMNNs, we design two similar feedback controllers. Based on Lyapunov stability theories, several criteria are established to guarantee that the drive and response MBAMNNs can realize synchronization in a fixed time. In particular, by changing the parameters of controllers, this fixed time can be adjusted to some desired value in advance, irrespective of the initial values of MBAMNNs. Numerical simulations are included to validate the derived results. [ABSTRACT FROM AUTHOR]

In this paper, a novel imbalance learning method for binary classes is proposed, named as Post-Boosting of classification boundary for Imbalanced data (PBI), which can significantly improve the performance of any trained neural networks (NN) classification boundary. The procedure of PBI simply consists of two steps: an (imbalanced) NN learning method is first applied to produce a classification boundary, which is then adjusted by PBI under the geometric mean (G-mean). For data imbalance, the geometric mean of the accuracies of both minority and majority classes is considered, that is statistically more suitable than the common metric accuracy . PBI also has the following advantages over traditional imbalance methods: (i) PBI can significantly improve the classification accuracy on minority class while improving or keeping that on majority class as well; (ii) PBI is suitable for large data even with high imbalance ratio (up to 0.001). For evaluation of (i), a new metric called Majority loss/Minority advance ratio (MMR) is proposed that evaluates the loss ratio of majority class to minority class. Experiments have been conducted for PBI and several imbalance learning methods over benchmark datasets of different sizes, different imbalance ratios, and different dimensionalities. By analyzing the experimental results, PBI is shown to outperform other imbalance learning methods on almost all datasets. [ABSTRACT FROM AUTHOR]

Abstract: This paper proposes a technique, called Evolving Probabilistic Neural Network (), that presents many interesting features, including incremental learning, evolving architecture, the capacity to learn continually throughout its existence and requiring that each training sample be used only once in the training phase without reprocessing. A series of experiments was performed on data sets in the public domain; the results indicate that is superior or equal to the other incremental neural networks evaluated in this paper. These results also demonstrate the advantage of the small architecture and show that its architecture is more stable than the other incremental neural networks evaluated. thus appears to be a promising alternative for a quick learning system and a fast classifier with a low computational cost. [Copyright &y& Elsevier]

Abstract: The neocognitron, which was proposed by , is a hierarchical multi-layered neural network capable of robust visual pattern recognition. It acquires the ability to recognize patterns through learning. This paper proposes a new rule for competitive learning, named winner-kill-loser, and apply it to the neocognitron. The winner-kill-loser rule resembles the winner-take-all rule. Every time when a training stimulus is presented, non-silent cells compete with each other. The winner, however, not only takes all, but also kills losers. In other words, the winner learns the training stimulus, and losers are removed from the network. If all cells are silent, a new cell is generated and it learns the training stimulus. Thus feature-extracting cells gradually come to distribute uniformly in the feature space. The use of winner-kill-loser rule is not limited to the neocognitron. It is useful for various types of competitive learning, in general. This paper also proposes several improvements made on the neocognitron: such as, disinhibition to the inhibitory surround in the connections to C-cells (or complex cells) from S-cells (or simple cells); and square root shaped saturation in the input-to-output characteristics of C-cells. As a result of these improvements, the recognition rate of the neocognitron has been largely increased. [Copyright &y& Elsevier]

Abstract: Selection of the optimal neural architecture to solve a pattern classification problem entails to choose the relevant input units, the number of hidden neurons and its corresponding interconnection weights. This problem has been widely studied in many research works but their solutions usually involve excessive computational cost in most of the problems and they do not provide a unique solution. This paper proposes a new technique to efficiently design the MultiLayer Perceptron (MLP) architecture for classification using the Extreme Learning Machine (ELM) algorithm. The proposed method provides a high generalization capability and a unique solution for the architecture design. Moreover, the selected final network only retains those input connections that are relevant for the classification task. Experimental results show these advantages. [Copyright &y& Elsevier]

Abstract: A Recurrent Neural Network (RNN) is a powerful connectionist model that can be applied to many challenging sequential problems, including problems that naturally arise in language and speech. However, RNNs are extremely hard to train on problems that have long-term dependencies, where it is necessary to remember events for many timesteps before using them to make a prediction. In this paper we consider the problem of training RNNs to predict sequences that exhibit significant long-term dependencies, focusing on a serial recall task where the RNN needs to remember a sequence of characters for a large number of steps before reconstructing it. We introduce the Temporal-Kernel Recurrent Neural Network (TKRNN), which is a variant of the RNN that can cope with long-term dependencies much more easily than a standard RNN, and show that the TKRNN develops short-term memory that successfully solves the serial recall task by representing the input string with a stable state of its hidden units. [Copyright &y& Elsevier]

Abstract: In this paper, the local coupled feedforward neural network is presented. Its connection structure is same as that of Multilayer Perceptron with one hidden layer. In the local coupled feedforward neural network, each hidden node is assigned an address in an input space, and each input activates only the hidden nodes near it. For each input, only the activated hidden nodes take part in forward and backward propagation processes. Theoretical analysis and simulation results show that this neural network owns the “universal approximation” property and can solve the learning problem of feedforward neural networks. In addition, its characteristic of local coupling makes knowledge accumulation possible. [Copyright &y& Elsevier]

Abstract: In this paper, a novel approach termed Enhanced Dynamic Self-Generated Fuzzy Q-Learning (EDSGFQL) for automatically generating Fuzzy Inference Systems (FISs) is presented. In the EDSGFQL approach, structure identification and parameter estimations of FISs are achieved via Unsupervised Learning (UL) (including Reinforcement Learning (RL)). Instead of using Supervised Learning (SL), UL clustering methods are adopted for input space clustering when generating FISs. At the same time, structure and preconditioning parts of a FIS are generated in a RL manner in that fuzzy rules are adjusted and deleted according to reinforcement signals. The proposed EDSGFQL methodologies can automatically create, delete and adjust fuzzy rules dynamically. Simulation studies on wall-following and obstacle avoidance tasks by a mobile robot show that the proposed approach is superior in generating efficient FISs. [Copyright &y& Elsevier]

Abstract: This paper proposes a novel learning method for a mixture of recurrent neural network (RNN) experts model, which can acquire the ability to generate desired sequences by dynamically switching between experts. Our method is based on maximum likelihood estimation, using a gradient descent algorithm. This approach is similar to that used in conventional methods; however, we modify the likelihood function by adding a mechanism to alter the variance for each expert. The proposed method is demonstrated to successfully learn Markov chain switching among a set of 9 Lissajous curves, for which the conventional method fails. The learning performance, analyzed in terms of the generalization capability, of the proposed method is also shown to be superior to that of the conventional method. With the addition of a gating network, the proposed method is successfully applied to the learning of sensory–motor flows for a small humanoid robot as a realistic problem of time series prediction and generation. [Copyright &y& Elsevier]

Abstract: The radial basis function (RBF) networks are one of the most widely used models for function approximation in the regression problem. In the learning paradigm, the best approximation is recursively or iteratively searched for based on observed data (teacher signals). One encounters difficulties in such a process when two component basis functions become identical, or when the magnitude of one component becomes null. In this case, the number of the components reduces by one, and then the reduced component recovers as the learning process proceeds further, provided such a component is necessary for the best approximation. Strange behaviors, especially the plateau phenomena, have been observed in dynamics of learning when such reduction occurs. There exist singularities in the space of parameters, and the above reduction takes place at the singular regions. This paper focuses on a detailed analysis of the dynamical behaviors of learning near the overlap and elimination singularities in RBF networks, based on the averaged learning equation that is applicable to both on-line and batch mode learning. We analyze the stability on the overlap singularity by solving the eigenvalues of the Hessian explicitly. Based on the stability analysis, we plot the analytical dynamic vector fields near the singularity, which are then compared to those real trajectories obtained by a numeric method. We also confirm the existence of the plateaus in both batch and on-line learning by simulation. [Copyright &y& Elsevier]

Abstract: This paper focuses on the evolution of Fuzzy ARTMAP neural network classifiers, using genetic algorithms, with the objective of improving generalization performance (classification accuracy of the ART network on unseen test data) and alleviating the ART category proliferation problem (the problem of creating more than necessary ART network categories to solve a classification problem). We refer to the resulting architecture as GFAM. We demonstrate through extensive experimentation that GFAM exhibits good generalization and is of small size (creates few ART categories), while consuming reasonable computational effort. In a number of classification problems, GFAM produces the optimal classifier. Furthermore, we compare the performance of GFAM with other competitive ARTMAP classifiers that have appeared in the literature and addressed the category proliferation problem in ART. We illustrate that GFAM produces improved results over these architectures, as well as other competitive classifiers. [Copyright &y& Elsevier]

Abstract: An environmentally adaptive system for prediction of acoustic transmission loss (TL) in the atmosphere is developed in this paper. This system uses several back propagation neural network predictors, each corresponding to a specific environmental condition. The outputs of the expert predictors are combined using a fuzzy confidence measure and a nonlinear fusion system. Using this prediction methodology the computational intractability of traditional acoustic model-based approaches is eliminated. The proposed TL prediction system is tested on two synthetic acoustic data sets for a wide range of geometrical, source and environmental conditions including both nonturbulent and turbulent atmospheres. Test results of the system showed root mean square (RMS) errors of 1.84 dB for the nonturbulent and 1.36 dB for the turbulent conditions, respectively, which are acceptable levels for near real-time performance. Additionally, the environmentally adaptive system demonstrated improved TL prediction accuracy at high frequencies and large values of horizontal separation between source and receiver. [Copyright &y& Elsevier]

Abstract: Fuzzy ARTMAP neural networks have been proven to be good classifiers on a variety of classification problems. However, the time that Fuzzy ARTMAP takes to converge to a solution increases rapidly as the number of patterns used for training is increased. In this paper we examine the time Fuzzy ARTMAP takes to converge to a solution and we propose a coarse grain parallelization technique, based on a pipeline approach, to speed-up the training process. In particular, we have parallelized Fuzzy ARTMAP without the match-tracking mechanism. We provide a series of theorems and associated proofs that show the characteristics of Fuzzy ARTMAP’s, without matchtracking, parallel implementation. Results run on a BEOWULF cluster with three large databases show linear speedup as a function of the number of processors used in the pipeline. The databases used for our experiments are the Forrest CoverType database from the UCI Machine Learning repository and two artificial databases, where the data generated were 16-dimensional Gaussian distributed data belonging to two distinct classes, with different amounts of overlap (5% and 15%). [Copyright &y& Elsevier]