Search

One of the greatest challenges in data mining is related to processing and analysis of massive data streams. Contrary to traditional static data mining problems, data streams require that each element is processed only once, the amount of allocated memory is constant and the models incorporate changes of investigated streams. A vast majority of available methods have been developed for data stream classification and only a few of them attempted to solve regression problems, using various heuristic approaches. In this paper, we develop mathematically justified regression models working in a time-varying environment. More specifically, we study incremental versions of generalized regression neural networks, called IGRNNs, and we prove their tracking properties - weak (in probability) and strong (with probability one) convergence assuming various concept drift scenarios. First, we present the IGRNNs, based on the Parzen kernels, for modeling stationary systems under nonstationary noise. Next, we extend our approach to modeling time-varying systems under nonstationary noise. We present several types of concept drifts to be handled by our approach in such a way that weak and strong convergence holds under certain conditions. Finally, in the series of simulations, we compare our method with commonly used heuristic approaches, based on forgetting mechanism or sliding windows, to deal with concept drift. Finally, we apply our concept in a real life scenario solving the problem of currency exchange rates prediction. [ABSTRACT FROM AUTHOR]

The construction of a Spiking Neural Network (SNN), i.e. the choice of an appropriate topology and the configuration of its internal parameters, represents a great challenge for SNN based applications. Evolutionary Algorithms (EAs) offer an elegant solution for these challenges and methods capable of exploring both types of search spaces simultaneously appear to be the most promising ones. A variety of such heterogeneous optimization algorithms have emerged recently, in particular in the field of probabilistic optimization. In this paper, a literature review on heterogeneous optimization algorithms is presented and an example of probabilistic optimization of SNN is discussed in detail. The paper provides an experimental analysis of a novel Heterogeneous Multi-Model Estimation of Distribution Algorithm (hMM-EDA). First, practical guidelines for configuring the method are derived and then the performance of hMM-EDA is compared to state-of-the-art optimization algorithms. Results show hMM-EDA as a light-weight, fast and reliable optimization method that requires the configuration of only very few parameters. Its performance on a synthetic heterogeneous benchmark problem is highly competitive and suggests its suitability for the optimization of SNN. [ABSTRACT FROM AUTHOR]

In this paper we investigate a hybrid model based on the Discrete Gradient method and an evolutionary strategy for determining the weights in a feed forward artificial neural network. Also we discuss different variants for hybrid models using the Discrete Gradient method and an evolutionary strategy for determining the weights in a feed forward artificial neural network. The Discrete Gradient method has the advantage of being able to jump over many local minima and find very deep local minima. However, earlier research has shown that a good starting point for the discrete gradient method can improve the quality of the solution point. Evolutionary algorithms are best suited for global optimisation problems. Nevertheless they are cursed with longer training times and often unsuitable for real world application. For optimisation problems such as weight optimisation for ANNs in real world applications the dimensions are large and time complexity is critical. Hence the idea of a hybrid model can be a suitable option. In this paper we propose different fusion strategies for hybrid models combining the evolutionary strategy with the discrete gradient method to obtain an optimal solution much quicker. Three different fusion strategies are discussed: a linear hybrid model, an iterative hybrid model and a restricted local search hybrid model. Comparative results on a range of standard datasets are provided for different fusion hybrid models. [ABSTRACT FROM AUTHOR]

In this paper, we present a novel approach of implementing a combination methodology to find appropriate neural network architecture and weights using an evolutionary least square based algorithm (GALS). This paper focuses on aspects such as the heuristics of updating -weights using an evolutionary least square based algorithm, finding the number of hidden neurons for a two layer feed forward neural network, the stopping criterion for the algorithm and finally some comparisons of the results with other existing methods for searching optimal or near optimal solution in the multidimensional complex search space comprising the architecture and the weight variables. We explain how the weight updating algorithm using evolutionary least square based approach can be combined with the growing architecture model to find the optimum number of hidden neurons. We also discuss the issues of finding a probabilistic solution space as a starting point for the least square method and address the problems involving fitness breaking. We apply the proposed approach to XOR problem, 10 bit odd parity problem and many realworld benchmark data sets such as handwriting data set from CEDAR, breast cancer and heart disease data sets from UCI ML repository. The comparative results based on classification accuracy and the time complexity are discussed. [ABSTRACT FROM AUTHOR]

This paper presents a novel model for performing classification and visualization of high-dimensional data by means of combining two enhancing techniques. The first is a semi-supervised learning, an extension of the supervised learning used to incorporate unlabeled information to the learning process. The second is an ensemble learning to replicate the analysis performed, followed by a fusion mechanism that yields as a combined result of previously performed analysis in order to improve the result of a single model. The proposed learning schema, termed S2-Ensemble, is applied to several unsupervised learning algorithms within the family of topology maps, such as the Self-Organizing Maps and the Neural Gas. This study also includes a thorough research of the characteristics of these novel schemes, by means quality measures, which allow a complete analysis of the resultant classifiers from the viewpoint of various perspectives over the different ways that these classifiers are used. The study conducts empirical evaluations and comparisons on various real-world datasets from the UCI repository, which exhibit different characteristics, so to enable an extensive selection of situations where the presented new algorithms can be applied. [ABSTRACT FROM AUTHOR]

The assessment of the risk of default on credit is important for financial institutions. Different Artificial Neural Networks (ANN) have been suggested to tackle the credit scoring problem, however, the obtained error rates are often high. In the search for the best ANN algorithm for credit scoring, this paper contributes with the application of an ANN Training Algorithm inspired by the neurons' biological property of metaplasticity. This algorithm is especially efficient when few patterns of a class are available, or when information inherent to low probability events is crucial for a successful application, as weight updating is overemphasized in the less frequent activations than in the more frequent ones. Two well-known and readily available such as: Australia and German data sets has been used to test the algorithm. The results obtained by AMMLP shown have been superior to state-of-the-art classification algorithms in credit scoring. [ABSTRACT FROM AUTHOR]

A new approach for nonlinear system identification and control based on modular neural networks (MNN) is proposed in this paper. The computational complexity of neural identification can be greatly reduced if the whole system is decomposed into several subsystems. This is obtained using a partitioning algorithm. Each local nonlinear model is associated with a nonlinear controller. These are also implemented by neural networks. The switching between the neural controllers is done by a dynamical switcher, also implemented by neural networks, that tracks the different operating points. The proposed multiple modelling and control strategy has been successfully tested on simulated laboratory scale liquid-level system. [ABSTRACT FROM AUTHOR]

This paper proposes a supervised training algorithm for Spiking Neural Networks (SNNs) which modifies the Spike Timing Dependent Plasticity (STDP)learning rule to support both local and network level training with multiple synaptic connections and axonal delays. The training algorithm applies the rule to two and three layer SNNs, and is benchmarked using the Iris and Wisconsin Breast Cancer (WBC) data sets. The effectiveness of hidden layer dynamic threshold neurons is also investigated and results are presented. [ABSTRACT FROM AUTHOR]

This paper demonstrates how knowledge can be extracted from evolving spiking neural networks with rank order population coding. Knowledge discovery is a very important feature of intelligent systems. Yet, a disproportionally small amount of research is centered on the issue of knowledge extraction from spiking neural networks which are considered to be the third generation of artificial neural networks. The lack of knowledge representation compatibility is becoming a major detriment to end users of these networks. We show that a high-level knowledge can be obtained from evolving spiking neural networks. More specifically, we propose a method for fuzzy rule extraction from an evolving spiking network with rank order population coding. The proposed method was used for knowledge discovery on two benchmark taste recognition problems where the knowledge learnt by an evolving spiking neural network was extracted in the form of zero-order Takagi-Sugeno fuzzy IF-THEN rules. [ABSTRACT FROM AUTHOR]

This paper presents a repair algorithm for the design of a Radial Basis Function (RBF) neural network. The proposed repair RBF (RRBF) algorithm starts from a single prototype randomly initialized in the feature space. The algorithm has two main phases: an architecture learning phase and a parameter adjustment phase. The architecture learning phase uses a repair strategy based on a sensitivity analysis (SA) of the network's output to judge when and where hidden nodes should be added to the network. New nodes are added to repair the architecture when the prototype does not meet the requirements. The parameter adjustment phase uses an adjustment strategy where the capabilities of the network are improved by modifying all the weights. The algorithm is applied to two application areas: approximating a non-linear function, and modeling the key parameter, chemical oxygen demand (COD) used in the waste water treatment process. The results of simulation show that the algorithm provides an efficient solution to both problems. [ABSTRACT FROM AUTHOR]

This paper investigates robust exponential stability for discrete-time recurrent neural networks with both time-varying delay (0 ≤ τm ≤ τ(k) ≤ τM) and distributed one. Through partitioning delay intervals [0,τm] and [τm,τM], respectively, and choosing an augmented Lyapunov-Krasovskii functional, the delay-dependent sufficient conditions are obtained by using free-weighting matrix and convex combination methods. These criteria are presented in terms of linear matrix inequalities (LMIs) and their feasibility can be easily checked by resorting to LMI in Matlab Toolbox in Ref. 1. The activation functions are not required to be differentiable or strictly monotonic, which generalizes those earlier forms. As an extension, we further consider the robust stability of discrete-time delayed Cohen-Grossberg neural networks. Finally, the effectiveness of the proposed results is further illustrated by three numerical examples in comparison with the reported ones. [ABSTRACT FROM AUTHOR]

This paper presents a new approach, known as symmetry axis based feature extraction and recognition (SAFER), for recognizing objects under translation, rotation and scaling. Unlike most previous invariant object recognition (IOR) systems, SAFER puts emphasis on both simplicity and accuracy of the recognition system. To achieve simplicity, it uses simple formulae for extracting invariant features from an object. The scheme used in feature extraction is based on the axis of symmetry and angles of concentric circles drawn around the object. SAFER divides the extracted features into a number of groups based on their similarity. To improve the recognition performance, SAFER uses a number of neural networks (NNs) instead of single NN are used for training and recognition of extracted features. The new approach, SAFER, has been tested on two of real world problems i.e., English characters with two different fonts and images of different shapes. The experimental results show that SAFER can produce good recognition performance in comparison with other algorithms. [ABSTRACT FROM AUTHOR]

Recently, a novel learning algorithm called extreme learning machine (ELM) was proposed for efficiently training single-hidden-layer feedforward neural networks (SLFNs). It was much faster than the traditional gradient-descent-based learning algorithms due to the analytical determination of output weights with the random choice of input weights and hidden layer biases. However, this algorithm often requires a large number of hidden units and thus slowly responds to new observations. Evolutionary extreme learning machine (E-ELM) was proposed to overcome this problem; it used the differential evolution algorithm to select the input weights and hidden layer biases. However, this algorithm required much time for searching optimal parameters with iterative processes and was not suitable for data sets with a large number of input features. In this paper, a new approach for training SLFNs is proposed, in which the input weights and biases of hidden units are determined based on a fast regularized least-squares scheme. Experimental results for many real applications with both small and large number of input features show that our proposed approach can achieve good generalization performance with much more compact networks and extremely high speed for both learning and testing. [ABSTRACT FROM AUTHOR]

HAM (Hopfield Associative Memory) and BAM (Bidirectinal Associative Memory) are representative associative memories by neural networks. The storage capacity by the Hebb rule, which is often used, is extremely low. In order to improve it, some learning methods, for example, pseudo-inverse matrix learning and gradient descent learning, have been introduced. Oh introduced pseudo-relaxation learning algorithm to HAM and BAM. In order to accelerate it, Hattori proposed quick learning. Noest proposed CAM (Complex-valued Associative Memory), which is complex-valued HAM. The storage capacity of CAM by the Hebb rule is also extremely low. Pseudo-inverse matrix learning and gradient descent learning have already been generalized to CAM. In this paper, we apply pseudo-relaxation learning algorithm to CAM in order to improve the capacity. [ABSTRACT FROM AUTHOR]

The present manuscript aims at illustrating fundamental challenges and solutions arising in the design of learning theories by optimization on manifolds in the context of complex-valued neural systems. The special case of a unitary unimodular group of matrices is dealt with. The unitary unimodular group under analysis is a low dimensional and easy-to-handle matrix group. Notwithstanding, it exhibits a rich geometrical structure and gives rise to interesting speculations about methods to solve optimization problems on manifolds. Also, its low dimension allows us to treat most of the quantities involved in computation in closed form as well as to render them in graphical format. Some numerical experiments are presented and discussed within the paper, which deal with complex-valued independent component analysis. [ABSTRACT FROM AUTHOR]

This paper introduces an Independent Component Analysis (ICA) approach to the separation of nonlinear mixtures in the complex domain. Source separation is performed by a complex INFOMAX approach. The neural network which realizes the separation employs the so called "Mirror Model" and is based on adaptive activation functions, whose shape is properly modified during learning. Nonlinear functions involved in the processing of complex signals are realized by pairs of spline neurons called "splitting functions", working on the real and the imaginary part of the signal respectively. Theoretical proof of existence and uniqueness of the solution under proper assumptions is also provided. In particular a simple adaptation algorithm is derived and some experimental results that demonstrate the effectiveness of the proposed solution are shown. [ABSTRACT FROM AUTHOR]

A method for arbitrary surface reconstruction from 3D large scattered points is proposed in this paper. According to the properties of 3D points, e.g. the non-uniform distribution and unknown topology, an implicit surface model is adopted based on radial basis functions network. And because of the property of locality of radial basis function, the method is fast and robust in surface reconstruction. Furthermore, an adapted K-Means algorithm is used to reduce reconstruction centers. For features completeness, two effective methods are introduced to extract the feature points before the adapted K-Means algorithm. Experiment results show that the presented approach is a good solution for reconstruction from 3D large scattered points. [ABSTRACT FROM AUTHOR]

In this paper we introduce methods to build a SOM that can be used as an isometric map for mobile robots. That is, given a dataset of sensor readings collected at points uniformly distributed with respect to the ground, we wish to build a SOM whose neurons (prototype vectors in sensor space) correspond to points uniformly distributed on the ground. Manifold learning techniques have already been used for dimensionality reduction of sensor space in navigation systems. Our focus is on the isometric property of the SOM. For reliable path-planning and information sharing between several robots, it is desirable that the robots build an internal representation of the sensor manifold, a map, that is isometric with the environment. We show experimentally that standard Non-Linear Dimensionality Reduction (NLDR) algorithms do not provide isometric maps for range data and bearing data. However, the auxiliary low dimensional manifolds created can be used to improve the distribution of the neurons of a SOM (that is, make the neurons more evenly distributed with respect to the ground). We also describe a method to create an isometric map from a sensor readings collected along a polygonal line random walk. [ABSTRACT FROM AUTHOR]

In this paper, the delayed projection neural network for a class of solving convex programming problem is proposed. The existence of solution and global exponential stability of the proposed network are proved, which can guarantee to converge at an exact optimal solution of the convex programming problems. Several examples are given to show the effectiveness of the proposed network. [ABSTRACT FROM AUTHOR]

A large class of neural network models have their units organized in a lattice with fixed topology or generate their topology during the learning process. These network models can be used as neighborhood preserving map of the input manifold, but such a structure is difficult to manage since these maps are graphs with a number of nodes that is just one or two orders of magnitude less than the number of input points (i.e., the complexity of the map is comparable with the complexity of the manifold) and some hierarchical algorithms were proposed in order to obtain a high-level abstraction of these structures. In this paper a general structure capable to extract high order information from the graph generated by a large class of self–organizing networks is presented. This algorithm will allow to build a two layers hierarchical structure starting from the results obtained by using the suitable neural network for the distribution of the input data. Moreover the proposed algorithm is also capable to build a topology preserving map if it is trained using a graph that is also a topology preserving map. [ABSTRACT FROM AUTHOR]

By analyzing the dynamic behaviors of the transiently chaotic neural network and greedy heuristic for the maximum independent set (MIS) problem, we present an improved transiently chaotic neural network for the MIS problem in this paper. Extensive simulations are performed and the results show that this proposed transiently chaotic neural network can yield better solutions to p-random graphs than other existing algorithms. The efficiency of the new model is also confirmed by the results on the complement graphs of some DIMACS clique instances in the second DIMACS challenge. Moreover, the improved model uses fewer steps to converge to stable state in comparison with the original transiently chaotic neural network. [ABSTRACT FROM AUTHOR]

In this paper, introducing stochastic dynamics into an optimal competitive Hopfield network model (OCHOM), we propose a new algorithm that permits temporary energy increases which helps the OCHOM escape from local minima. The goal of the maximum cut problem, which is an NP-complete problem, is to partition the node set of an undirected graph into two parts in order to maximize the cardinality of the set of edges cut by the partition. The problem has many important applications including the design of VLSI circuits and design of communication networks. Recently, Galán-Marín et al. proposed the OCHOM, which can guarantee convergence to a global/local minimum of the energy function, and performs better than the other competitive neural approaches. However, the OCHOM has no mechanism to escape from local minima. The proposed algorithm introduces stochastic dynamics which helps the OCHOM escape from local minima, and it is applied to the maximum cut problem. A number of instances have been simulated to verify the proposed algorithm. [ABSTRACT FROM AUTHOR]

This paper introduces a flexible neural tree model. The model is computed as a flexible multi-layer feed-forward neural network. A hybrid learning/evolutionary approach to automatically optimize the neural tree model is also proposed. The approach includes a modified probabilistic incremental program evolution algorithm (MPIPE) to evolve and determine a optimal structure of the neural tree and a parameter learning algorithm to optimize the free parameters embedded in the neural tree. The performance and effectiveness of the proposed method are evaluated using function approximation, time series prediction and system identification problems and compared with the related methods. [ABSTRACT FROM AUTHOR]

Fast and robust classification of feature vectors is a crucial task in a number of real-time systems. A cellular neural/nonlinear network universal machine (CNN-UM) can be very efficient as a feature detector. The next step is to post-process the results for object recognition. This paper shows how a robust classification scheme based on adaptive resonance theory (ART) can be mapped to the CNN-UM. Moreover, this mapping is general enough to include different types of feed-forward neural networks. The designed analogic CNN algorithm is capable of classifying the extracted feature vectors keeping the advantages of the ART networks, such as robust, plastic and fault-tolerant behaviors. An analogic algorithm is presented for unsupervised classification with tunable sensitivity and automatic new class creation. The algorithm is extended for supervised classification. The presented binary feature vector classification is implemented on the existing standard CNN-UM chips for fast classification. The experimental evaluation shows promising performance after 100% accuracy on the training set. [ABSTRACT FROM AUTHOR]

The aim of this paper is to present a unifying view of four SVD-neural-computation techniques found in the scientific literature and to present some theoretical results on their behavior. The considered SVD neural algorithms are shown to arise as Riemannian-gradient flows on double Stiefel manifold and their geometric and dynamical properties are investigated with the help of differential geometry. [ABSTRACT FROM AUTHOR]

This paper is concerned with large scale associative memory design. A serious problem with neural associative memories is the quadratic growth of the number of interconnections with the problem size. An overlapping decomposition algorithm is proposed to attack this problem. Specifically, a pattern to be processed is decomposed into overlapping sub-patterns. Then, neural sub-networks are constructed that process the sub-patterns. An error correction algorithm operates on the outputs of each sub-network in order to correct the mismatches between sub-patterns that are obtained from the independent recall processes of individual sub-networks. The performance of the proposed large scale associative memory is illustrated using two-dimensional images. It is shown that the proposed method reduces the computing cost of the design of the associative memories compared with non-interconnected associative memories. [ABSTRACT FROM AUTHOR]

This paper presents an approach of using Simulated Annealing and Tabu Search for the simultaneous optimization of neural network architectures and weights. The problem considered is the odor recognition in an artificial nose. Both methods have produced networks with high classification performance and low complexity. Generalization has been improved by using the backpropagation algorithm for fine tuning. The combination of simple and traditional search methods has shown to be very suitable for generating compact and efficient networks. [ABSTRACT FROM AUTHOR]

Vector quantization plays an important role in many signal processing problems, such as speech/speaker recognition and signal compression. This paper presents an unsupervised algorithm for vector quantizer design. Although the proposed method is inspired in Kohonen learning, it does not incorporate the classical definition of topological neighborhood as an array of nodes. Simulations are carried out to compare the performance of the proposed algorithm, named SOA (self-organizing algorithm), to that of the traditional LBG (Linde-Buzo-Gray) algorithm. The authors present an evaluation concerning the codebook design for Gauss-Markov and Gaussian sources, since the theoretic optimal performance bounds for these sources, as described by Shannon's Rate-Distortion Theory, are known. In speech and image compression, SOA codebooks lead to reconstructed (vector-quantized) signals with better quality as compared to the ones obtained by using LBG codebooks. Additionally, the influence of the initial codebook in the algorithm performance is investigated and the algorithm ability to learn representative patterns is evaluated. In a speaker identification system, it is shown that the the codebooks designed by SOA lead to higher identification rates when compared to the ones designed by LBG. [ABSTRACT FROM AUTHOR]

In this paper, we compare the performance of five prominent neural or adaptive algorithms designed for Independent Component Analysis (ICA) and blind source separation (BSS). In the first part of the study, we use artificial data for comparing the accuracy, convergence speed, computational load, and other relevant properties of the algorithms. In the second part, the algorithms are applied to three different real-world data sets. The task is either blind source separation or finding interesting directions in the data for visualisation purposes. We develop criteria for selecting the most meaningful basis vectors of ICA and measuring the quality of the results. The comparison reveals characteristic differences between the studied ICA algorithms. The most important conclusions of our comparison are robustness of the ICA algorithms with respect to modest modeling imperfections, and the superiority of fixed-point algorithms with respect to the computational load. [ABSTRACT FROM AUTHOR]