Search

Memory is one of the most important mechanisms in recurrent neural networks (RNNs) learning. It plays a crucial role in practical applications, such as sequence learning. With a good memory mechanism, long term history can be fused with current information, and can thus improve RNNs learning. Developing a suitable memory mechanism is always desirable in the field of RNNs. This paper proposes a novel memory mechanism for RNNs. The main contributions of this paper are: 1) an auxiliary memory unit (AMU) is proposed, which results in a new special RNN model (AMU-RNN), separating the memory and output explicitly and 2) an efficient learning algorithm is developed by employing the technique of error flow truncation. The proposed AMU-RNN model, together with the developed learning algorithm, can learn and maintain stable memory over a long time range. This method overcomes both the learning conflict problem and gradient vanishing problem. Unlike the traditional method, which mixes the memory and output with a single neuron in a recurrent unit, the AMU provides an auxiliary memory neuron to maintain memory in particular. By separating the memory and output in a recurrent unit, the problem of learning conflicts can be eliminated easily. Moreover, by using the technique of error flow truncation, each auxiliary memory neuron ensures constant error flow during the learning process. The experiments demonstrate good performance of the proposed AMU-RNNs and the developed learning algorithm. The method exhibits quite efficient learning performance with stable convergence in the AMU-RNN learning and outperforms the state-of-the-art RNN models in sequence generation and sequence classification tasks. [ABSTRACT FROM AUTHOR]

The traditional multilayer perceptron (MLP) using a McCulloch–Pitts neuron model is inherently limited to a set of neuronal activities, i.e., linear weighted sum followed by nonlinear thresholding step. Previously, generalized operational perceptron (GOP) was proposed to extend the conventional perceptron model by defining a diverse set of neuronal activities to imitate a generalized model of biological neurons. Together with GOP, a progressive operational perceptron (POP) algorithm was proposed to optimize a predefined template of multiple homogeneous layers in a layerwise manner. In this paper, we propose an efficient algorithm to learn a compact, fully heterogeneous multilayer network that allows each individual neuron, regardless of the layer, to have distinct characteristics. Based on the complexity of the problem, the proposed algorithm operates in a progressive manner on a neuronal level, searching for a compact topology, not only in terms of depth but also width, i.e., the number of neurons in each layer. The proposed algorithm is shown to outperform other related learning methods in extensive experiments on several classification problems. [ABSTRACT FROM AUTHOR]

In this paper, we propose a new scheme based on neural networks for predicting the packet disordering and sliding mode control (SMC) to stabilize the nonlinear networked control systems (NCSs). It is assumed that the packet disordering is unknown in the NCSs. The stochastic configuration networks (SCNs), which randomly assign the input weights and biases and analytically evaluate the output weights, are designed to solve the problem of unknown packet disordering. A new SMC scheme is developed by integrating the SCNs algorithm to learn and control the system in advance. Specifically, a novel measurement of packet disordering is constructed for the quantization of the packet disordering. In addition, the newest signal principle leads to the existence of stochastic parameters, thereby resulting in a Markovian jumping system. The effectiveness of the proposed approach is verified by some simulation results. [ABSTRACT FROM AUTHOR]

Extreme learning machine (ELM) is an emerging learning algorithm for the generalized single hidden layer feedforward neural networks, of which the hidden node parameters are randomly generated and the output weights are analytically computed. However, due to its shallow architecture, feature learning using ELM may not be effective for natural signals (e.g., images/videos), even with a large number of hidden nodes. To address this issue, in this paper, a new ELM-based hierarchical learning framework is proposed for multilayer perceptron. The proposed architecture is divided into two main components: 1) self-taught feature extraction followed by supervised feature classification and 2) they are bridged by random initialized hidden weights. The novelties of this paper are as follows: 1) unsupervised multilayer encoding is conducted for feature extraction, and an ELM-based sparse autoencoder is developed via \ell 1 constraint. By doing so, it achieves more compact and meaningful feature representations than the original ELM; 2) by exploiting the advantages of ELM random feature mapping, the hierarchically encoded outputs are randomly projected before final decision making, which leads to a better generalization with faster learning speed; and 3) unlike the greedy layerwise training of deep learning (DL), the hidden layers of the proposed framework are trained in a forward manner. Once the previous layer is established, the weights of the current layer are fixed without fine-tuning. Therefore, it has much better learning efficiency than the DL. Extensive experiments on various widely used classification data sets show that the proposed algorithm achieves better and faster convergence than the existing state-of-the-art hierarchical learning methods. Furthermore, multiple applications in computer vision further confirm the generality and capability of the proposed learning scheme. [ABSTRACT FROM AUTHOR]

The paper addresses analytical and practical aspects of integration of equation-based (classical) and signal-derived [artificial neural network (ANN)] dynamic models for transient analysis of large-scale dynamical systems. Our signal-based part is based on two ANNs, and is derived from measurements at boundary points. In this paper, we describe this hybrid modeling technique, and focus on: 1) a least square-based mechanism for on-line correction of dynamic variable predictions that is based on actual operating conditions; 2) the resilience of the algorithm to missing measurements due to failed communication links; and 3) a complete two-way interaction between the differential-algebraic equation based subsystem and the ANN-based subsystem. The paper demonstrates the feasibility of implementing our approach in standard power system software by integrating the ANN-based model with the transient analysis toolbox from Matlab. We illustrate capabilities of the proposed approach for transient analysis on a benchmark multi-machine example derived from the New England/New-York interconnected power system. [ABSTRACT FROM AUTHOR]

In this paper, we consider a modified error-correction learning rule for the multilayer neural network with multivalued neurons (MLMVN). This modification is based on the soft margins technique, which leads to the minimization of the distance between a cluster center and the learning samples belonging to this cluster. MLMVN has a derivative-free learning algorithm based on the error-correction learning rule and demonstrate a higher functionality and better generalization capability than a number of other machine learning techniques. The discrete $k$ -valued multivalued neuron activation function divides a complex plane into $k$ equal sectors. For more efficient and reliable solving of classification problems it is possible to modify the MLMVN learning algorithm in such a way that learning samples belonging to different classes (clusters) will be located as close as possible to the bisector of a desired sector (the cluster center) and as far as possible from each other, respectively. Such a modification based on the soft margins learning technique is considered in this paper. This modified learning algorithm improves the generalization capability of MLMVN when solving classification problems. [ABSTRACT FROM AUTHOR]

GoDec is an efficient low-rank matrix decomposition algorithm. However, optimal performance depends on sparse errors and Gaussian noise. This paper aims to address the problem that a matrix is composed of a low-rank component and unknown corruptions. We introduce a robust local similarity measure called correntropy to describe the corruptions and, in doing so, obtain a more robust and faster low-rank decomposition algorithm: GoDec+. Based on half-quadratic optimization and greedy bilateral paradigm, we deliver a solution to the maximum correntropy criterion (MCC)-based low-rank decomposition problem. Experimental results show that GoDec+ is efficient and robust to different corruptions including Gaussian noise, Laplacian noise, salt & pepper noise, and occlusion on both synthetic and real vision data. We further apply GoDec+ to more general applications including classification and subspace clustering. For classification, we construct an ensemble subspace from the low-rank GoDec+ matrix and introduce an MCC-based classifier. For subspace clustering, we utilize GoDec+ values low-rank matrix for MCC-based self-expression and combine it with spectral clustering. Face recognition, motion segmentation, and face clustering experiments show that the proposed methods are effective and robust. In particular, we achieve the state-of-the-art performance on the Hopkins 155 data set and the first 10 subjects of extended Yale B for subspace clustering. [ABSTRACT FROM AUTHOR]

In this paper, we propose four new variants of the backpropagation algorithm to improve the generalization ability for feedforward neural networks. The basic idea of these methods stems from the Group Lasso concept which deals with the variable selection problem at the group level. There are two main drawbacks when the Group Lasso penalty has been directly employed during network training. They are numerical oscillations and theoretical challenges in computing the gradients at the origin. To overcome these obstacles, smoothing functions have then been introduced by approximating the Group Lasso penalty. Numerical experiments for classification and regression problems demonstrate that the proposed algorithms perform better than the other three classical penalization methods, Weight Decay, Weight Elimination, and Approximate Smoother, on both generalization and pruning efficiency. In addition, detailed simulations based on a specific data set have been performed to compare with some other common pruning strategies, which verify the advantages of the proposed algorithm. The pruning abilities of the proposed strategy have been investigated in detail for a relatively large data set, MNIST, in terms of various smoothing approximation cases. [ABSTRACT FROM AUTHOR]

The spiking neural network (SNN) is the third generation of neural networks and performs remarkably well in cognitive tasks, such as pattern recognition. The temporal neural encode mechanism found in biological hippocampus enables SNN to possess more powerful computation capability than networks with other encoding schemes. However, this temporal encoding approach requires neurons to process information serially on time, which reduces learning efficiency significantly. To keep the powerful computation capability of the temporal encoding mechanism and to overcome its low efficiency in the training of SNNs, a new training algorithm, the accurate synaptic-efficiency adjustment method is proposed in this paper. Inspired by the selective attention mechanism of the primate visual system, our algorithm selects only the target spike time as attention areas, and ignores voltage states of the untarget ones, resulting in a significant reduction of training time. Besides, our algorithm employs a cost function based on the voltage difference between the potential of the output neuron and the firing threshold of the SNN, instead of the traditional precise firing time distance. A normalized spike-timing-dependent-plasticity learning window is applied to assigning this error to different synapses for instructing their training. Comprehensive simulations are conducted to investigate the learning properties of our algorithm, with input neurons emitting both single spike and multiple spikes. Simulation results indicate that our algorithm possesses higher learning performance than the existing other methods and achieves the state-of-the-art efficiency in the training of SNN. [ABSTRACT FROM PUBLISHER]

In this paper, we introduce a novel form of association rules (ARs) that do not require discretization of continuous variables or the use of intervals in either sides of the rule. This rule form captures nonlinear relationships among variables, and provides an alternative pattern representation for mining essential relations hidden in a given data set. We refer to the new rule form as a functional AR (FAR). A new neural network-based, co-operative, coevolutionary algorithm is presented for FAR mining. The algorithm is applied to both synthetic and real-world data sets, and its performance is analyzed. The experimental results show that the proposed mining algorithm is able to discover valid and essential underlying relations in the data. Comparison experiments are also carried out with the two state-of-the-art AR mining algorithms that can handle continuous variables to demonstrate the competitive performance of the proposed method. [ABSTRACT FROM PUBLISHER]

This paper proposes a control algorithm based on adaptive dynamic programming to solve the infinite-horizon optimal control problem for known deterministic nonlinear systems with saturating actuators and nonquadratic cost functionals. The algorithm is based on an actor/critic framework, where a critic neural network (NN) is used to learn the optimal cost, and an actor NN is used to learn the optimal control policy. The adaptive control nature of the algorithm requires a persistence of excitation condition to be a priori validated, but this can be relaxed using previously stored data concurrently with current data in the update of the critic NN. A robustifying control term is added to the controller to eliminate the effect of residual errors, leading to the asymptotically stability of the closed-loop system. Simulation results show the effectiveness of the proposed approach for a controlled Van der Pol oscillator and also for a power system plant. [ABSTRACT FROM AUTHOR]

In this paper, we consider the learning problem of multilayer perceptrons (MLPs) formulated as the problem of minimizing a smooth error function. As well known, the learning problem of MLPs can be a difficult nonlinear nonconvex optimization problem. Typical difficulties can be the presence of extensive flat regions and steep sided valleys in the error surface, and the possible large number of training data and of free network parameters. We define a wide class of batch learning algorithms for MLP, based on the use of block decomposition techniques in the minimization of the error function. The learning problem is decomposed into a sequence of smaller and structured minimization problems in order to advantageously exploit the structure of the objective function. Theoretical convergence results are established, and a specific algorithm is constructed and evaluated through an extensive numerical experimentation. The comparisons with the state-of-the-art learning algorithms show the effectiveness of the proposed techniques. [ABSTRACT FROM AUTHOR]

In this paper, a new method for the control of the shape of the conditional output probability density function (pdf) for general nonlinear dynamic stochastic systems is presented using two-step neural networks (NNs). Following the square-root B-spline NN approximation to the measured output pdf, the problem is transferred into the tracking of dynamic weights. Different from the previous related works, time-delay dynamic NNs with undetermined parameters are employed to identify the non- linear relationships between the control input and the weighting vectors. In order to achieve the required control objective and satisfy the state constraints due to the property of output pdfs, a constrained PI tracking controller is designed by solving a class of linear matrix inequalities and algebraic equations. With the proposed tracking controller and adaptive projection algorithms, both identification and tracking errors can be made to converge to zero, and the state constraints can also be simultaneously guaranteed. Finally, two simulated examples are given, which effectively demonstrate the use of the proposed control algorithm. [ABSTRACT FROM AUTHOR]

In this paper, first, we investigate the CNN templates producing periodic patterns. Next, two image processing algorithms using periodic pattern formation in cellular neural networks are proposed. By applying one to a certain image, area having spatial uniformity is removed from the original image and only parts without spatial uniformity remain. Simulation results show that when this algorithm is applied to an image containing characters (texts) or a human face image, only characters or eyes and mouth are extracted. Also, by applying the other to texture image, periodicity is enhanced in area having spatial periodicity, and texture can be segmented. Simulation results show that when this algorithm is applied to image that consists of four kinds of textures, each texture area can be segmented. [ABSTRACT FROM AUTHOR]

In this paper, a biologically inspired, CNN-based, multi-channel, texture boundary detection technique is presented. The proposed approach is similar to human vision system. The algorithm is simple and straightforward such that it can be implemented on the cellular neural networks (CNNs). CNN contains several important advantages, such as efficient real-time processing capability and feasible very large-scale integration (VLSI) implementation. The proposed algorithm also had been widely tested on synthetic texture images. Those texture images are randomly selected from the Brodatz textures database [1]. According to our simulation results, the boundaries of uniform textures can be detected quite successfully. For the nonuniform or nonregular textures, the results also indicate meaningful properties, and the properties also are consistent to the human visual sensation. The proposed algorithm also has been implemented on the CNN Universal Machine (CNN-UM), and yields similar results as the simulation on the PC. Based on the efficient performance of CNN-UM, the algorithm becomes very fast. [ABSTRACT FROM AUTHOR]

In this paper, we continue to explore application of nonsmooth analysis to the study of global asymptotic robust stability (GARS) of delayed neural networks. In combination with Lyapunov theory, our approach gives several new types of sufficient conditions ensuring GARS. A significant common aspect of our results is their low computational complexity. It is demonstrated that the reported results can be verified either by conducting spectral decompositions of symmetric matrices associated with the uncertainty sets of network parameters, or by solving a semidefinite programming problem. Nontrivial examples are constructed to compare with some closely related existing results. [ABSTRACT FROM AUTHOR]

Blind source separation (BSS) aims at recovering statistically independent source signals from their linear mixtures without knowing the mixing coefficients. Besides independent component analysis, nonlinear principal component analysis (NPCA) is shown to be another useful tool for solving this problem, but it requires that the measured data be prewhitened. By taking into account the autocorrelation matrix of the measured data, we present in this paper a modified NPCA criterion, and develop a least-mean-square (LMS) algorithm and a recursive least-squares algorithm. They can perform the online BSS using directly the unwhitened observations. Since a natural gradient learning is applied and the prewhitening process is removed, the proposed algorithms work more efficiently than the existing NPCA algorithms, as verified by computer simulations on man-made sources as well as practical speech signals. [ABSTRACT FROM AUTHOR]

This paper proposes a new image-descreening technique based on texture classification using a cellular neural network (CNN) with template trained by genetic algorithm (GA), called GA-CNN. Instead of using the fixed filters for image descreening, we are equipped with a more pliable mechanism for classifications in screening patterns. Using CNN makes it possible to get an accurate texture classification result in a faster speed by its superiority of implementable hardware and the flexible choices of templates. The use of the GA here helps us to look for the most appropriate template for CNNs more adaptively and methodically. The evolved parameters in the template for CNNs can not only provide a quicker classification mechanism but also help us with a better texture classification for screening patterns. After the class of screening patterns in the querying images is determined by the trained GA-CNN-based texture classification system, the recommendatory filters are induced to solve the screening problems. The induction of the classification in screening patterns has simplified the choke of filters and made it valueless to determine a new structured filter. Eventually, our comprehensive methodology is going to be topped off with more desirable results and the indication for the decrease in time complexity. [ABSTRACT FROM AUTHOR]

The promising potential of cellular neural networks (CNN) has resulted in the development of several template design methods. The CNN universal machine (CNN-UM), a programmable CNN, has made it possible to create image-processing algorithms that run on this platform. However, very large-scale integration implementations of CNN-UMs presented parameter deviations that do not occur on ideal CNN structures. Consequently, new design methods were developed aiming at more robust templates. Although these new templates were indeed more robust, erroneous behavior can still be observed. An alternative for chip-independent robustness is chip-specific optimization, where the template is targeted to an individual chip. This paper describes a solution proposal in this sense to automatically tune templates in order to make the chip react as an ideal CNN structure. The approach uses measurements of actual CNN-UM chips as part of the cost function for a global optimization method to find an optimal template given an initial approximation. Further improvements are achieved by generating chip-specific robust templates by doing a search for the best template among the optimal ones. The tuned templates are therefore customized versions that are expected to be much less sensitive to imperfections on the operation of CNN-UM chips. Results are presented for the binary and grayscale cases, including the case of grayscale output. It is expected that as this technique matures, it will give CNN-UM chips enough reliability to compete with digital systems in terms of robustness in addition to advantages of speed. [ABSTRACT FROM AUTHOR]

In this paper, an on-chip implementation of the active contour technique called pixel-level snakes is proposed. This is based on an optimized cellular neural network (CNN) algorithm with capabilities to support changes in the contour topology. The entire algorithm has been implemented on a 64 × 64 CNN universal machine chip-set architecture for which the results of the time performance measurements are given. To illustrate the validity and capabilities of the proposed implementation some on-chip experiments are also included. [ABSTRACT FROM AUTHOR]

In many real-world applications, data are represented by matrices or high-order tensors. Despite the promising performance, the existing 2-D discriminant analysis algorithms employ a single projection model to exploit the discriminant information for projection, making the model less flexible. In this paper, we propose a novel compound rank- $k$ projection (CRP) algorithm for bilinear analysis. The CRP deals with matrices directly without transforming them into vectors, and it, therefore, preserves the correlations within the matrix and decreases the computation complexity. Different from the existing 2-D discriminant analysis algorithms, objective function values of CRP increase monotonically. In addition, the CRP utilizes multiple rank- $k$ projection models to enable a larger search space in which the optimal solution can be found. In this way, the discriminant ability is enhanced. We have tested our approach on five data sets, including UUIm, CVL, Pointing’04, USPS, and Coil20. Experimental results show that the performance of our proposed CRP performs better than other algorithms in terms of classification accuracy. [ABSTRACT FROM PUBLISHER]

As the most important property of neural networks (NNs), the universal approximation capability of NNs is widely used in many applications. However, this property is generally proven for continuous systems. Most industrial systems are hybrid systems (e.g., piecewise continuous), which is a significant limitation for real applications. Recently, many identification methods have been proposed for hybrid system approximation; however, these methods only operate in linear hybrid systems. In this paper, the progressive learning machine—a new learning algorithm based on multi-NNs—is proposed for general hybrid nonlinear/linear system approximation. This algorithm classifies hybrid systems into several continuous systems and can approximate any hybrid system with zero output error. The performance of the proposed learning method is demonstrated via numerical examples and with experimental data from real applications. [ABSTRACT FROM AUTHOR]

In this paper, the challenging issue of estimating virtual unmodeled dynamics is addressed. A novel estimation algorithm based on historical data and the output of low-order approximation models for virtual un-modeled dynamics is presented. In particular, the virtual un-modeled dynamics are decomposed into known and unknown parts, where only the unknown part is to be estimated. The method effectively avoids the need to use the unknown control input directly, and enables the estimation of the un-modeled dynamics with a relatively simple algorithm. Moreover, it is shown that the proposed algorithm overcomes the difficulty in obtaining the control solutions caused by the fact that the controller input is embedded in un-modeled dynamics. Finally, simulation studies are presented to demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

In this paper, the stochastic synchronization problem is studied for a class of delayed dynamical networks under delayed impulsive control. Different from the existing results on the synchronization of dynamical networks under impulsive control, impulsive input delays are considered in our model. By assuming that the impulsive intervals belong to a certain interval and using the mathematical induction method, several conditions are derived to guarantee that complex networks are exponentially synchronized in mean square. The derived conditions reveal that the frequency of impulsive occurrence, impulsive input delays, and stochastic perturbations can heavily affect the synchronization performance. A control algorithm is then presented for synchronizing stochastic dynamical networks with delayed synchronizing impulses. Finally, two examples are given to demonstrate the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

It is by now well-known that wireless networks with file arrivals and departures are stable if one uses $\alpha$-fair congestion control and back-pressure based scheduling and routing. In this paper, we examine whether $\alpha$-fair congestion control is necessary for flow-level stability. We show that stability can be ensured even with very simple congestion control mechanisms, such as a fixed window size scheme which limits the maximum number of packets that are allowed into the ingress queue of a flow. A key ingredient of our result is the use of the difference between the logarithms of queue lengths as the link weights. This result is reminiscent of results in the context of CSMA algorithms, but for entirely different reasons. [ABSTRACT FROM AUTHOR]

This paper considers a cooperative tracking problem for a group of nonlinear multiagent systems under a directed graph that characterizes the interaction between the leader and the followers. All the networked systems can have different dynamics and all the dynamics are unknown. A neural network (NN) with flexible structure is used to approximate the unknown dynamics at each node. Considering that the leader is a neighbor of only a subset of the followers and the followers have only local interactions, we introduce a cooperative dynamic observer at each node to overcome the deficiency of the traditional tracking control strategies. An observer-based cooperative controller design framework is proposed with the aid of graph tools, Lyapunov-based design method, self-structuring NN, and separation principle. It is proved that each agent can follow the active leader only if the communication graph contains a spanning tree. Simulation results on networked robots are provided to show the effectiveness of the proposed control algorithms. [ABSTRACT FROM AUTHOR]

Abstract-The use of a neural network (NN) as a feedforward control element to compensate for nonlinear system uncertainties has been investigated for over a decade. Typical NN-based controllers yield uniformly ultimately bounded (UUB) stability results due to residual functional reconstruction inaccuracies and an inability to compensate for some system disturbances. Several researchers have proposed discontinuous feedback controllers (e.g., variable structure or sliding mode controllers) to reject the residual errors and yield asymptotic results. The research in this paper describes how a recently developed continuous robust integral of the sign of the error (RISE) feedback term can be incorporated with a NN-based feedforward term to achieve semi-global asymptotic tracking. To achieve this result, the typical stability analysis for the RISE method is modified to enable the incorporation of the NN-based feedforward terms, and a projection algorithm is developed to guarantee bounded NN weight estimates. [ABSTRACT FROM AUTHOR]

A new result is developed for separating nonlinearly mixed signals in which the nonlinearity is characterized by a class of strictly monotonic continuously differentiable functions. The structure of the blind inverse system is explicitly derived within the framework of maximum likelihood estimation and the system culminates to a special architecture of the 3-layer perceptron neural network where the parameters in the first layer are inversely related to the output layer. The proposed approach exploits both the structural and signal constraints to search for the solution and assumes that the cumulants of the source signals are known a priori. A novel statistical algorithm based on the hybridization of the generalized gradient algorithm and metropolis algorithm has been derived for training the proposed perceptron which results in improved performance in terms of accuracy and convergence speed. Simulations and real-life experiment have also been conducted to verify the efficacy of the proposed scheme in separating the nonlinearly mixed signals. [ABSTRACT FROM AUTHOR]