Search

Activation function is crucial to the recent successes of deep neural networks. In this paper, we first propose a new activation function, Multiple Parametric Exponential Linear Units (MPELU), aiming to generalize and unify the rectified and exponential linear units. As the generalized form, MPELU shares the advantages of Parametric Rectified Linear Unit (PReLU) and Exponential Linear Unit (ELU), leading to better classification performance and convergence property. In addition, weight initialization is very important to train very deep networks. The existing methods laid a solid foundation for networks using rectified linear units but not for exponential linear units. This paper complements the current theory and extends it to the wider range. Specifically, we put forward a way of initialization, enabling training of very deep networks using exponential linear units. Experiments demonstrate that the proposed initialization not only helps the training process but leads to better generalization performance. Finally, utilizing the proposed activation function and initialization, we present a deep MPELU residual architecture that achieves state-of-the-art performance on the CIFAR-10/100 datasets. The code is available at https://github.com/Coldmooon/Code-for-MPELU . [ABSTRACT FROM AUTHOR]

This paper presents an artificial neural network to solve the quadratic zero-one programming problems under linear constraints. In this paper, by using the connection between integer and nonlinear programming, the quadratic zero-one programming problem is transformed into the quadratic programming problem with nonlinear constraints. Then, by using the nonlinear complementarity problem (NCP) function and penalty method this problem is transformed into an unconstrained optimization problem. It is shown that the Hessian matrix of the associated function in the unconstrained optimization problem is positive definite in the optimal point. To solve the unconstrained optimization problem an artificial neural network is used. The proposed neural network has a simple structure and a low complexity of implementation. It is shown here that the proposed artificial neural network is stable in the sense of Lyapunov. Finally, some numerical examples are given to show that the proposed model finds the optimal solution of this problem in the low convergence time. [ABSTRACT FROM AUTHOR]

The attention of this paper is mainly concentrated on the H ∞ state estimator's design problem for a kind of discrete-time memristive neural networks (MNNs) with multiple fading measurements. The phenomenon of multiple fading measurements is represented by a set of individual stochastic variables obeying a predetermined distribution on interval [0,1]. Firstly, the augmented system comprised of MNNs and the dynamics of estimation errors are put forward to implement the performance analysis. Then, under the framework of the difference inclusion theory combined with the Lyapunov function method, several sufficient conditions are established to guarantee the exponential mean-square stability as well as H ∞ performance index. Furthermore, the desired estimator parameter is obtained in view of the solution of a convex optimization problem. In the end, an illustrative numerical example is exploited to check the reliability and usefulness of the design scheme in this paper. [ABSTRACT FROM AUTHOR]

Visual recognition of crowd dynamics has had a huge impact on several applications including surveillance, situation awareness, homeland security and intelligent environments. However, the state-of-the-art in crowd analysis has become diverse due to factors such as: (a) the underlying definition of a crowd, (b) the constituent elements of the crowd processing model, (c) its application, hence (d) the dataset and (e) the evaluation criteria available for benchmarking. Although such diversity is healthy, the techniques for crowd modelling thus developed have failed to establish credibility therefore becoming unreliable and of questionable validity across different research disciplines. This paper aims to give an account of such issues by deducing key statistical evidence from the existing literature and providing recommendations towards focusing on the general aspects of techniques rather than any specific algorithm. The advances and trends in crowd analysis are drawn in the context of crowd modelling studies published in leading journals and conferences over the past 7 years. Finally, this paper shall also provide a qualitative and quantitative comparison of some existing methods using various publicly available crowd datasets to substantiate some of the theoretical claims. [ABSTRACT FROM AUTHOR]

A teaching-learning-based optimization algorithm (TLBO) which uses a variable population size in the form of a triangle form (VTTLBO) is proposed in the paper. The main goal of the proposed method is to decrease the computing cost of original TLBO and extend it for optimizing the parameters of artificial neural network (ANN). In the proposed algorithm, the evolutionary process is divided into some equal periods according to the maximal generation. The population size in each period is changed in form of triangle. In the linear increasing phase of population’s number, some new individuals are generated with gauss distribution by using the adaptive mean and variance of the population. In the linear decreasing phase of population׳s number, some highly similar individuals are deleted. To compare the performance of the proposed method with some other methods, Saw-tooth teaching-learning-based optimization algorithm is also designed with simulating the basic principle of Saw-tooth genetic algorithm (STGA), and some other EAs with the fixed population size are also simulated. A variety of benchmark problems and system modeling and prediction problems with ANN are tested in this paper, the results indicate that the computation cost of the given method is small and the convergence accuracy and speed of it are high. [ABSTRACT FROM AUTHOR]

This paper proposes a novel optimization method that utilizes a multi-population parallel genetic algorithm (GA) to simultaneously optimize the structure and system latency of evolvable block-based neural networks (BbNNs). In the past, BbNNs have been successfully deployed for various kinds of classification problems with promising results. However, existing BbNN architectures do not explicitly model or optimize the latency of the system. Optimization of system latency is important, because it improves performance, reduces power consumption, and allows a more deterministic behavior that is needed for recurrent modes. The BbNN hardware model proposed in this paper is implemented as a synchronous architecture with registered outputs. This architecture will permit the system latency to be deterministic. To facilitate the automated layout of the neuron block interconnects within the BbNN structure, an array interconnect algorithm is presented. The proposed latency-optimized BbNN is implemented in hardware, and the architecture resembles mobile devices with integrated graphics processing units (GPUs). The hardware implementation is verified and analyzed using various case studies. The power consumption of our prototyping platform was measured, and it is shown that approximately 364 nJ of energy can be saved for every computational cycle (latency) reduced. All presented case studies show that the proposed BbNN model provides an optimal latency value while still matching the high accuracy reported in previous works. [ABSTRACT FROM AUTHOR]

As one of standard approaches to train deep neural networks, dropout has been applied to regularize large models to avoid overfitting, and the improvement in performance by dropout has been explained as avoiding co-adaptation between nodes. However, when correlations between nodes are compared after training the networks with or without dropout, one question arises if co-adaptation avoidance explains the dropout effect completely. In this paper, we propose an additional explanation of why dropout works and propose a new technique to design better activation functions. First, we show that dropout can be explained as an optimization technique to push the input towards the saturation area of nonlinear activation function by accelerating gradient information flowing even in the saturation area in backpropagation. Based on this explanation, we propose a new technique for activation functions, gradient acceleration in activation function (GAAF) , that accelerates gradients to flow even in the saturation area. Then, input to the activation function can climb onto the saturation area which makes the network more robust because the model converges on a flat region. Experiment results support our explanation of dropout and confirm that the proposed GAAF technique improves image classification performance with expected properties. [ABSTRACT FROM AUTHOR]

Classification is a key factor in accuracy, simplicity, and expressiveness, and it is difficult to optimize all of these factors at the same time. The learning classifier system (LCS) is a suitable technique for addressing an adaptive classification problem. It is a combination of fast approximation and evolutionary optimization techniques. A neural-based learning classifier system (N-LCS) includes an architecture for maintaining expressiveness by incorporating neural networks into a supervised classifier system, which is also an LCS specializing in classification studies. In recent years, studies using deep artificial neural networks have been actively conducted. In particular, deep convolutional neural networks (CNN) provide a powerful representation in an extremely fundamental method and demonstrates the high performance in various domains. In this paper, we exploit various deep CNN architectures in convolutional neural-based learning classifier systems (CN-LCS) combining the CNN and LCS to explore the possibility of a CN-LCS. By using various CNNs as an action of a classifier in an N-LCS, better classification accuracy can be obtained and classifier can be optimized. Experimental results show that our models achieve the higher performance than N-LCS for database intrusion detection as well as two other datasets, and extract effective features from deep representation by projecting data samples learned by several deep CNN models into the feature space. [ABSTRACT FROM AUTHOR]

Abstract: This paper aims at automatic classification of power quality events using Wavelet Packet Transform (WPT) and Support Vector Machines (SVM). The features of the disturbance signals are extracted using WPT and given to the SVM for effective classification. Recent literature dealing with power quality establishes that support vector machine methods generally outperform traditional statistical and neural methods in classification problems involving power disturbance signals. However, the two vital issues namely the determination of the most appropriate feature subset and the model selection, if suitably addressed, could pave way for further improvement of their performances in terms of classification accuracy and computation time. This paper addresses these issues through a classification system using two optimization techniques, the genetic algorithms and simulated annealing. This system detects the best discriminative features and estimates the best SVM kernel parameters in a fully automatic way. Effectiveness of the proposed detection method is shown in comparison with the conventional parameter optimization methods discussed in literature like grid search method, neural classifiers like Probabilistic Neural Network (PNN), fuzzy k-nearest neighbor classifier (FkNN) and hence proved that the proposed method is reliable as it produces consistently better results. [Copyright &y& Elsevier]

Abstract In video recognition, rank-pooling operators are a type of models for sorting video sequences, which act on either the raw inputs or the intermediate feature maps of convolutional neural network (CNN). However, such models are currently restricted in the optimization of the linear ranking function by Rank-SVM and Rank-SVR. In this paper, we first propose a CNN architecture called RGB Rank Pooling Dynamic Network (RGB-RPDN), mapping a video to multiple frame-level dynamic spaces with the same size as the input. Importantly, a classical classification (e.g. FC, CNN) advanced in 2D image can be jointly positioned behind the generated representation for action classification, thus the joint architecture can be trained in an end-to-end manner. Second, we analyze how the flow-level evolution can be modeled by the hand-crafted rank-pooling machine, and extend the dynamic space of frame-level to that of flow-level by the Flow Rank Pooling Dynamic Network (Flow-RPDN). Third, equivalence relations between hand-crafted rank-pooling and RPDN are formulated, further the comparison of computing cost are qualitatively analyzed. Finally, the frame-level and flow-level pipelines are combined to achieve the final prediction by the late fusion. Specifically, with the models pre-trained on the large-scale Kinetics dataset, we train the two-stream RPDN on the UCF101 and HMDB51, where the parameters are initialized by the pre-trained models above. Experimental results demonstrate that the RPDN significantly improves the hand-crafted rank-pooling machines by a large margin of promotion, and achieves the correct rate of more excellent classification in action recognition. [ABSTRACT FROM AUTHOR]

Abstract In this paper, we use deep feedforward artificial neural networks to approximate solutions to partial differential equations in complex geometries. We show how to modify the backpropagation algorithm to compute the partial derivatives of the network output with respect to the space variables which is needed to approximate the differential operator. The method is based on an ansatz for the solution which requires nothing but feedforward neural networks and an unconstrained gradient based optimization method such as gradient descent or a quasi-Newton method. We show an example where classical mesh based methods cannot be used and neural networks can be seen as an attractive alternative. Finally, we highlight the benefits of deep compared to shallow neural networks and device some other convergence enhancing techniques. [ABSTRACT FROM AUTHOR]

Abstract: For the analysis of natural neural responses it is necessary to evaluate and compare the reliability of the produced spike sequences. The same occurs in the development and evaluation of neural models, which should mimic the real neural centers that are being modeled. Several neural metrics have been proposed to analyze neural responses, and to tune and evaluate neural models. Neural metrics measure different characteristics of the neural code and can be grouped into distinct classes, as they follow a firing rate or time-code perspective. In this paper, several metrics belonging to the firing rate, spike train and firing event classes are reviewed. Using sets of neuronal responses and a set of models, the metrics are analyzed and compared to disclose their advantages and drawbacks. In most cases these metrics depend on a free parameter, that establishes their sensitivity to particular characteristics of the neural code. After showing that the incorrect choice of these parameters can lead to meaningless results, methods are presented in this paper to define a valid range of values for the parameters. These methods are based on a statistical analysis of the inter-trials errors. The application of neural metrics to the tuning and assessment of neural models of distinct classes reveals important results. Some of the analyzed metrics possess pronounced minima, specifically around the origin, which makes the optimization process more difficult; nonetheless, they provide insightful results for the evaluation of models. This paper also discusses the application of the neural metrics to evaluate neural models, providing relevant guidelines for their utilization. [Copyright &y& Elsevier]

Abstract: In this paper, the global exponential stability is discussed for static recurrent neural networks. Without assuming the boundedness, monotonicity and differentiability of the activation functions, a new sufficient condition is obtained to ensure the existence and uniqueness of the equilibrium based on the nonlinear measure. Meanwhile, the condition obtained also guarantees the global exponential stability of the delayed neural networks via constructing a proper Lyapunov functional. The results, which are independent of the time delay, can be checked easily by convex optimization algorithms. In the end of this paper, two illustrative examples are also given to show the effectiveness of our results. [Copyright &y& Elsevier]

This paper is concerned with the finite-time synchronization problem of drive-response inertial memristive neural networks with time delay. First, by choosing suitable variable substitution, the original system can be transformed into the first order differential equations. Next, a delay-dependent controller is designed to ensure that finite-time synchronization can be achieved between drive system and response system based on finite time stability theory. Moreover, the settling time is estimated, and optimized based on the relationship between the settling time and parameter η . Finally, an example is presented to substantiate the effectiveness for those theoretical results. [ABSTRACT FROM AUTHOR]

The stock market is affected by many factors, such as political events, general economic conditions, and traders’ expectations. Predicting the direction of stock markets movement has been one of the most widely investigated and challenging problems for investors and researchers as well. Many researchers focus on stock market analysis using advanced knowledge of mathematics, computer sciences, economics and many other disciplines. In this paper, we present an improved sine cosine algorithm (ISCA), which introduces an additional parameter into the sine cosine algorithm (SCA), to optimize the weights and basis of back propagation neural networks (BPNN). Thus, ISCA and BPNN are combined to create a new network, ISCA-BPNN, for predicting the directions of the opening stock prices for the S&P 500 and Dow Jones Industrial Average Indices, respectively. In addition, Google Trends data are taken into consideration for improving stock prediction. We analyze two types of prediction: Type I is the prediction without Google Trends and Type II is the prediction with Google Trends. The predictability of stock price direction is verified by using the hybrid ISCA-BPNN model. The experimental results indicate that ISCA–BPNN outperforms BPNN, GWO-BPNN, PSO-BPNN, WOA-BPNN and SCA-BPNN in terms of predicting the direction of the opening price for both types and significantly for Type II. The hit ratios for ISCA-BPNN with Google Trends reach 86.81% for the S&P 500 Index, and 88.98% for the Dow Jones Industrial Average Index. Our results show that Google Trends can help in predicting the direction of the stock market index. [ABSTRACT FROM AUTHOR]

For the purpose of enhancing the discriminability of convolutional neural networks (CNNs) and facilitating the optimization, we investigate the activation function for a neural network and the corresponding initialization method in this paper. Firstly, a trainable activation function with a multi-layer structure (named “Multi-layer Maxout Network”, MMN) is proposed. MMN is a multi-layer structured maxout, inheriting advantages of both a non-saturated activation function and a trainable activation function approximator. Secondly, we derive a robust initialization method specifically for the MMN activation with a theoretical proof, which works for the maxout activation as well. Our novel initialization method could reduce internal covariate shift when signals are propagated through layers and solve the so called “exploding/vanishing gradient” problem, which leads a more efficient training procedure of deep neural networks. Experimental results show that our proposed model yields better performance on three image classification benchmark datasets (CIFAR-10, CIFAR-100 and ImageNet) than quite a few state-of-the-art methods and our novel initialization method improves performance further. Furthermore, the influence of MMN in different hidden layers is analyzed, and a trade-off scheme between the accuracy and computing resources is given. [ABSTRACT FROM AUTHOR]

This paper investigates the non-zerosum game issue for unknown nonlinear systems with multi-player by using data-based adaptive dynamic programming (ADP) methods. It is known that the traditional ADP approaches require accurate system models to compute the solutions of non-zerosum games. However, for the practical nonlinear systems, system models are difficult to be obtained and thus these methods will be invalid. To overcome this difficulty, we propose two neural-network-based identification schemes. Combined with the identification results, we design a data-based actor-critic algorithm to learn and approximate the optimal solutions in real time. Subsequently, in order to reduce the computation burden of dual network algorithm, we develop a single network one. To test the feasibility and validity of our schemes, we provide two simulation examples including a linear one and a nonlinear one. [ABSTRACT FROM AUTHOR]

In this paper, the global asymptotic stability of two-time-scale competitive neural networks(CNNs) with multiple time-varying delays is investigated. By constructing a new ε-dependent Lyapunov functional, sufficient conditions for the global asymptotic stability of the concerned systems are established, and an optimization problem is formulated to get the best estimate of the ε-bound. Compared with the existing results, the proposed results are more general and less conservative in the sense of determining an upper bound for the time-scale parameter ε. Finally, three examples are given to illustrate the advantages of the obtained results. [ABSTRACT FROM AUTHOR]

Energy usage and demand forecasting is an essential and complex task in real time implementation. Proper coordination is required between the consumer and power companies for monitoring, scheduling and operating the electrical devices without any damages. In this paper, we propose a novel neural network based optimization approach for energy demand prediction. Initially, the Conventional Neural Network (CNN) approach is employed to find the required energy demand prediction at the consumer end. Secondly, Neural Network based Genetic Algorithm (NNGA) and Neural Network based Particle Swarm Optimization (NNPSO) approaches are implemented where the weights of the neural network are automatically adjusted. Closer observation from the result reveals that the proposed NNGA approach performs better for short term load forecasting and proposed NNPSO is more suitable for the long term energy prediction. For the experimental results, real time data are taken from pecan street (Pecan Street Inc.). From the simulations, it can be concluded that the proposed optimization approach algorithm yield better results than the CNN approach in predicting the future energy demand. Further, the result reveals that it is possible to manage the demand and supply, planning of power grid and prediction of future energy requirement in the smart grid. [ABSTRACT FROM AUTHOR]

In this work, we address to boost the discriminative capability of deep neural network by alleviating the over-fitting problem. Previous works often deal with the problem of learning a neural network by optimizing one or more objective functions with some existing regularization methods (such as dropout, weight decay, stochastic pooling, data augmentation, etc.). We argue that these approaches may be difficult to further improve the classification performance of a neural network, due to not well employing its own learned knowledge. In this paper, we introduce a self-regularized strategy for learning a neural network, named as a Self-Regularized Neural Network (SRNN). The intuition behind the SRNN is that the sample-wise soft targets of a neural network may have potentials to drag its own neural network out of its local optimum. More specifically, an initial neural network is firstly pre-trained by optimizing one or more objective functions with ground truth labels. We then gradually mine sample-wise soft targets, which enables to reveal the correlation/similarity among classes predicted from its own neural network. The parameters of neural network are further updated for fitting its sample-wise soft targets. This self-regularization learning procedure minimizes the objective function by integrating the sample-wise soft targets of neural network and the ground truth label of training samples. Three characteristics in this SRNN are summarized as: (1) gradually mining the learned knowledge from a single neural network, and then correcting and enhancing this part of learned knowledge, resulting in the sample-wise soft targets; (2) regularly optimizing the parameters of this neural network with their sample-wise soft targets; (3) boosting the discriminative capability of a neural network with the self-regularization strategy. Extensive experiments on four public datasets, i.e., CIFAR-10, CIFAR-100, Caltech101 and MIT, well demonstrate the effectiveness of the proposed SRNN for image classification. [ABSTRACT FROM AUTHOR]

Dendrite morphological neurons are a type of artificial neural network that works with min and max operators instead of algebraic products. These morphological operators build hyperboxes in N -dimensional space. These hyperboxes allow the proposal of training methods based on heuristics without using an optimisation method. In literature, it has been claimed that these heuristic-based trainings have advantages: there are no convergence problems, perfect classification can always be reached and training is performed in only one epoch. In this paper, we show that these assumed advantages come with a cost: these heuristics increase classification errors in the test set because they are not optimal and learning generalisation is poor. To solve these problems, we introduce a novel method to train dendrite morphological neurons based on stochastic gradient descent for classification tasks, using these heuristics just for initialisation of learning parameters. Experiments show that we can enhance the testing error in comparison with solely heuristic-based training methods. This approach can reach competitive performance with respect to other popular machine learning algorithms. [ABSTRACT FROM AUTHOR]

In this paper, a smooth function is constructed to approximate the nonsmooth output of max –min  fuzzy neural networks (FNNs) and its approximation is also presented. In place of the output of max –min  FNNs by its smoothing approximation function, the error function, defining the discrepancy between the actual outputs and desired outputs of max –min  FNNs, becomes a continuously differentiable function. Then, a smoothing gradient decent-based algorithm with Armijo–Goldstein step size rule is formulated to train max –min  FNNs. Based on the existing convergent result, the convergence of our proposed algorithm can easily be obtained. Furthermore, the proposed algorithm also provides a feasible procedure to solve fuzzy relational equations with max –min composition. Finally, some numerical examples are implemented to support our results and demonstrate that the proposed smoothing algorithm has better learning performance than other two gradient decent-based algorithms. [ABSTRACT FROM AUTHOR]

In this paper, the sampled-data control is applied to synchronize chaotic neural networks subject to actuator saturation. By employing a time-dependent Lyapunov functional that captures the characteristic information of actual sampling pattern, we derive a local stability condition for the synchronization error systems. By this condition, we design a sampled-data controller to regionally synchronize the drive neural networks and response neural networks subject to actuator saturation. Moreover, an optimization method is given to design the desired sampled-data controller such that the set of admissible initial conditions is maximized. A numerical example is given to demonstrate the effectiveness and merits of the proposed design technique. [ABSTRACT FROM AUTHOR]

In this paper, the optimal tracking control problem (OTCP) for a class of continuous-time nonlinear systems with infinite horizon cost is discussed. An online adaptive critic design method is proposed to learn the solution of OTCP by constructing an augmented system associated with a discounted performance function, which is composed of the tracking errors and reference trajectory dynamics. Only one neural network (NN) is used as critic module for approximating the performance function in the solution procedure, and thus the architecture is simpler than the typical action-critic structure, which needs more computational load from neural networks. Therefore, by the means of the approximate policy iteration, the tracking errors get converged to a region near zero and the parameters of critic module get converged to the optimal ones based on our proposed method. Both the convergence of the NN weights and the stability of the tracking error dynamics are guaranteed by the Lyapunov theory. Two simulation examples are proposed to verify the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Distinct from conventional approaches to generic vehicle type recognition, this paper proposes an approach to fine-grained vehicle type recognition that leverages multiple cues jointly to convolutional neural networks (CNNs). Fine-grained vehicle recognition is inherently difficult with many challenges, including (i) how to incorporate multiple cues through joint optimization instead of separately handling, (ii) how to learn data-driven representation for vehicles instead of hand-crafted features and (iii) how to perceive and localize the vehicle region instead of using the whole image. Our multi-task CNNs localize vehicles in the first stage and recognize subclasses in the second stage, allowing us to handle samples with cluttered backgrounds and those where vehicles do not fill most of the image. Through collaborative feature learning from multiple tasks in each stage, our approach can handle subtle inter-class variations at the subordinate level. We also develop new vehicle-oriented data augmentation strategies in CNN training. To advance research on vehicle-centered tasks, we release a new vehicle dataset that provides both semantic labels and bounding boxes. This dataset can be used in vehicle localization, recognition, viewpoint estimation, and so on. Extensive experiments on two benchmark vehicle datasets demonstrate that our approach outperforms state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

In this paper, we address the challenging task of simultaneously optimizing (i) the weights of a neural network, (ii) the number of neurons for each hidden layer, and (iii) the subset of active input features (i.e., feature selection). While these problems are traditionally dealt with separately, we propose an efficient regularized formulation enabling their simultaneous parallel execution, using standard optimization routines. Specifically, we extend the group Lasso penalty, originally proposed in the linear regression literature, to impose group-level sparsity on the network’s connections, where each group is defined as the set of outgoing weights from a unit. Depending on the specific case, the weights can be related to an input variable, to a hidden neuron, or to a bias unit, thus performing simultaneously all the aforementioned tasks in order to obtain a compact network. We carry out an extensive experimental evaluation, in comparison with classical weight decay and Lasso penalties, both on a toy dataset for handwritten digit recognition, and multiple realistic mid-scale classification benchmarks. Comparative results demonstrate the potential of our proposed sparse group Lasso penalty in producing extremely compact networks, with a significantly lower number of input features, with a classification accuracy which is equal or only slightly inferior to standard regularization terms. [ABSTRACT FROM AUTHOR]

This paper focuses on the issue of robust stability of artificial delayed neural networks. A free-matrix-based inequality strategy is produced by presenting an arrangement of slack variables, which can be optimized by means of existing convex optimization algorithms. To reflect a large portion of the dynamical behaviors of the framework, uncertain parameters are considered. By constructing an augmented Lyapunov functional, sufficient conditions are derived to guarantee that the considered neural systems are completely stable. The conditions are presented in the form of as linear matrix inequalities (LMIs). Finally, numerical cases are given to show the suitability of the results presented. [ABSTRACT FROM AUTHOR]

This paper is concerned with the non-fragile mixed H ∞ and passive asynchronous state estimation problem for uncertain discrete-time Markov jump neural networks (MJNNs). Both the uncertainties of system and the sensor nonlinearity are considered to be randomly occurring which are governed by a set of Bernoulli distributed white sequences. Since inaccuracies or uncertainties may occur in the designed state estimator and the complete mode synchronization between plant and state estimator is hardly possible, a non-fragile asynchronous state estimator design method is presented. By using an optimize matrix decoupling approach and Lyapunov-Krasovskii methodology, some sufficient conditions for the existence of non-fragile mixed H ∞ and passive asynchronous state estimator are proposed. A numerical example is presented to demonstrate the effectiveness of our proposed method. [ABSTRACT FROM AUTHOR]

Recently, inspired by nature, diversiform successful and effective optimization methods have been proposed for solving many complex and challenging applications in different domains. This paper proposes a new meta-heuristic technique, collective decision optimization algorithm (CDOA), for training artificial neural networks. It simulates the social behavior of human based on their decision-making characteristics including experience-based phase, others'-based phase, group thinking-based phase, leader-based phase and innovation-based phase. Different corresponding operators are designed in the methodology. Experimental results carried out on a comprehensive set of benchmark functions and two nonlinear function approximation examples demonstrate that CDOA is competitive with respect to other state-of-art optimization algorithms. [ABSTRACT FROM AUTHOR]

Short term power load forecasting plays an important role in the security of power system. In the past few years, application of artificial neural network (ANN) for short-term load forecasting (STLF) has become a research hotspots. Generalized regression neural network (GRNN) has been proved to be suitable for solving the non-linear problems. And according to the historical load curve, it can be known that STLF is a non-linear problem. Thus, the GRNN was used for STLF in this paper. However, the value of spread parameter σ determines the performance of the GRNN. The fruit fly optimization algorithm with decreasing step size (SFOA) is introduced to select an appropriate spread parameter σ . Combined with the weather factors and the periodicity of short-term load, an effective STLF model based on the GRNN with decreasing step FOA was proposed. Performance of the proposed SFOA-GRNN model is compared with other ANN on the basis of prediction error. [ABSTRACT FROM AUTHOR]

Mixed-signal neural networks have higher energy efficiency and lower area consumption when compared with their equivalent digital implementations. However, the signal processing precision in mixed-signal implementations is limited. A mixed-signal arithmetic method called Continuous Valued Number Systems (CVNS) is employed for development and implementation of different functions required in a neuronal network. Analog digits of this number system enable the higher accuracy of analog operations, and can efficiently relate with digital and binary values. In this paper, design and implementation of a 2-2-1 mixed-signal CVNS network structure is presented, to confirm the developed arithmetic method. In the proposed structure, weights are stored in the digital registers while the arithmetic is based on the CVNS. The CVNS features have been exploited to address the limited signal processing precision issue, which requires resolutions higher than 12-bit for online training of the network. As a result, the proposed structure satisfies this requirement of neural networks, while yielding an optimized network configuration. The proposed network is designed, fabricated, and tested in 0.18 μ m technology. [ABSTRACT FROM AUTHOR]

This paper presents a non-dominated sorting based multi-objective artificial bee colony algorithm NSABC to solve multi-objective optimization problems. It is an extension of the artificial bee colony algorithm ABC, which is a single objective optimization algorithm, to the multi-objective optimization domain. It uses a novel approach in the employee bee phase to steer the solutions to simultaneously achieve both the orthogonal goals in the multi-objective optimization – convergence and diversity. The onlooker bee phase is similar to the ABC except for the fitness computation to exploit the promising solutions whereas there is no change in the scout bee phase, which is used to get rid of bad solutions and add diversity in the swarm by introducing random solutions. Along with a novel way of exploring new solutions, it uses non-dominated sorting and crowding distance, inspired by the NSGA-II, to maintain the best and diverse solutions in the swarm. It is tested on the 10 two-objective and three-objective unconstrained benchmark problems of varying nature and complexities from the CEC09 suite of test problems and is found better than or commensurable to thirteen state-of-the-art significant multi-objective optimization algorithms as well as other multi-objective variants of the ABC. Further, it is tested on the nine real-life data clustering problems considered from the UCI machine learning repository and proved itself better in comparison to the NSGA-II, MOVGA, and a recent multi-objective variant of the ABC named MOABC. Thus, it is observed that the NSABC is comparatively a simple, light, and powerful algorithm to solve multi-objective problems. [ABSTRACT FROM AUTHOR]

Similarity-preserving hashing is a widely used method for nearest neighbor search in large-scale image retrieval. Recently, supervised hashing methods are appealing in that they learn compact hash codes with fewer bits by incorporating supervised information. In this paper, we propose a new two-stage supervised hashing methods which decomposes the hash learning process into a stage of learning approximate hash codes followed by a stage of learning hash functions. In the first stage, we propose a margin-based objective to find approximate hash codes such that a pair of hash codes associating to a pair of similar (dissimilar) images has sufficiently small (large) Hamming distance. This objective results in a challenging optimization problem. We develop a coordinate descent algorithm to efficiently solve this optimization problem. In the second stage, we use convolutional neural networks to learn hash functions. We conduct extensive evaluations on several benchmark datasets with different kinds of images. The results show that the proposed margin-based hashing method has substantial improvement upon the state-of-the-art supervised or unsupervised hashing methods. [ABSTRACT FROM AUTHOR]

Point predictions of wind speed can hardly be reliable and accurate when the uncertainty level increases in data. Prediction intervals (PIs) provide a solution to quantify the uncertainty associated with point predictions. In this paper, we adopt radial basis function (RBF) neural networks to perform interval forecasting of the future wind speed. A two-step method is proposed to determine the RBF connection weights in a multi-objective optimization framework. In the first step, the centers of the RBF are determined using the K -means clustering algorithm and the hidden-output weights of the RBF are pre-trained using the least squares algorithm. In the second step, the hidden-output weights are further adjusted by the non-dominated sorting genetic algorithm-II (NSGA-II), which aims at concurrently minimizing the width and maximizing the coverage probability of the constructed intervals. We test the performance of the proposed method on three real data sets, which are collected from different wind farms in China. The experimental results indicate that the proposed method can provide higher quality PIs than the conventional multi-layer perceptron (MLP) based methods. [ABSTRACT FROM AUTHOR]

According to the thought “divide and conquer” that human perceives complicated things from multi-side and multi-view and balances the final decision, this paper puts forward the multi-sided multi-granular neural network ensemble optimization method based on feature selection, which divides attribute granularity of dataset from multi-side, and structures multi-granular individual neural networks using different attribute granularity and the corresponding subsets. In this way, we can gain multi-granular individual neural networks with greater diversity, and get better performance of neural network ensemble(NNE). Firstly, use feature selection method to calculate the importance of each attribute, according to the average weight to choose some attributes whose average weight is greater than a certain threshold, to form an attribute granularity and the corresponding sample subset, thus to construct an individual neural network. If samples are not properly identified, this attribute granularity is weak for the generalization ability of the sample. Secondly, again calculate the importance of the attributes of samples not properly identified, choose the attributes that can generalize the corresponding samples better, and add to the last attribute granularity to form a new attribute granularity, and at the same time random choose two-thirds of sample subset to construct an individual neural network. In turn, one can get a series of attribute granularities and the corresponding sample subsets and a series of multi-granular individual neural networks. These attribute granularities and the corresponding sample subsets constructed from multi-side and multi-view with greater diversity can construct multi-granular individual neural networks with greater diversity. This method not only reduces the dimension of the dataset, but also makes the attribute granularity to identify the corresponding sample as large as possible. Finally, by calculating the diversity of each of the two individual neural networks, optimal selects some individual neural networks with greater diversity to ensemble. The simulation experiments show that our proposed method here, multi-side multi-granular neural network ensemble optimization method, can gain better performance. [ABSTRACT FROM AUTHOR]

This paper introduces an effective approach to study the stability of static neural networks with interval time-varying delay using delay partitioning approach and tighter integral inequality lemma. By decomposing the delay interval into multiple equidistant subintervals and multiple nonuniform subintervals, some suitable Lyapunov–Krasovskii functionals are constructed on these intervals. A set of novel sufficient conditions are obtained to guarantee the stability analysis issue for the considered system. These conditions are expressed in the framework of linear matrix inequalities, which heavily depend on the lower and upper bounds of the time-varying delay. It is shown, by comparing with existing approaches, that the delay-partitioning approach can largely reduce the conservatism of the stability results. Finally, three examples are given to show the effectiveness of the theoretical results. [ABSTRACT FROM AUTHOR]

This paper is concerned with the problems of finite-time filter analysis and design for discrete time-delay neural networks with Markovian jump parameters and sector-bounded activation functions represented by Takagi–Sugeno fuzzy model. Firstly, a sufficient admissibility criterion is presented to guarantee that the augmented fuzzy jump neural network without parameter uncertainties is stochastically finite-time bounded in a prescribed time interval. From the admissibility criterion obtained, a sufficient condition on stochastic finite-time boundedness is provided for the augmented fuzzy jump neural networks. Then, a sufficient criterion to design a finite-time fuzzy filter is presented with uncertain parameters and Markovian jumps for discrete time-delay fuzzy neural networks. Moreover, conditions on stochastic finite-time stability are also established for nominal and uncertain delayed fuzzy jump neural networks without the presence of external disturbance. All criteria obtained can be represented as the form of linear matrix inequalities. Finally, numerical examples are given to illustrate the effectiveness of the obtained results. [ABSTRACT FROM AUTHOR]

This paper presents an autopilot design for a robotic unmanned surface vehicle in the presence of unknown yaw dynamics and measurement noises. A robust adaptive steering law is developed with the aid of a predictor, neural networks, and a modified dynamic surface control technique. Specifically, a predictor together with a low-frequency learning-based neural updating law is developed to identify the unknown yaw dynamics, as well as to reconstruct the states corrupted by measurement noises. Besides, to avoid the noise amplification effect of first-order filter, a linear tracking differentiator is incorporated into the dynamic surface control design approach to produce a noise-tolerant estimate of virtual control derivative. The stability of the closed-loop autopilot system is established by Lyapunov analysis. A salient feature of the developed controller is that it can achieve the desired performance in the presence of model uncertainty and measurement noises, simultaneously. Both simulation and experiment results are given to validate the efficacy of the proposed method. [ABSTRACT FROM AUTHOR]

This paper addresses the novel architectures׳ formulation and linear matrix inequality (LMI)-based design of dynamic nonlinear anti-windup compensator (AWC) for nonlinear time-delay systems with continuous interval time-varying delays under input saturation. An internal model control (IMC)-based AWC architecture is suggested for stable nonlinear time-delay systems and, in addition, a decoupling AWC architecture, applicable to a broader class of nonlinear time-delay systems, is proposed for compensation of the undesirable saturation effects. Further, a correspondent decoupled architecture is derived and recommended for characterizing the delayed nonlinear AWC synthesis goals. By employing Lyapunov–Krasovskii functional, local sector condition, Lipschitz condition, L 2 gain minimization, and the delay-interval information, several sufficient conditions are derived for the design of nonlinear time-delay AWC. Numerical examples for FitzHugh–Nagumo neuron and Hopfield neural network under input saturation and time-delay are presented to reveal effectiveness of the proposed anti-windup approach. [ABSTRACT FROM AUTHOR]

This paper provides an undulatory locomotion model of C. elegans to achieve the chemotaxis behaviors based on the biological neuronal and neuromuscular structure. The on-cell and off-cell mechanism, as well as the proprioceptive mechanism is incorporated into the locomotion model. The nervous system of C. elegans is modeled by a dynamic neural network (DNN) that involves two parts: head DNN and motor neurons. The head DNN perceives the outside concentrations and generates the undulatory wave to the body. The motor neurons are responsible for transiting the undulatory wave along the body. The body of C. elegans is represented as a multi-joint rigid link model with 11 links. The undulatory locomotion behavior is achieved by using the DNN to control the lengths of muscles on ventral and dorsal sides, and then using the muscle lengths to control the angles between two consecutive links. In this work, the relations between the outputs of DNN and muscle lengths, as well as the muscle lengths and the angles between two consecutive links, are determined. Furthermore, owing to the learning capability of DNN, a set of nonlinear functions that are designed to represent the chemotaxis behaviors of C. elegans are learned by the head DNN. The testing results show good performance of the locomotion model for the chemotaxis behaviors of finding food and avoiding toxin, as well as slight and Ω turns. At last, quantitative analyses by comparing with the experiment results are provided to verify the realness and effectiveness of the locomotion model, which could serve as a prototype for other footless animals. [ABSTRACT FROM AUTHOR]

Dynamic model is an important issue for research on stability, dynamic characteristics, surge and control technique of axial-flow pumping system, and such a model is usually characterized by complex nonlinearity, strong coupling and time-varying mathematical equation. For the convenience of establishing model and highly effective computing, dynamic characteristics of the whole system are divided into four parts: pump lift-flow characteristics, pipeline characteristics, mechanical characteristics of asynchronous motor and torque characteristics of pump load. Each part is a nonlinear subsystem, and there are complex coupling relations among each other. In the paper, each part of the pump system is modeled respectively by mechanisms of hydrodynamics, transmission dynamics, electromechanics and affinity law. Considering that the axial-flow pump is characterized by nonlinearity and parameters are difficultly estimated in the low flow operation area and that the data of pump head-flow can be easily tested under the speed of power frequency, a modeling method combined with the RBF neural network is proposed, where hidden layer parameters are optimized by K means clustering algorithm, and the weights are trained by least square method. At last the whole simulation model of the axial-flow pumping system is set up, and the validity of the proposed modeling method is verified through simulation. [ABSTRACT FROM AUTHOR]

A unified associative memory model with a novel method for designing associative memories is presented in this paper. Based on continuous recurrent neural networks, bipolar patterns inputted from external can cause the output of neural networks to be memorized patterns. In the method, two conditions relevant to external inputs are derived to ensure the network states converge to a stable interval, and an exponential stable criterion is proposed for the network being a bipolar associative memory with higher recall speed. By introducing a tunable slope activation function and considering time delay, the proposed model is general and can recall the memorized patterns in auto-associative and hetero-associative way, while higher robust and more flexible memory can be obtained through the proposed method. Experimental verification demonstrates the effectiveness and generalization of the proposed method. [ABSTRACT FROM AUTHOR]

This paper proposes an improved stochastic second order learning algorithm for supervised neural network training. The proposed algorithm, named bounded stochastic diagonal Levenberg–Marquardt (B-SDLM), utilizes both gradient and curvature information to achieve fast convergence while requiring only minimal computational overhead than the stochastic gradient descent (SGD) method. B-SDLM has only a single hyperparameter as opposed to most other learning algorithms that suffer from the hyperparameter overfitting problem due to having more hyperparameters to be tuned. Experiments using the multilayer perceptron (MLP) and convolutional neural network (CNN) models have shown that B-SDLM outperforms other learning algorithms with regard to the classification accuracies and computational efficiency (about 5.3% faster than SGD on the mnist-rot-bg-img database). It can classify all testing samples correctly on the face recognition case study based on AR Purdue database. In addition, experiments on handwritten digit classification case studies show that significant improvements of 19.6% on MNIST database and 17.5% on mnist-rot-bg-img database can be achieved in terms of the testing misclassification error rates (MCRs). The computationally expensive Hessian calculations are kept to a minimum by using just 0.05% of the training samples in its estimation or updating the learning rates once per two training epochs, while maintaining or even achieving lower testing MCRs. It is also shown that B-SDLM works well in the mini-batch learning mode, and we are able to achieve 3.32 × performance speedup when deploying the proposed algorithm in a distributed learning environment with a quad-core processor. [ABSTRACT FROM AUTHOR]

One very important component of Hardware implementations of Artificial Neural Networks (HNN) is the activation function. A common solution for activation function design in HNN is the Look-Up Table (LUT) representation that has acceptable area constraints and usually achieves acceptable execution time. Tools and techniques for automatic generation of LUTs have been developed but have some problems as high complexity, high dependency of external tools or languages and the optimization techniques that are not efficient enough for critical applications. In order to solve some of the presented problems this paper describes a method for automatic LUT generation that overcomes some deficiencies of existing methods using a simple approach. This method was evaluated initially considering the xor function and the sine function for evaluation purposes. After these initial tests, two other ANN based robotic systems were tested. These robotic systems will be implemented in hardware in the near future and they demonstrate a different behavior using different domains of the same activation function. Proposed method achieved good results that are directly related to how neural networks define the activation function domain. [ABSTRACT FROM AUTHOR]

Multi-instance learning (MIL) is one of promising paradigms in the supervised learning aiming to handle real world classification problems where a classification target contains several featured sections, e.g., an image typically contains several salient regions. In this paper, we propose a highly efficient learning method for MI classification based on hierarchical extreme learning machine (ELM), called MI-ELM. Specifically, a double-hidden layers feedforward network (DLFN) is designed to serve as the MI classifier. Then, the MI classification is formulated as an optimization problem. Moreover, the output weights of DLFN can be analytically determined by solving the aforementioned optimization problem. The merits of MI-ELM are as follows: (i) MI-ELM extends the single-layer ELM to be a hierarchical one that well fits for training DLFNs in MI classification. (ii) The input and hidden-layer parameters of DLFNs are assigned randomly rather than tuned iteratively, and the output weights of DLFNs can be determined analytically in one step. Therefore, MI-ELM significantly enhances the efficiency of the DLFN without notable loss of the classification accuracy. Experimental results over several real-world data sets demonstrate that the proposed MI-ELM method significantly outperforms existing kernel methods for MI classification in terms of the classification accuracy and the classification time. [ABSTRACT FROM AUTHOR]

Radial Basis Function Neural Networks (RBFNNs) have been widely used for classification and function approximation tasks. Hence, it is worthy to try improving and developing new learning algorithms for RBFNNs in order to get better results. This paper presents a new learning method for RBFNNs. An improved algorithm for center adjustment of RBFNNs and a novel algorithm for width determination have been proposed to optimize the efficiency of the Optimum Steepest Decent (OSD) algorithm. To initialize the radial basis function units more accurately, a modified approach based on Particle Swarm Optimization (PSO) is presented. The obtained results show fast convergence speed, better and same network response in fewer train data which states the generalization power of the improved neural network. The Improved PSO–OSD and Three-phased PSO–OSD algorithms have been tested on five benchmark problems and the results have been compared. Finally, using the improved radial basis function neural network we propose a new method for object image retrieval. The images to be retrieved are object images that can be divided into foreground and background. Experimental results show that the proposed method is really promising and achieves high performance. [ABSTRACT FROM AUTHOR]

To deal with equalities-constrained nonconvex optimization problem, an intelligence method of swarm neural networks (SNN) is introduced in this paper. The proposed method handles the problem into two parts, which combines local searching ability of one-layer recurrent neural network (RNN) and global searching ability of shuffled frog leaping algorithm (SFLA). First, a RNN model based on general nonconvex optimization is presented. Then the convergence property of RNN is analyzed and proven. Moreover, based on SFLA framework, neural networks are treated as frogs which must be divided into several memeplexes and evolve by their own differential equations to search a local exact solution. Next, through shuffling the best solution of each memeplex, we can obtain the global best point. Finally, numerical examples with simulation results are given to illustrate the effectiveness and good characteristics of the proposed method solving nonconvex optimization problem. [ABSTRACT FROM AUTHOR]

Brain Computer Interface (BCI) has been applied to augment impaired human cognitive function by converting mental signals into control signals. This paper presents a neural classifier optimized using Backtracking Search optimization Algorithm (BSANN) to classify three mental tasks consisting of right or left hand movement imagination and generation of word. BSA is an Evolutionary Algorithm (EA) which is suitable for deciphering non-linear and non-differentiable problems. Single control parameter gives BSA an upshot over other EA due to the lower degree of randomness. BSA keeps memory of old population to generate a new candidate set i.e. solution, so it gets the advantage of utilizing the search results of the previous population. The proposed method (BSANN) has been tested on the publicly available datasets of BCI Competition 3-5. Experimental result shows that BSANN exhibits better results than 21 other algorithms for classification of mental tasks in terms of classification accuracy. [ABSTRACT FROM AUTHOR]

In this paper, a new fuzzy neural network model with correlated fuzzy rules (CFNN) based on the Levenberg–Marquardt ( LM ) optimization method is proposed. The proposed method is a new fuzzy network structure that is presented to approximate nonlinear functions especially the functions with high correlation between input variables with less number of fuzzy rules. A multivariable Gaussian fuzzy membership function is introduced that can consider the correlation between input variables and consequently it can model non-separable relations for interactive variables. The LM optimization method is used to learn parameters of both premise and consequent parts of the fuzzy rules. The suggested algorithm is successfully applied to seven tested examples including static function approximation, time-series prediction, non-linear dynamic system identification and a real-world complex regression problem. According to test observations; it can approximate nonlinear functions better than the other past algorithms with more compact structure and less number of fuzzy rules. [ABSTRACT FROM AUTHOR]

This paper is concerned with the H ∞ state estimation problem for a class of discrete-time neural networks with time-varying delays, randomly occurring quantizations (ROQs) as well as missing measurements. The phenomena of ROQ and missing measurements are governed by a Bernoulli distributed stochastic sequence. The purpose of the addressed problem is to design a state estimator such that the dynamics of the estimation error is exponentially stable in the mean square and the prescribed H ∞ performance constraint is satisfied. By constructing proper Lyapunov–Krasovskii functionals and employing stochastic analysis techniques, sufficient conditions are derived to ensure the existence of the desired estimator. Furthermore, the explicit expression of the gain of the desired estimator is described in terms of the solution to a convex optimization problem that can be easily solved by using the semi-definite programme method. Finally, a numerical example is employed to demonstrate the effectiveness and applicability of the proposed estimator design approach. [ABSTRACT FROM AUTHOR]