Search

Prospective authors are requested to submit new, unpublished manuscripts for inclusion in the upcoming event described in this call for papers. [ABSTRACT FROM AUTHOR]

Describes the above-named upcoming conference event. May include topics to be covered or calls for papers. [ABSTRACT FROM PUBLISHER]

Conservation principles, such as conservation of charge, energy, or mass, provide a natural way to couple and constrain spatially separated variables. In this paper, we propose a dynamical system model that exploits these constraints for solving nonconvex and discrete global optimization problems. Unlike the traditional simulated annealing or quantum annealing-based global optimization techniques, the proposed method optimizes a target objective function by continuously evolving a driver functional over a conservation manifold, using a generalized variant of growth transformations. As a result, the driver functional asymptotically converges toward a Dirac-delta function that is centered at the global optimum of the target objective function. In this paper, we provide an outline of the proof of convergence for the dynamical system model and investigate different properties of the model using a benchmark nonlinear optimization problem. Also, we demonstrate how a discrete variant of the proposed dynamical system can be used for implementing decentralized optimization algorithms, where an ensemble of spatially separated entities (for example, biological cells or simple computational units) can collectively implement specific functions, such as winner-take-all and ranking, by exchanging signals only with its immediate substrate or environment. The proposed dynamical system model could potentially be used to implement continuous-time optimizers, annealers, and neural networks. [ABSTRACT FROM AUTHOR]

Hyperdimensional (HD) computing is a promising paradigm for future intelligent electronic appliances operating at low power. This paper discusses tradeoffs of selecting parameters of binary HD representations when applied to pattern recognition tasks. Particular design choices include density of representations and strategies for mapping data from the original representation. It is demonstrated that for the considered pattern recognition tasks (using synthetic and real-world data) both sparse and dense representations behave nearly identically. This paper also discusses implementation peculiarities which may favor one type of representations over the other. Finally, the capacity of representations of various densities is discussed. [ABSTRACT FROM AUTHOR]

With explosive growth of data volume and ever-increasing diversity of data modalities, cross-modal similarity search, which conducts nearest neighbor search across different modalities, has been attracting increasing interest. This paper presents a deep compact code learning solution for efficient cross-modal similarity search. Many recent studies have proven that quantization-based approaches perform generally better than hashing-based approaches on single-modal similarity search. In this paper, we propose a deep quantization approach, which is among the early attempts of leveraging deep neural networks into quantization-based cross-modal similarity search. Our approach, dubbed shared predictive deep quantization (SPDQ), explicitly formulates a shared subspace across different modalities and two private subspaces for individual modalities, and representations in the shared subspace and the private subspaces are learned simultaneously by embedding them to a reproducing kernel Hilbert space, where the mean embedding of different modality distributions can be explicitly compared. In addition, in the shared subspace, a quantizer is learned to produce the semantics preserving compact codes with the help of label alignment. Thanks to this novel network architecture in cooperation with supervised quantization training, SPDQ can preserve intramodal and intermodal similarities as much as possible and greatly reduce quantization error. Experiments on two popular benchmarks corroborate that our approach outperforms state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In the first issue of Nature 2015, Google DeepMind published a paper “Human-level control through deep reinforcement learning.” Furthermore, in the first issue of Nature 2016, it published a cover paper “Mastering the game of Go with deep neural networks and tree search” and proposed the computer Go program, AlphaGo. In March 2016, AlphaGo beat the world’s top Go player Lee Sedol by 4:1. This becomes a new milestone in artificial intelligence history, the core of which is the algorithm of deep reinforcement learning (RL). [ABSTRACT FROM AUTHOR]

The most popular tools for stream data mining are based on decision trees. In previous 15 years, all designed methods, headed by the very fast decision tree algorithm, relayed on Hoeffding’s inequality and hundreds of researchers followed this scheme. Recently, we have demonstrated that although the Hoeffding decision trees are an effective tool for dealing with stream data, they are a purely heuristic procedure; for example, classical decision trees such as ID3 or CART cannot be adopted to data stream mining using Hoeffding’s inequality. Therefore, there is an urgent need to develop new algorithms, which are both mathematically justified and characterized by good performance. In this paper, we address this problem by developing a family of new splitting criteria for classification in stationary data streams and investigating their probabilistic properties. The new criteria, derived using appropriate statistical tools, are based on the misclassification error and the Gini index impurity measures. The general division of splitting criteria into two types is proposed. Attributes chosen based on type- $I$ splitting criteria guarantee, with high probability, the highest expected value of split measure. Type- $II$ criteria ensure that the chosen attribute is the same, with high probability, as it would be chosen based on the whole infinite data stream. Moreover, in this paper, two hybrid splitting criteria are proposed, which are the combinations of single criteria based on the misclassification error and Gini index. [ABSTRACT FROM AUTHOR]

Batch normalization (BN) has recently become a standard component for accelerating and improving the training of deep neural networks (DNNs). However, BN brings in additional calculations, consumes more memory, and significantly slows down the training iteration. Furthermore, the nonlinear square and sqrt operations in the normalization process impede low bit-width quantization techniques, which draw much attention to the deep learning hardware community. In this paper, we propose an $L1$ -norm BN (L1BN) with only linear operations in both forward and backward propagations during training. L1BN is approximately equivalent to the conventional $L2$ -norm BN (L2BN) by multiplying a scaling factor that equals $({\pi }/{2})^{1/2}$. Experiments on various convolutional neural networks and generative adversarial networks reveal that L1BN can maintain the same performance and convergence rate as L2BN but with higher computational efficiency. In real application-specified integrated circuit synthesis with reduced resources, L1BN achieves 25% speedup and 37% energy saving compared to the original L2BN. Our hardware-friendly normalization method not only surpasses L2BN in speed but also simplifies the design of deep learning accelerators. Last but not least, L1BN promises a fully quantized training of DNNs, which empowers future artificial intelligence applications on mobile devices with transfer and continual learning capability. [ABSTRACT FROM AUTHOR]

In this paper, we consider the composition optimization with two expected-value functions in the form of $({1}/{n})\sum _{i = 1}^{n} F_{i}\left({({1}/{m})\sum _{j = 1}^{m} G_{j}(x)}\right)+R(x)$ , which formulates many important problems in statistical learning and machine learning such as solving Bellman equations in reinforcement learning and nonlinear embedding. Full gradient- or classical stochastic gradient descent-based optimization algorithms are unsuitable or computationally expensive to solve this problem due to the inner expectation $({1}/{m})\sum _{j = 1}^{m} G_{j}(x)$. We propose a dualityfree-based stochastic composition method that combines the variance reduction methods to address the stochastic composition problem. We apply the stochastic variance reduction gradient- and stochastic average gradient algorithm-based methods to estimate the inner function and the dualityfree method to estimate the outer function. We prove the linear convergence rate not only for the convex composition problem but also for the case that the individual outer functions are nonconvex, while the objective function is strongly convex. We also provide the results of experiments that show the effectiveness of our proposed methods. [ABSTRACT FROM AUTHOR]

Since neural networks renaissance, convolutional neural networks (ConvNets) have demonstrated a state-of-the-art performance in several emerging artificial intelligence tasks. The deployment of ConvNets in real-life applications requires power-efficient designs that meet the application-level performance needs. In this context, field-programmable gate arrays (FPGAs) can provide a potential platform that can be tailored to application-specific requirements. However, with the complexity of ConvNet models increasing rapidly, the ConvNet-to-FPGA design space becomes prohibitively large. This paper presents fpgaConvNet, an end-to-end framework for the optimized mapping of ConvNets on FPGAs. The proposed framework comprises an automated design methodology based on the synchronous dataflow (SDF) paradigm and defines a set of SDF transformations in order to efficiently navigate the architectural design space. By proposing a systematic multiobjective optimization formulation, the presented framework is able to generate hardware designs that are cooptimized for the ConvNet workload, the target device, and the application’s performance metric of interest. Quantitative evaluation shows that the proposed methodology yields hardware designs that improve the performance by up to $6.65\times $ over highly optimized graphics processing unit designs for the same power constraints and achieve up to $2.94\times $ higher performance density compared with the state-of-the-art FPGA-based ConvNet architectures. [ABSTRACT FROM AUTHOR]

Most feature selection methods first compute a similarity matrix by assigning a fixed value to pairs of objects in the whole data or to pairs of objects in a class or by computing the similarity between two objects from the original data. The similarity matrix is fixed as a constant in the subsequent feature selection process. However, the similarities computed from the original data may be unreliable, because they are affected by noise features. Moreover, the local structure within classes cannot be recovered if the similarities between the pairs of objects in a class are equal. In this paper, we propose a novel local adaptive projection (LAP) framework. Instead of computing fixed similarities before performing feature selection, LAP simultaneously learns an adaptive similarity matrix $\mathbf{S}$ and a projection matrix $\mathbf{W}$ with an iterative method. In each iteration, $\mathbf{S}$ is computed from the projected distance with the learned $\mathbf{W}$ and W is computed with the learned $\mathbf{S}$. Therefore, LAP can learn better projection matrix $\mathbf{W}$ by weakening the effect of noise features with the adaptive similarity matrix. A supervised feature selection with LAP (SLAP) method and an unsupervised feature selection with LAP (ULAP) method are proposed. Experimental results on eight data sets show the superiority of SLAP compared with seven supervised feature selection methods and the superiority of ULAP compared with five unsupervised feature selection methods. [ABSTRACT FROM AUTHOR]

This paper investigates the exponential stability analysis issue for a class of delayed recurrent neural networks (RNNs) with semi-Markovian parameters. By constructing a stochastic Lyapunov functional and using some zoom techniques to estimate its weak infinitesimal operator, the exponential mean square stability criteria have been proposed for the Markovian neural networks with certain transition probabilities. We then generalize the homogeneous polynomial approach for the delayed Markovian RNNs with uncertain transition probabilities during the stability analysis. Theoretical results have obtained by introducing an appropriate technique for dealing with a large number of complex homogeneous polynomial matrix inequalities. Finally, numerical examples are provided to demonstrate the effectiveness of the proposed technique. [ABSTRACT FROM AUTHOR]

The pervasiveness of wearable devices furnished with state-of-the-art sensors has shown the powerful capability in context-aware applications. However, embedded sensors also become targets for adversaries to launch potential side-channel attacks. In this paper, we present a self-adaptive and pretraining-independent pattern attack that infers a graphical password by recovering the victim’s hand movement trajectory via motion sensors of a wrist-worn smart device. With the adaptive pattern inference algorithm, the discovered attack can be launched remotely without requiring previous training data from victims or the prior knowledge about the keyboard input settings. Toward the proposed attack, we create a method to detect the sliding behavior that draws a graphical password on the screen. We also propose an inference algorithm to generate password candidates from hand movement trajectories for different keypad input settings. We implement the discovered attack on a smartwatch and conduct experiments to evaluate the impact of this attack. The evaluation results show that for complex graphical patterns, with a single try, the attack can infer the passwords at a success rate as high as 80%, and the success rate can be further boosted to over 90% within five attempts, which reveals the overlooked privacy information threat caused by sensor data leakage. [ABSTRACT FROM AUTHOR]

In this paper, kurtosis-based complex-valued real-time recurrent learning (KCRTRL) and kurtosis-based augmented CRTRL (KACRTRL) algorithms are proposed for training fully connected recurrent neural networks (FCRNNs) in the complex domain. These algorithms are designed by minimizing the cost functions based on the kurtosis of a complex-valued error signal. The KCRTRL algorithm exploits the circularity properties of the complex-valued signals, and this algorithm not only provides a faster convergence rate but also results in a lower steady-state error. However, the KCRTRL algorithm is suboptimal in the processing of noncircular (NC) complex-valued signals. On the other hand, the KACRTRL algorithm contains a complete second-order information due to the augmented statistics, thus considerably improves the performance of the FCRNN in the processing of NC complex-valued signals. Simulation results on the one-step-ahead prediction problems show that the proposed KCRTRL algorithm significantly enhances the performance for only circular complex-valued signals, whereas the proposed KACRTRL algorithm provides more superior performance than existing algorithms for NC complex-valued signals in terms of the convergence rate and the steady-state error. [ABSTRACT FROM AUTHOR]

Deep convolutional neural networks (CNNs) have become a key element in the recent breakthrough of salient object detection. However, existing CNN-based methods are based on either patchwise (regionwise) training and inference or fully convolutional networks. Methods in the former category are generally time-consuming due to severe storage and computational redundancies among overlapping patches. To overcome this deficiency, methods in the second category attempt to directly map a raw input image to a predicted dense saliency map in a single network forward pass. Though being very efficient, it is arduous for these methods to detect salient objects of different scales or salient regions with weak semantic information. In this paper, we develop hybrid contrast-oriented deep neural networks to overcome the aforementioned limitations. Each of our deep networks is composed of two complementary components, including a fully convolutional stream for dense prediction and a segment-level spatial pooling stream for sparse saliency inference. We further propose an attentional module that learns weight maps for fusing the two saliency predictions from these two streams. A tailored alternate scheme is designed to train these deep networks by fine-tuning pretrained baseline models. Finally, a customized fully connected conditional random field model incorporating a salient contour feature embedding can be optionally applied as a postprocessing step to improve spatial coherence and contour positioning in the fused result from these two streams. Extensive experiments on six benchmark data sets demonstrate that our proposed model can significantly outperform the state of the art in terms of all popular evaluation metrics. [ABSTRACT FROM AUTHOR]

Nowadays, the control technology of the robotic manipulator with flexible joints (RMFJ) is not mature enough. The flexible-joint manipulator dynamic system possesses many uncertainties, which brings a great challenge to the controller design. This paper is motivated by this problem. In order to deal with this and enhance the system robustness, the full-state feedback neural network (NN) control is proposed. Moreover, output constraints of the RMFJ are achieved, which improve the security of the robot. Through the Lyapunov stability analysis, we identify that the proposed controller can guarantee not only the stability of flexible-joint manipulator system but also the boundedness of system state variables by choosing appropriate control gains. Then, we make some necessary simulation experiments to verify the rationality of our controllers. Finally, a series of control experiments are conducted on the Baxter. By comparing with the proportional–derivative control and the NN control with the rigid manipulator model, the feasibility and the effectiveness of NN control based on flexible-joint manipulator model are verified. [ABSTRACT FROM AUTHOR]

In this paper, we present an approach for minimizing the computational complexity of the trained convolutional neural networks (ConvNets). The idea is to approximate all elements of a given ConvNet and replace the original convolutional filters and parameters (pooling and bias coefficients; and activation function) with an efficient approximations capable of extreme reductions in computational complexity. Low-complexity convolution filters are obtained through a binary (zero and one) linear programming scheme based on the Frobenius norm over sets of dyadic rationals. The resulting matrices allow for multiplication-free computations requiring only addition and bit-shifting operations. Such low-complexity structures pave the way for low power, efficient hardware designs. We applied our approach on three use cases of different complexities: 1) a “light” but efficient ConvNet for face detection (with around 1000 parameters); 2) another one for hand-written digit classification (with more than 180 000 parameters); and 3) a significantly larger ConvNet: AlexNet with $\approx 1.2$ million matrices. We evaluated the overall performance on the respective tasks for different levels of approximations. In all considered applications, very low-complexity approximations have been derived maintaining an almost equal classification performance. [ABSTRACT FROM AUTHOR]

This paper focuses on the event-triggered guaranteed cost control design of nonlinear systems via a self-learning technique. In brief, an event-based guaranteed cost control strategy of nonlinear systems subjects to matched uncertainties is developed, thereby balancing the performance of guaranteed cost and the actuality of limited communication resource. The original control design is transformed into an optimal control problem with an event-based mechanism, where the relationship of guaranteed cost performance compared to the time-based formulation is discussed. A critic neural network is constructed for implementing the event-based optimal control design with stability guarantee. Simulation experiments are carried out to verify the theoretical results in detail. [ABSTRACT FROM AUTHOR]

Most existing object detection algorithms are trained based upon a set of fully annotated object regions or bounding boxes, which are typically labor-intensive. On the contrary, nowadays there is a significant amount of image-level annotations cheaply available on the Internet. It is hence a natural thought to explore such “weak” supervision to benefit the training of object detectors. In this paper, we propose a novel scheme to perform weakly supervised object localization, termed object-specific pixel gradient (OPG). The OPG is trained by using image-level annotations alone, which performs in an iterative manner to localize potential objects in a given image robustly and efficiently. In particular, we first extract an OPG map to reveal the contributions of individual pixels to a given object category, upon which an iterative mining scheme is further introduced to extract instances or components of this object. Moreover, a novel average and max pooling layer is introduced to improve the localization accuracy. In the task of weakly supervised object localization, the OPG achieves a state-of-the-art 44.5% top-5 error on ILSVRC 2013, which outperforms competing methods, including Oquabet al.and region-based convolutional neural networks on the Pascal VOC 2012, with gains of 2.6% and 2.3%, respectively. In the task of object detection, OPG achieves a comparable performance of 27.0% mean average precision on Pascal VOC 2007. In all experiments, the OPG only adopts the off-the-shelf pretrained CNN model, without using any object proposals. Therefore, it also significantly improves the detection speed, i.e., achieving three times faster compared with the state-of-the-art method. [ABSTRACT FROM AUTHOR]

The robust Huber’s M-estimator is widely used in signal and image processing, classification, and regression. From an optimization point of view, Huber’s M-estimation problem is often formulated as a large-sized quadratic programming (QP) problem in view of its nonsmooth cost function. This paper presents a generalized regression estimator which minimizes a reduced-sized QP problem. The generalized regression estimator may be viewed as a significant generalization of several robust regression estimators including Huber’s M-estimator. The performance of the generalized regression estimator is analyzed in terms of robustness and approximation accuracy. Furthermore, two low-dimensional recurrent neural networks (RNNs) are introduced for robust estimation. The two RNNs have low model complexity and enhanced computational efficiency. Finally, the experimental results of two examples and an application to image restoration are presented to substantiate superior performance of the proposed method over conventional algorithms for robust regression estimation in terms of approximation accuracy and convergence rate. [ABSTRACT FROM AUTHOR]

An intelligent data-driven predictive control strategy is proposed in this paper. The predictive controller is designed by combining predictive control and local weighted projection regression. The presented control strategy needs less prior knowledge and has fewer parameters that are hard to determine compared to other data-driven predictive controller, e.g., the one in dynamic partial least square (PLS) framework. Furthermore, the proposed predictive controller performs better in the control of nonlinear processes and is able to update its parameters based on the online data. The predictive model validity and intelligence of the control strategy are guaranteed by the online updating strategy to a certain degree. The control performance of the proposed predictive controller against the model predictive control (MPC) in dynamic PLS framework is illustrated through the simulation of a typical numerical example and the benchmark of a continuous stirred tank heater system. It can be observed from the simulation that the proposed MPC strategy has higher prediction precision and stronger ability in coping with nonlinear dynamic processes which are quite common in practical applications, for instance, the industrial process. [ABSTRACT FROM AUTHOR]

Image captioning aims to generate natural language sentences to describe the salient parts of a given image. Although neural networks have recently achieved promising results, a key problem is that they can only describe concepts seen in the training image-sentence pairs. Efficient learning of novel concepts has thus been a topic of recent interest to alleviate the expensive manpower of labeling data. In this paper, we propose a novel method,Image-Text Surgery, to synthesize pseudoimage-sentence pairs. The pseudopairs are generated under the guidance of a knowledge base, with syntax from a seed data set (i.e., MSCOCO) and visual information from an existing large-scale image base (i.e., ImageNet). Via pseudodata, the captioning model learns novel concepts without any corresponding human-labeled pairs. We further introduce adaptive visual replacement, which adaptively filters unnecessary visual features in pseudodata with an attention mechanism. We evaluate our approach on a held-out subset of the MSCOCO data set. The experimental results demonstrate that the proposed approach provides significant performance improvements over state-of-the-art methods in terms of F1 score and sentence quality. An ablation study and the qualitative results further validate the effectiveness of our approach. [ABSTRACT FROM AUTHOR]

As the interest in event-based vision sensors for mobile and aerial applications grows, there is an increasing need for high-speed and highly robust algorithms for performing visual tasks using event-based data. As event rate and network structure have a direct impact on the power consumed by such systems, it is important to explore the efficiency of the event-based encoding used by these sensors. The work presented in this paper represents the first study solely focused on the effects of both spatial and temporal downsampling on event-based vision data and makes use of a variety of data sets chosen to fully explore and characterize the nature of downsampling operations. The results show that both spatial downsampling and temporal downsampling produce improved classification accuracy and, additionally, a lower overall data rate. A finding is particularly relevant for bandwidth and power constrained systems. For a given network containing 1000 hidden layer neurons, the spatially downsampled systems achieved a best case accuracy of 89.38% on N-MNIST as opposed to 81.03% with no downsampling at the same hidden layer size. On the N-Caltech101 data set, the downsampled system achieved a best case accuracy of 18.25%, compared with 7.43% achieved with no downsampling. The results show that downsampling is an important preprocessing technique in event-based visual processing, especially for applications sensitive to power consumption and transmission bandwidth. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel adaptive piecewise linear model for dynamic system identification. It has four unique features. First, the model designs a new kind of basis function for function approximation. It maintains the uniform shape for each basis function, so as to achieve a satisfactory tradeoff between generalization ability and model complexity. Second, the model takes the structure of basis functions as decision variables to optimize the formulated identification problems instead of taking expansion coefficients as decision variables as proposed by many existing approaches. Third, we establish an incremental design strategy to solve the system identification problems. In each step of the identification, the selection of optimal basis function is a Lipschitz continuous optimization problem that is likely to be easily handled with some mature toolboxes. This incremental design strategy greatly reduces the estimation cost. Fourth, we introduce a smoothing mechanism to avoid overfitting, when the output of dynamic systems is disturbed by noise. Tests on several benchmark dynamic systems demonstrate the potential of the proposed model. [ABSTRACT FROM AUTHOR]

Learning discriminative feature representations has shown remarkable importance due to its promising performance for machine learning problems. This paper presents a discriminative data representation learning framework by employing a simple yet powerful marginal regression function with probabilistic graphical structure adaptation. A marginally structured representation learning (MSRL) method is proposed by seamlessly incorporating distinguishable regression targets analysis, graph structure adaptation, and robust linear structural learning into a joint framework. Specifically, MSRL learns marginal regression targets from data rather than exploiting the conventional zero–one matrix that greatly hinders the freedom of regression fitness and degrades the performance of regression results. Meanwhile, an optimized graph regularization term with self-improving adaptation is constructed based on probabilistic connection knowledge to improve the compactness of the learned representation. Additionally, the regression targets are further predicted by utilizing the explanatory factors from the latent subspace of data, which can uncover the underlying feature correlations to enhance the reliability. The resulting optimization problem can be elegantly solved by an efficient iterative algorithm. Finally, the proposed method is evaluated by eight diverse but related tasks, including object, face, texture, and scene, categorization data sets. The encouraging experimental results and the explicit theoretical analysis demonstrate the efficacy of the proposed representation learning method in comparison with state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

This paper presents the Chebyshev neural network (ChNN) as an improved artificial intelligence technique for power system protection studies and examines the performances of two ChNN learning algorithms for fault classification of series compensated transmission line. The training algorithms are least-square Levenberg–Marquardt (LSLM) and recursive least-square algorithm with forgetting factor (RLSFF). The performances of these algorithms are assessed based on their generalization capability in relating the fault current parameters with an event of fault in the transmission line. The proposed algorithm is fast in response as it utilizes postfault samples of three phase currents measured at the relaying end corresponding to half-cycle duration only. After being trained with only a small part of the generated fault data, the algorithms have been tested over a large number of fault cases with wide variation of system and fault parameters. Based on the studies carried out in this paper, it has been found that although the RLSFF algorithm is faster for training the ChNN in the fault classification application for series compensated transmission lines, the LSLM algorithm has the best accuracy in testing. The results prove that the proposed ChNN-based method is accurate, fast, easy to design, and immune to the level of compensations. Thus, it is suitable for digital relaying applications. [ABSTRACT FROM PUBLISHER]

In this paper, the stability problem is studied for a class of stochastic neural networks (NNs) with local impulsive effects. The impulsive effects considered can be not only nonidentical in different dimensions of the system state but also various at distinct impulsive instants. Hence, the impulses here can encompass several typical impulses in NNs. The aim of this paper is to derive stability criteria such that stochastic NNs with local impulsive effects are exponentially stable in mean square. By means of the mathematical induction method, several easy-to-check conditions are obtained to ensure the mean square stability of NNs. Three examples are given to show the effectiveness of the proposed stability criterion. [ABSTRACT FROM PUBLISHER]

This paper focuses on the deep and shallow architecture of multilayer neural networks (MNNs). The demonstration of whether or not an MNN can be replaced by another MNN with fewer layers is equivalent to studying the topological conjugacy of its hidden layers. This paper provides a systematic methodology to indicate when two hidden spaces are topologically conjugated. Furthermore, some criteria are presented for some specific cases. [ABSTRACT FROM PUBLISHER]

This paper investigates how to train a recurrent neural network (RNN) using the Levenberg–Marquardt (LM) algorithm as well as how to implement optimal control of a grid-connected converter (GCC) using an RNN. To successfully and efficiently train an RNN using the LM algorithm, a new forward accumulation through time (FATT) algorithm is proposed to calculate the Jacobian matrix required by the LM algorithm. This paper explores how to incorporate FATT into the LM algorithm. The results show that the combination of the LM and FATT algorithms trains RNNs better than the conventional backpropagation through time algorithm. This paper presents an analytical study on the optimal control of GCCs, including theoretically ideal optimal and suboptimal controllers. To overcome the inapplicability of the optimal GCC controller under practical conditions, a new RNN controller with an improved input structure is proposed to approximate the ideal optimal controller. The performance of an ideal optimal controller and a well-trained RNN controller was compared in close to real-life power converter switching environments, demonstrating that the proposed RNN controller can achieve close to ideal optimal control performance even under low sampling rate conditions. The excellent performance of the proposed RNN controller under challenging and distorted system conditions further indicates the feasibility of using an RNN to approximate optimal control in practical applications. [ABSTRACT FROM AUTHOR]

This paper reviews the current state of the art on reinforcement learning (RL)-based feedback control solutions to optimal regulation and tracking of single and multiagent systems. Existing RL solutions to both optimal \mathcal H2 and \mathcal H_\infty control problems, as well as graphical games, will be reviewed. RL methods learn the solution to optimal control and game problems online and using measured data along the system trajectories. We discuss Q-learning and the integral RL algorithm as core algorithms for discrete-time (DT) and continuous-time (CT) systems, respectively. Moreover, we discuss a new direction of off-policy RL for both CT and DT systems. Finally, we review several applications. [ABSTRACT FROM AUTHOR]

In this paper, a new training paradigm is proposed for deep reinforcement learning using self-paced prioritized curriculum learning with coverage penalty. The proposed deep curriculum reinforcement learning (DCRL) takes the most advantage of experience replay by adaptively selecting appropriate transitions from replay memory based on the complexity of each transition. The criteria of complexity in DCRL consist of self-paced priority as well as coverage penalty. The self-paced priority reflects the relationship between the temporal-difference error and the difficulty of the current curriculum for sample efficiency. The coverage penalty is taken into account for sample diversity. With comparison to deep Q network (DQN) and prioritized experience replay (PER) methods, the DCRL algorithm is evaluated on Atari 2600 games, and the experimental results show that DCRL outperforms DQN and PER on most of these games. More results further show that the proposed curriculum training paradigm of DCRL is also applicable and effective for other memory-based deep reinforcement learning approaches, such as double DQN and dueling network. All the experimental results demonstrate that DCRL can achieve improved training efficiency and robustness for deep reinforcement learning. [ABSTRACT FROM AUTHOR]

Sequence transfer learning is of interest in both academia and industry with the emergence of numerous new text domains from Twitter and other social media tools. In this paper, we put forward the data-sensitive granularity for transfer learning, and then, a novel substructural regularization transfer learning model (STLM) is proposed to preserve target domain features at substructural granularity in the light of the condition of labeled data set size. Our model is underpinned by hidden Markov model and regularization theory, where the substructural representation can be integrated as a penalty after measuring the dissimilarity of substructures between target domain and STLM with relative entropy. STLM can achieve the competing goals of preserving the target domain substructure and utilizing the observations from both the target and source domains simultaneously. The estimation of STLM is very efficient since an analytical solution can be derived as a necessary and sufficient condition. The relative usability of substructures to act as regularization parameters and the time complexity of STLM are also analyzed and discussed. Comprehensive experiments of part-of-speech tagging with both Brown and Twitter corpora fully justify that our model can make improvements on all the combinations of source and target domains. [ABSTRACT FROM AUTHOR]

Traditional classification systems rely heavily on sufficient training data with accurate labels. However, the quality of the collected data depends on the labelers, among which inexperienced labelers may exist and produce unexpected labels that may degrade the performance of a learning system. In this paper, we investigate the multiclass classification problem where a certain amount of training examples are randomly labeled. Specifically, we show that this issue can be formulated as a label noise problem. To perform multiclass classification, we employ the widely used importance reweighting strategy to enable the learning on noisy data to more closely reflect the results on noise-free data. We illustrate the applicability of this strategy to any surrogate loss functions and to different classification settings. The proportion of randomly labeled examples is proved to be upper bounded and can be estimated under a mild condition. The convergence analysis ensures the consistency of the learned classifier to the optimal classifier with respect to clean data. Two instantiations of the proposed strategy are also introduced. Experiments on synthetic and real data verify that our approach yields improvements over the traditional classifiers as well as the robust classifiers. Moreover, we empirically demonstrate that the proposed strategy is effective even on asymmetrically noisy data. [ABSTRACT FROM AUTHOR]

This paper focuses on the connecting structure of deep neural networks and proposes a layerwise structure learning method based on multiobjective optimization. A model with better generalization can be obtained by reducing the connecting parameters in deep networks. The aim is to find the optimal structure with high representation ability and better generalization for each layer. Then, the visible data are modeled with respect to structure based on the products of experts. In order to mitigate the difficulty of estimating the denominator in PoE, the denominator is simplified and taken as another objective, i.e., the connecting sparsity. Moreover, for the consideration of the contradictory nature between the representation ability and the network connecting sparsity, the multiobjective model is established. An improved multiobjective evolutionary algorithm is used to solve this model. Two tricks are designed to decrease the computational cost according to the properties of input data. The experiments on single-layer level, hierarchical level, and application level demonstrate the effectiveness of the proposed algorithm, and the learned structures can improve the performance of deep neural networks. [ABSTRACT FROM AUTHOR]

This paper is devoted to distributed adaptive containment control for a class of nonlinear multiagent systems with input quantization. By employing a matrix factorization and a novel matrix normalization technique, some assumptions involving control gain matrices in existing results are relaxed. By fusing the techniques of sliding mode control and backstepping control, a two-step design method is proposed to construct controllers and, with the aid of neural networks, all system nonlinearities are allowed to be unknown. Moreover, a linear time-varying model and a similarity transformation are introduced to circumvent the obstacle brought by quantization, and the controllers need no information about the quantizer parameters. The proposed scheme is able to ensure the boundedness of all closed-loop signals and steer the containment errors into an arbitrarily small residual set. The simulation results illustrate the effectiveness of the scheme. [ABSTRACT FROM AUTHOR]

As data sets become larger and more complicated, an extreme learning machine (ELM) that runs in a traditional serial environment cannot realize its ability to be fast and effective. Although a parallel ELM (PELM) based on MapReduce to process large-scale data shows more efficient learning speed than identical ELM algorithms in a serial environment, some operations, such as intermediate results stored on disks and multiple copies for each task, are indispensable, and these operations create a large amount of extra overhead and degrade the learning speed and efficiency of the PELMs. In this paper, an efficient ELM based on the Spark framework (SELM), which includes three parallel subalgorithms, is proposed for big data classification. By partitioning the corresponding data sets reasonably, the hidden layer output matrix calculation algorithm, matrix \mathbf \hat U decomposition algorithm, and matrix \mathbf V decomposition algorithm perform most of the computations locally. At the same time, they retain the intermediate results in distributed memory and cache the diagonal matrix as broadcast variables instead of several copies for each task to reduce a large amount of the costs, and these actions strengthen the learning ability of the SELM. Finally, we implement our SELM algorithm to classify large data sets. Extensive experiments have been conducted to validate the effectiveness of the proposed algorithms. As shown, our SELM achieves an $8.71\times$ speedup on a cluster with ten nodes, and reaches a $13.79\times$ speedup with 15 nodes, an $18.74\times$ speedup with 20 nodes, a $23.79\times$ speedup with 25 nodes, a $28.89\times$ speedup with 30 nodes, and a $33.81\times$ speedup with 35 nodes. [ABSTRACT FROM AUTHOR]

There are three important concepts related to learning processes in neural networks: reducibility, nonminimality, and singularity. Although the definitions of these three concepts differ, they are equivalent in real-valued neural networks. This is also true of complex-valued neural networks (CVNNs) with hidden neurons not employing biases. The situation of CVNNs with hidden neurons employing biases, however, is very complicated. Exceptional reducibility was found, and it was shown that reducibility and nonminimality are not the same. Irreducibility consists of minimality and exceptional reducibility. The relationship between minimality and singularity has not yet been established. In this paper, we describe our surprising finding that minimality and singularity are independent. We also provide several examples based on exceptional reducibility. [ABSTRACT FROM AUTHOR]

In this paper, we study the consensus problem of discrete-time and continuous-time multiagent systems with distance-dependent communication networks, respectively. The communication weight between any two agents is assumed to be a nonincreasing function of their distance. First, we consider the networks with fixed connectivity. In this case, the interaction between adjacent agents always exists but the influence could possibly become negligible if the distance is long enough. We show that consensus can be reached under arbitrary initial states if the decay rate of the communication weight is less than a given bound. Second, we study the networks with distance-dependent connectivity. It is assumed that any two agents interact with each other if and only if their distance does not exceed a fixed range. With the validity of some conditions related to the property of the initial communication graph, we prove that consensus can be achieved asymptotically. Third, we present some applications of the main results to opinion consensus problems and formation control problems. Finally, several simulation examples are presented to illustrate the effectiveness of the theoretical findings. [ABSTRACT FROM PUBLISHER]

In numerous industrial applications where safety, efficiency, and reliability are among primary concerns, condition-based maintenance (CBM) is often the most effective and reliable maintenance policy. Prognostics, as one of the key enablers of CBM, involves the core task of estimating the remaining useful life (RUL) of the system. Neural networks-based approaches have produced promising results on RUL estimation, although their performances are influenced by handcrafted features and manually specified parameters. In this paper, we propose a multiobjective deep belief networks ensemble (MODBNE) method. MODBNE employs a multiobjective evolutionary algorithm integrated with the traditional DBN training technique to evolve multiple DBNs simultaneously subject to accuracy and diversity as two conflicting objectives. The eventually evolved DBNs are combined to establish an ensemble model used for RUL estimation, where combination weights are optimized via a single-objective differential evolution algorithm using a task-oriented objective function. We evaluate the proposed method on several prognostic benchmarking data sets and also compare it with some existing approaches. Experimental results demonstrate the superiority of our proposed method. [ABSTRACT FROM AUTHOR]

Learning an ideal metric is crucial to many tasks in computer vision. Diverse feature representations may combat this problem from different aspects; as visual data objects described by multiple features can be decomposed into multiple views, thus often provide complementary information. In this paper, we propose a cross-view fusion algorithm that leads to a similarity metric for multiview data by systematically fusing multiple similarity measures. Unlike existing paradigms, we focus on learning distance measure by exploiting a graph structure of data samples, where an input similarity matrix can be improved through a propagation of graph random walk. In particular, we construct multiple graphs with each one corresponding to an individual view, and a cross-view fusion approach based on graph random walk is presented to derive an optimal distance measure by fusing multiple metrics. Our method is scalable to a large amount of data by enforcing sparsity through an anchor graph representation. To adaptively control the effects of different views, we dynamically learn view-specific coefficients, which are leveraged into graph random walk to balance multiviews. However, such a strategy may lead to an over-smooth similarity metric where affinities between dissimilar samples may be enlarged by excessively conducting cross-view fusion. Thus, we figure out a heuristic approach to controlling the iteration number in the fusion process in order to avoid over smoothness. Extensive experiments conducted on real-world data sets validate the effectiveness and efficiency of our approach. [ABSTRACT FROM PUBLISHER]

The task of natural language parsing can naturally be embedded in the maximum-margin framework for structured output prediction using an appropriate joint feature map and a suitable structured loss function. While there are efficient learning algorithms based on the cutting-plane method for optimizing the resulting quadratic objective with potentially exponential number of linear constraints, their efficiency crucially depends on the inference algorithms used to infer the most violated constraint in a current iteration. In this paper, we derive an extension of the well-known Cocke–Kasami–Younger (CKY) algorithm used for parsing with probabilistic context-free grammars for the case of loss-augmented inference enabling an effective training in the cutting-plane approach. The resulting algorithm is guaranteed to find an optimal solution in polynomial time exceeding the running time of the CKY algorithm by a term, which only depends on the number of possible loss values. In order to demonstrate the feasibility of the presented algorithm, we perform a set of experiments for parsing English sentences. [ABSTRACT FROM PUBLISHER]

Two types of coupled neural networks with reaction–diffusion terms are considered in this paper. In the first one, the nodes are coupled through their states. In the second one, the nodes are coupled through the spatial diffusion terms. For the former, utilizing Lyapunov functional method and pinning control technique, we obtain some sufficient conditions to guarantee that network can realize synchronization. In addition, considering that the theoretical coupling strength required for synchronization may be much larger than the needed value, we propose an adaptive strategy to adjust the coupling strength for achieving a suitable value. For the latter, we establish a criterion for synchronization using the designed pinning controllers. It is found that the coupled reaction–diffusion neural networks with state coupling under the given linear feedback pinning controllers can realize synchronization when the coupling strength is very large, which is contrary to the coupled reaction–diffusion neural networks with spatial diffusion coupling. Moreover, a general criterion for ensuring network synchronization is derived by pinning a small fraction of nodes with adaptive feedback controllers. Finally, two examples with numerical simulations are provided to demonstrate the effectiveness of the theoretical results. [ABSTRACT FROM AUTHOR]

This paper proposes a novel optimal tracking control scheme for nonlinear continuous-time systems in strict-feedback form with uncertain dynamics. The optimal tracking problem is transformed into an equivalent optimal regulation problem through a feedforward adaptive control input that is generated by modifying the standard backstepping technique. Subsequently, a neural network-based optimal control scheme is introduced to estimate the cost, or value function, over an infinite horizon for the resulting nonlinear continuous-time systems in affine form when the internal dynamics are unknown. The estimated cost function is then used to obtain the optimal feedback control input; therefore, the overall optimal control input for the nonlinear continuous-time system in strict-feedback form includes the feedforward plus the optimal feedback terms. It is shown that the estimated cost function minimizes the Hamilton–Jacobi–Bellman estimation error in a forward-in-time manner without using any value or policy iterations. Finally, optimal output feedback control is introduced through the design of a suitable observer. Lyapunov theory is utilized to show the overall stability of the proposed schemes without requiring an initial admissible controller. Simulation examples are provided to validate the theoretical results. [ABSTRACT FROM PUBLISHER]

Sparse representation learning has been successfully applied into image classification, which represents a given image as a linear combination of an over-complete dictionary. The classification result depends on the reconstruction residuals. Normally, the images are stretched into vectors for convenience, and the representation residuals are characterized by l2 -norm or l1 -norm, which actually assumes that the elements in the residuals are independent and identically distributed variables. However, it is hard to satisfy the hypothesis when it comes to some structural errors, such as illuminations, occlusions, and so on. In this paper, we represent the image data in their intrinsic matrix form rather than concatenated vectors. The representation residual is considered as a matrix variate following the matrix elliptically contoured distribution, which is robust to dependent errors and has long tail regions to fit outliers. Then, we seek the maximum a posteriori probability estimation solution of the matrix-based optimization problem under sparse regularization. An alternating direction method of multipliers (ADMMs) is derived to solve the resulted optimization problem. The convergence of the ADMM is proven theoretically. Experimental results demonstrate that the proposed method is more effective than the state-of-the-art methods when dealing with the structural errors. [ABSTRACT FROM PUBLISHER]

This paper develops a neural network architecture and a new processing method for solving in real time, the nonlinear equality constrained multiobjective optimization problem (NECMOP), where several nonlinear objective functions must be optimized in a conflicting situation. In this processing method, the NECMOP is converted to an equivalent scalar optimization problem (SOP). The SOP is then decomposed into several-separable subproblems processable in parallel and in a reasonable time by multiplexing switched capacitor circuits. The approach which we propose makes use of a decomposition–coordination principle that allows nonlinearity to be treated at a local level and where coordination is achieved through the use of Lagrange multipliers. The modularity and the regularity of the neural networks architecture herein proposed make it suitable for very large scale integration implementation. An application to the resolution of a physical problem is given to show that the approach used here possesses some advantages of the point of algorithmic view, and provides processes of resolution often simpler than the usual techniques. [ABSTRACT FROM PUBLISHER]

Learning in multilayer neural networks (MNNs) relies on continuous updating of large matrices of synaptic weights by local rules. Such locality can be exploited for massive parallelism when implementing MNNs in hardware. However, these update rules require a multiply and accumulate operation for each synaptic weight, which is challenging to implement compactly using CMOS. In this paper, a method for performing these update operations simultaneously (incremental outer products) using memristor-based arrays is proposed. The method is based on the fact that, approximately, given a voltage pulse, the conductivity of a memristor will increment proportionally to the pulse duration multiplied by the pulse magnitude if the increment is sufficiently small. The proposed method uses a synaptic circuit composed of a small number of components per synapse: one memristor and two CMOS transistors. This circuit is expected to consume between 2% and 8% of the area and static power of previous CMOS-only hardware alternatives. Such a circuit can compactly implement hardware MNNs trainable by scalable algorithms based on online gradient descent (e.g., backpropagation). The utility and robustness of the proposed memristor-based circuit are demonstrated on standard supervised learning tasks. [ABSTRACT FROM PUBLISHER]

In this paper, adaptive synchronization of memristor-based neural networks (MNNs) with time-varying delays is investigated. The dynamical analysis here employs results from the theory of differential equations with discontinuous right-hand sides as introduced by Filippov. Sufficient conditions for the global synchronization of MNNs are established with a general adaptive controller. The update gain of the controller can be adjusted to control the synchronization speed. The obtained results complement and improve the previously known results. Finally, numerical simulations are carried out to demonstrate the effectiveness of the obtained results. [ABSTRACT FROM AUTHOR]

As the most important property of neural networks (NNs), the universal approximation capability of NNs is widely used in many applications. However, this property is generally proven for continuous systems. Most industrial systems are hybrid systems (e.g., piecewise continuous), which is a significant limitation for real applications. Recently, many identification methods have been proposed for hybrid system approximation; however, these methods only operate in linear hybrid systems. In this paper, the progressive learning machine—a new learning algorithm based on multi-NNs—is proposed for general hybrid nonlinear/linear system approximation. This algorithm classifies hybrid systems into several continuous systems and can approximate any hybrid system with zero output error. The performance of the proposed learning method is demonstrated via numerical examples and with experimental data from real applications. [ABSTRACT FROM AUTHOR]