Search

As model size and the number of layers increase, Deep Neural Networks (DNNs) demand enormous computational power and throughput to meet exceedingly high prediction accuracy’s of today’s machine learning (ML) applications. Spatial hardware accelerators have been proposed that optimize the dataflow and exploit sparsity to provide a significant decrease in power consumption. As spatial architectures are traditionally designed with metallic interconnects, significant power is expended for data movement for different dataflows. In this paper, we exploit extended wireless technology to design a power-efficient and high-throughput DNN accelerator, e-WiNN, that can be configured for all representative dataflows and arithmetic precisions. We leverage novel circuit design by utilizing Dadda-algorithm based Multiply-and-Accumulate (MAC) circuits for 4-bit, 8-bit and 16-bit inputs to reduce area, power and delay constraints in 14 nm predictive technology. Our novel wireless transmitter integrates on- off keying (OOK) modulator with power amplifier that results in significant energy savings. To reduce the area overhead, we cluster wireless transceivers into groups of four such that both weights and input features can be effectively multicast to reduce the data movement. The energy efficient transceiver circuit is implemented in state-of-the-art BSIM 32 nm FinFET technology model and our link budget considers required RF power for different frequencies and inter-PE distance at three different antenna directivities including isotropic. Our detailed RTL modeling and cycle-accurate simulation results show that e-WiNN achieves 36.3% latency reduction and 76.1% energy saving when compared to state-of-art wire interconnected accelerators; 70.3% area reduction and 41.6% energy saving at the cost of 11% latency increase when compared to prior wireless accelerators on various neural networks (AlexNet, VGG16, and ResNet-9/50). [ABSTRACT FROM AUTHOR]

Dropout and DropConnect are two techniques to facilitate the regularization of neural network models, having achieved the state-of-the-art results in several benchmarks. In this paper, to improve the generalization capability of spiking neural networks (SNNs), the two drop techniques are first applied to the state-of-the-art SpikeProp learning algorithm resulting in two improved learning algorithms called SPDO (SpikeProp with Dropout) and SPDC (SpikeProp with DropConnect). In view that a higher membrane potential of a biological neuron implies a higher probability of neural activation, three adaptive drop algorithms, SpikeProp with Adaptive Dropout (SPADO), SpikeProp with Adaptive DropConnect (SPADC), and SpikeProp with Group Adaptive Drop (SPGAD), are proposed by adaptively adjusting the keep probability for training SNNs. A convergence theorem for SPDC is proven under the assumptions of the bounded norm of connection weights and a finite number of equilibria. In addition, the five proposed algorithms are carried out in a collaborative neurodynamic optimization framework to improve the learning performance of SNNs. The experimental results on the four benchmark data sets demonstrate that the three adaptive algorithms converge faster than SpikeProp, SPDO, and SPDC, and the generalization errors of the five proposed algorithms are significantly smaller than that of SpikeProp. Furthermore, the experimental results also show that the five algorithms based on collaborative neurodynamic optimization can be improved in terms of several measures. [ABSTRACT FROM AUTHOR]

Markov blankets (MBs) in Bayesian networks (BNs) play an important role in both local causal discovery and large-scale BN structure learning. Almost all existing MB discovery algorithms are designed under the assumption of causal sufficiency, which states that there are no latent common causes for two or more of the observed variables in data. However, latent common causes are ubiquitous in many applications, and hence, this assumption is often violated in practice. Thus, developing algorithms for discovering MBs without assuming causal sufficiency is of practical significance, and it is crucial for causal structure learning in real-world data. In this paper, we focus on addressing this problem. Specifically, we adopt a maximal ancestral graph (MAG) model to represent latent common causes and the concept of MBs without assuming causal sufficiency. Then, we propose an effective and efficient algorithm to discover the MB of a target variable in an MAG. Using benchmark and real-world data sets, the experiments validate the algorithm proposed in this paper. [ABSTRACT FROM AUTHOR]

Hyperdimensional (HD) computing is a promising paradigm for future intelligent electronic appliances operating at low power. This paper discusses tradeoffs of selecting parameters of binary HD representations when applied to pattern recognition tasks. Particular design choices include density of representations and strategies for mapping data from the original representation. It is demonstrated that for the considered pattern recognition tasks (using synthetic and real-world data) both sparse and dense representations behave nearly identically. This paper also discusses implementation peculiarities which may favor one type of representations over the other. Finally, the capacity of representations of various densities is discussed. [ABSTRACT FROM AUTHOR]

In this paper, we deal with the learning problem when using an adaptive filtering method. For the learning system in filtering, the knowledge is obtained and updated based on the newly acquired information that is extracted and learned from the sequential samples over time. Effective measurement on the informativeness of a sample and reasonable subsequent treatment on the sample will improve the learning performance. This paper proposes a sequential outlier criterion for sparsification of online adaptive filtering. The method is proposed to achieve effective informativeness measurement of online filtering to obtain a more accurate and more compact network in the learning process. In the proposed method, the measurement on the samples’ informativeness is established based on the historical sequentially adjacent samples, and then the informative-measured samples are treated individually by the learning system based on whether the sample is informative, redundant, or abnormal. With our method, a more sensible learning process can be achieved with valid knowledge extracted, and the optimal network in the learning system can be obtained. Simulations based on static function estimation, Mackey-Glass time series prediction, and Lorenz chaotic time series prediction demonstrate that the proposed method can provide more effective classification on samples and more accurate networks in online adaptive filtering. [ABSTRACT FROM AUTHOR]

This paper presents the integration of reinforcement learning-based differential evolution (DE) with the cooperative coevolution (R-CCDE) method in a compensatory neuro-fuzzy controller (CNFC). The CNFC model employs compensatory fuzzy operations, which increase the adaptability and effectiveness of the controller. The R-CCDE method was used to determine an adequate control policy for nonlinear system problems. The evolution of a population involved the use of DE with cooperative coevolution to adjust CNFC parameters, and the fitness function of the R-CCDE method is used by a reinforcement signal to determine the controller that can be used to solve the control problem. This paper identified the best performing controller to solve nonlinear system problems. The simulation results of the proposed R-CCDE method were compared with those of various DE methods and the performance of the proposed R-CCDE method was superior to that of the other methods. [ABSTRACT FROM AUTHOR]

In the first issue of Nature 2015, Google DeepMind published a paper “Human-level control through deep reinforcement learning.” Furthermore, in the first issue of Nature 2016, it published a cover paper “Mastering the game of Go with deep neural networks and tree search” and proposed the computer Go program, AlphaGo. In March 2016, AlphaGo beat the world’s top Go player Lee Sedol by 4:1. This becomes a new milestone in artificial intelligence history, the core of which is the algorithm of deep reinforcement learning (RL). [ABSTRACT FROM AUTHOR]

Improved data visualization will be a significant tool to enhance cluster analysis. In this paper, an information-theoretic-based method for cluster visualization using self-organizing maps (SOMs) is presented. The information-theoretic visualization (IT-vis) has the same structure as the unified distance matrix, but instead of depicting Euclidean distances between adjacent neurons, it displays the similarity between the distributions associated with adjacent neurons. Each SOM neuron has an associated subset of the data set whose cardinality controls the granularity of the IT-vis and with which the first- and second-order statistics are computed and used to estimate their probability density functions. These are used to calculate the similarity measure, based on Renyi’s quadratic cross entropy and cross information potential (CIP). The introduced visualizations combine the low computational cost and kernel estimation properties of the representative CIP and the data structure representation of a single-linkage-based grouping algorithm to generate an enhanced SOM-based visualization. The visual quality of the IT-vis is assessed by comparing it with other visualization methods for several real-world and synthetic benchmark data sets. Thus, this paper also contains a significant literature survey. The experiments demonstrate the IT-vis cluster revealing capabilities, in which cluster boundaries are sharply captured. Additionally, the information-theoretic visualizations are used to perform clustering of the SOM. Compared with other methods, IT-vis of large SOMs yielded the best results in this paper, for which the quality of the final partitions was evaluated using external validity indices. [ABSTRACT FROM AUTHOR]

In this paper, we first address the dynamics of the elegant multivalued quaternionic Hopfield neural network (MV-QHNN) proposed by Minemoto et al. Contrary to what was expected, we show that the MV-QHNN, as well as one of its variation, does not always come to rest at an equilibrium state under the usual conditions. In fact, we provide simple examples in which the network yields a periodic sequence of quaternionic state vectors. Afterward, we turn our attention to the continuous-valued quaternionic Hopfield neural network (CV-QHNN), which can be derived from the MV-QHNN by means of a limit process. The CV-QHNN can be implemented more easily than the MV-QHNN model. Furthermore, the asynchronous CV-QHNN always settles down into an equilibrium state under the usual conditions. Theoretical issues are all illustrated by examples in this paper. [ABSTRACT FROM AUTHOR]

This paper investigates the dynamical properties of a network of neurons, each of which implements an asynchronous mapping based on polynomial growth transforms. In the first part of this paper, we present a geometric approach for visualizing the dynamics of the network where each of the neurons traverses a trajectory in a dual optimization space, whereas the network itself traverses a trajectory in an equivalent primal optimization space. We show that as the network learns to solve basic classification tasks, different choices of primal-dual mapping produce unique but interpretable neural dynamics like noise shaping, spiking, and bursting. While the proposed framework is general enough, in this paper, we demonstrate its use for designing support vector machines (SVMs) that exhibit noise-shaping properties similar to those of $\Sigma \Delta $ modulators, and for designing SVMs that learn to encode information using spikes and bursts. It is demonstrated that the emergent switching, spiking, and burst dynamics produced by each neuron encodes its respective margin of separation from a classification hyperplane whose parameters are encoded by the network population dynamics. We believe that the proposed growth transform neuron model and the underlying geometric framework could serve as an important tool to connect well-established machine learning algorithms like SVMs to neuromorphic principles like spiking, bursting, population encoding, and noise shaping. [ABSTRACT FROM AUTHOR]

Memory is one of the most important mechanisms in recurrent neural networks (RNNs) learning. It plays a crucial role in practical applications, such as sequence learning. With a good memory mechanism, long term history can be fused with current information, and can thus improve RNNs learning. Developing a suitable memory mechanism is always desirable in the field of RNNs. This paper proposes a novel memory mechanism for RNNs. The main contributions of this paper are: 1) an auxiliary memory unit (AMU) is proposed, which results in a new special RNN model (AMU-RNN), separating the memory and output explicitly and 2) an efficient learning algorithm is developed by employing the technique of error flow truncation. The proposed AMU-RNN model, together with the developed learning algorithm, can learn and maintain stable memory over a long time range. This method overcomes both the learning conflict problem and gradient vanishing problem. Unlike the traditional method, which mixes the memory and output with a single neuron in a recurrent unit, the AMU provides an auxiliary memory neuron to maintain memory in particular. By separating the memory and output in a recurrent unit, the problem of learning conflicts can be eliminated easily. Moreover, by using the technique of error flow truncation, each auxiliary memory neuron ensures constant error flow during the learning process. The experiments demonstrate good performance of the proposed AMU-RNNs and the developed learning algorithm. The method exhibits quite efficient learning performance with stable convergence in the AMU-RNN learning and outperforms the state-of-the-art RNN models in sequence generation and sequence classification tasks. [ABSTRACT FROM AUTHOR]

In the past few years, the demand for real-time hardware implementations of deep neural networks (DNNs), especially convolutional neural networks (CNNs), has dramatically increased, thanks to their excellent performance on a wide range of recognition and classification tasks. When considering real-time action recognition and video/image classification systems, latency is of paramount importance. Therefore, applications strive to maximize the accuracy while keeping the latency under a given application-specific maximum: in most cases, this threshold cannot exceed a few hundred milliseconds. Until now, the research on DNNs has mainly focused on achieving a better classification or recognition accuracy, whereas very few works in literature take in account the computational complexity of the model. In this paper, we propose an efficient computational method, which is inspired by a computational core of fully connected neural networks, to process convolutional layers of state-of-the-art deep CNNs within strict latency requirements. To this end, we implemented our method customized for VGG and VGG-based networks which have shown state-of-the-art performance on different classification/recognition data sets. The implementation results in 65-nm CMOS technology show that the proposed accelerator can process convolutional layers of VGGNet up to 9.5 times faster than state-of-the-art accelerators reported to-date while occupying 3.5 mm2. [ABSTRACT FROM AUTHOR]

Independent component analysis (ICA) is a blind source separation technique where data are modeled as linear combinations of several independent non-Gaussian sources. The independence and linear restrictions are relaxed using several ICA mixture models (ICAMMs) obtaining a two-layer artificial neural network structure. This allows for dependence between sources of different classes, and thus, a myriad of multidimensional probability density functions can be accurate modeled. This paper proposes a new probabilistic distance (PDI) between the parameters learned for two ICAMMs. The PDI is computed explicitly, unlike the popular Kullback–Leibler divergence (KLD) and other similar metrics, removing the need for numerical integration. Furthermore, the PDI is symmetric and bounded within 0 and 1, which enables its use as a posterior probability in fusion approaches. In this paper, the PDI is employed for change detection by measuring the distance between two ICAMMs learned in consecutive time windows. The changes might be associated with relevant states from a process under analysis that are explicitly reflected in the learned ICAMM parameters. The proposed distance was tested in two challenging applications using simulated and real data: 1) detecting flaws in materials using ultrasounds and 2) detecting changes in electroencephalography signals from humans performing neuropsychological tests. The results demonstrate that the PDI outperforms the KLD in change-detection capabilities. [ABSTRACT FROM PUBLISHER]

Energy optimization for plug-in hybrid electric vehicles (PHEVs) is a challenging problem due to the system complexity and many physical and operational constraints in PHEVs. In this paper, we present a Q-learning-based in-vehicle learning system that is free of physical models and can robustly converge to an optimal energy control solution. The proposed machine learning algorithms combine neuro-dynamic programming (NDP) with future trip information to effectively estimate the expected future energy cost (expected cost-to-go) for a given vehicle state and control actions. The convergences of these learning algorithms were demonstrated on both fixed and randomly selected drive cycles. Based on the characteristics of these learning algorithms, we propose a two-stage deployment solution for PHEV power management applications. Furthermore, we introduce a new initialization strategy, which combines the optimal learning with a properly selected penalty function. This initialization scheme can reduce the learning convergence time by 70%, which is a significant improvement for in-vehicle implementation efficiency. Finally, we develop a neural network (NN) for predicting battery state-of-charge (SoC), rendering the proposed power management controller completely free of physical models. [ABSTRACT FROM AUTHOR]

Convolutional long short-term memory (ConvLSTM) networks have been widely used for action/gesture recognition, and different attention mechanisms have also been embedded into ConvLSTM networks. This paper explores the redundancy of spatial convolutions and the effects of the attention mechanism in ConvLSTM, based on our previous gesture recognition architectures that combine the 3-D convolutional neural network (CNN) and ConvLSTM. Depthwise separable, group, and shuffle convolutions are used to replace the convolutional structures in ConvLSTM for the redundancy analysis. In addition, four ConvLSTM variants are derived for attention analysis: 1) by removing the convolutional structures of the three gates in ConvLSTM; 2) by applying the attention mechanism on the ConvLSTM input; and 3) by reconstructing the input and 4) output gates with the modified channelwise attention mechanism. Evaluation results demonstrate that the spatial convolutions in the three gates scarcely contribute to the spatiotemporal feature fusion and that the attention mechanisms embedded into the input and output gates cannot improve the feature fusion. In other words, ConvLSTM mainly contributes to the temporal fusion along with the recurrent steps to learn long-term spatiotemporal features when taking spatial or spatiotemporal features as input. A new LSTM variant is derived on this basis in which the convolutional structures are embedded only into the input-to-state transition of LSTM. The code of the LSTM variants is publicly available. 1 https://github.com/GuangmingZhu/ConvLSTMForGR. [ABSTRACT FROM AUTHOR]

Prospective authors are requested to submit new, unpublished manuscripts for inclusion in the upcoming event described in this call for papers. [ABSTRACT FROM AUTHOR]

Error functions are normally based on the distance between output spikes and target spikes in supervised learning algorithms for spiking neural networks (SNNs). Due to the discontinuous nature of the internal state of spiking neuron, it is challenging to ensure that the number of output spikes and target spikes kept identical in multispike learning. This problem is conventionally dealt with by using the smaller of the number of desired spikes and that of actual output spikes in learning. However, if this approach is used, information is lost as some spikes are neglected. In this paper, a probability-modulated timing mechanism is built on the stochastic neurons, where the discontinuous spike patterns are converted to the likelihood of generating the desired output spike trains. By applying this mechanism to a probability-modulated spiking classifier, a probability-modulated SNN (PMSNN) is constructed. In its multilayer and multispike learning structure, more inputs are incorporated and mapped to the target spike trains. A clustering rule connection mechanism is also applied to a reservoir to improve the efficiency of information transmission among synapses, which can map the highly correlated inputs to the adjacent neurons. Results of comparisons between the proposed method and popular the SNN algorithms showed that the PMSNN yields higher efficiency and requires fewer parameters. [ABSTRACT FROM AUTHOR]

Online peer-to-peer (P2P) lending is expected to benefit both investors and borrowers due to their low transaction cost and the elimination of expensive intermediaries. From the lenders’ perspective, maximizing their return on investment is an ultimate goal during their decision-making procedure. In this paper, we explore and address a fundamental problem underlying such a goal: how to represent the two competing risks, charge-off and prepayment, in funded loans. We propose to model both potential risks simultaneously, which remains largely unexplored until now. We first develop a hierarchical grading framework to integrate two risks of loans both qualitatively and quantitatively. Afterward, we introduce an end-to-end deep learning approach to solve this problem by breaking it down into multiple binary classification subproblems that are amenable to both feature representation and risks learning. Particularly, we leverage deep neural networks to jointly solve these subtasks, which leads to the in-depth exploration of the interaction involved in these tasks. To the best of our knowledge, this is the first attempt to characterize competing risks for loans in P2P lending via deep neural networks. The comprehensive experiments on real-world loan data show that our methodology is able to achieve an appealing investment performance by modeling the competition within and between risks explicitly and properly. The feature analysis based on saliency maps provides useful insights into payment dynamics of loans for potential investors intuitively. [ABSTRACT FROM AUTHOR]

An efficient deep learning requires a memory-efficient construction of a neural network. This paper introduces a layerwise tensorized formulation of a multilayer neural network, called LTNN, such that the weight matrix can be significantly compressed during training. By reshaping the multilayer neural network weight matrix into a high-dimensional tensor with a low-rank approximation, significant network compression can be achieved with maintained accuracy. An according layerwise training is developed by a modified alternating least-squares method with backward propagation for fine-tuning only. LTNN can provide the state-of-the-art results on various benchmarks with significant compression. For MNIST benchmark, LTNN shows $64 \times $ compression rate without accuracy drop. For Imagenet12 benchmark, our proposed LTNN achieves $35.84 \times $ compression of the neural network with around 2% accuracy drop. We have also shown $1.615 \times $ faster on inference speed than the existing works due to the smaller tensor core ranks. [ABSTRACT FROM AUTHOR]

Image recognition based on convolutional neural networks (CNNs) has recently been shown to deliver the state-of-the-art performance in various areas of computer vision and image processing. Nevertheless, applying a deep CNN to no-reference image quality assessment (NR-IQA) remains a challenging task due to critical obstacles, i.e., the lack of a training database. In this paper, we propose a CNN-based NR-IQA framework that can effectively solve this problem. The proposed method—deep image quality assessor (DIQA)—separates the training of NR-IQA into two stages: 1) an objective distortion part and 2) a human visual system-related part. In the first stage, the CNN learns to predict the objective error map, and then the model learns to predict subjective score in the second stage. To complement the inaccuracy of the objective error map prediction on the homogeneous region, we also propose a reliability map. Two simple handcrafted features were additionally employed to further enhance the accuracy. In addition, we propose a way to visualize perceptual error maps to analyze what was learned by the deep CNN model. In the experiments, the DIQA yielded the state-of-the-art accuracy on the various databases. [ABSTRACT FROM AUTHOR]

Imbalanced data with a skewed class distribution are common in many real-world applications. Deep Belief Network (DBN) is a machine learning technique that is effective in classification tasks. However, conventional DBN does not work well for imbalanced data classification because it assumes equal costs for each class. To deal with this problem, cost-sensitive approaches assign different misclassification costs for different classes without disrupting the true data sample distributions. However, due to lack of prior knowledge, the misclassification costs are usually unknown and hard to choose in practice. Moreover, it has not been well studied as to how cost-sensitive learning could improve DBN performance on imbalanced data problems. This paper proposes an evolutionary cost-sensitive deep belief network (ECS-DBN) for imbalanced classification. ECS-DBN uses adaptive differential evolution to optimize the misclassification costs based on the training data that presents an effective approach to incorporating the evaluation measure (i.e., G-mean) into the objective function. We first optimize the misclassification costs, and then apply them to DBN. Adaptive differential evolution optimization is implemented as the optimization algorithm that automatically updates its corresponding parameters without the need of prior domain knowledge. The experiments have shown that the proposed approach consistently outperforms the state of the art on both benchmark data sets and real-world data set for fault diagnosis in tool condition monitoring. [ABSTRACT FROM AUTHOR]

In this paper, we propose an adaptive neural-network-based asymptotic control algorithm for a class of nonlinear systems subject to unknown actuator quantization. To this end, we exploit the sector property of the quantization nonlinearity and transform actuator quantization control problem into analyzing its upper bounds, which are then handled by a dynamic loop gain function-based approach. In our adaptive control scheme, there is only one parameter required to be estimated online for updating weights of neural networks. Within the framework of Lyapunov theory, it is shown that the proposed algorithm ensures that all the signals in the closed-loop system are ultimately bounded. Moreover, an asymptotic tracking error is obtained by means of introducingBarbalat’s lemmato the proposed adaptive law. [ABSTRACT FROM AUTHOR]

In feature selection, a measure that captures nonlinear relationships between features and class is the mutual information (MI), which is based on how information in the features reduces the uncertainty in the output. In this paper, we propose a new measure that is related to MI, called neighborhood entropy, and a novel filter method based on its minimization in a greedy procedure. Our algorithm integrates sequential forward selection with approximated nearest-neighbors techniques and locality-sensitive hashing. Experiments show that the classification accuracy is usually higher than that of other state-of-the-art algorithms, with the best results obtained with problems that are highly unbalanced and nonlinearly separable. The order by which the features are selected is also better, leading to a higher accuracy for fewer features. The experimental results indicate that our technique can be employed effectively in offline scenarios when one can dedicate more CPU time to achieve superior results and more robustness to noise and to class imbalance. [ABSTRACT FROM AUTHOR]

As a variant of nonnegative matrix factorization (NMF), symmetric NMF (SNMF) has shown to be effective for capturing the cluster structure embedded in the graph representation. In contrast to the existing SNMF-based clustering methods that empirically construct the similarity matrix and rigidly introduce the supervisory information to the assignment matrix, in this paper, we propose a novel SNMF-based semisupervised clustering method, namely, pairwise constraint propagation-induced SNMF (PCPSNMF). By formulating a single-constrained optimization problem, PCPSNMF is capable of learning the similarity and assignment matricesadaptively and simultaneously, in which a small amount of supervisory information in the form of pairwise constraints is introduced in a flexible way to guide the construction of the similarity matrix, and the two matrices communicate with each other to achieve mutual refinement until convergence. In addition, we propose an efficient alternating iterative algorithm to solve the optimization problem, whose convergence is theoretically proven. Experimental results over several benchmark image data sets demonstrate that PCPSNMF is less sensitive to initialization and produces higher clustering performance, compared with the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In this paper, a novel learning method calledpostboosting using extended G-mean(PBG) is proposed for online sequential multiclass imbalance learning (OS-MIL) in neural networks. PBG is effective due to three reasons. 1) Through postadjusting a classification boundary under extended G-mean, the challenging issue of imbalanced class distribution for sequentially arriving multiclass data can be effectively resolved. 2) A newly derived update rule for online sequential learning is proposed, which produces a high G-mean for current model and simultaneously possesses almost the same information of its previous models. 3) Adynamic adjustment mechanismprovided by extended G-mean is valid to deal with the unresolved challenging dense-majority problem and two dynamic changing issues, namely, dynamic changing data scarcity (DCDS) and dynamic changing data diversity (DCDD). Compared to other OS-MIL methods, PBG is highly effective on resolving DCDS, while PBG is the only method to resolve dense-majority and DCDD. Furthermore, PBG can directly and effectively handle unscaled data stream. Experiments have been conducted for PBG and two popular OS-MIL methods for neural networks under massive binary and multiclass data sets. Through the analyses of experimental results, PBG is shown to outperform the other compared methods on all data sets in various aspects including the issues of data scarcity, dense-majority, DCDS, DCDD, and unscaled data. [ABSTRACT FROM AUTHOR]

Mean square error (MSE) is the most prominent criterion in training neural networks and has been employed in numerous learning problems. In this paper, we suggest a group of novel robust information theoretic backpropagation (BP) methods, as correntropy-based conjugate gradient BP (CCG-BP). CCG-BP algorithms converge faster than the common correntropy-based BP algorithms and have better performance than the common CG-BP algorithms based on MSE, especially in nonGaussian environments and in cases with impulsive noise or heavy-tailed distributions noise. In addition, a convergence analysis of this new type of method is particularly considered. Numerical results for several samples of function approximation, synthetic function estimation, and chaotic time series prediction illustrate that our new BP method is more robust than the MSE-based method in the sense of impulsive noise, especially when SNR is low. [ABSTRACT FROM AUTHOR]

In this paper, kurtosis-based complex-valued real-time recurrent learning (KCRTRL) and kurtosis-based augmented CRTRL (KACRTRL) algorithms are proposed for training fully connected recurrent neural networks (FCRNNs) in the complex domain. These algorithms are designed by minimizing the cost functions based on the kurtosis of a complex-valued error signal. The KCRTRL algorithm exploits the circularity properties of the complex-valued signals, and this algorithm not only provides a faster convergence rate but also results in a lower steady-state error. However, the KCRTRL algorithm is suboptimal in the processing of noncircular (NC) complex-valued signals. On the other hand, the KACRTRL algorithm contains a complete second-order information due to the augmented statistics, thus considerably improves the performance of the FCRNN in the processing of NC complex-valued signals. Simulation results on the one-step-ahead prediction problems show that the proposed KCRTRL algorithm significantly enhances the performance for only circular complex-valued signals, whereas the proposed KACRTRL algorithm provides more superior performance than existing algorithms for NC complex-valued signals in terms of the convergence rate and the steady-state error. [ABSTRACT FROM AUTHOR]

This paper proposes a novel feature selection method, namely, unified simultaneous clustering feature selection (USCFS). A regularized regression with a new type of target matrix is formulated to select the most discriminative features among the original features from labeled or unlabeled data. The regression with $l_{2,1}$ -norm regularization allows the projection matrix to represent an effective selection of discriminative features. For unsupervised feature selection, the target matrix discovers label-like information not from the original data points but rather from projected data points, which are of a reduced dimensionality. Without the aid of an affinity graph-based local structure learning method, USCFS allows the target matrix to capture latent cluster centers via orthogonal basis clustering and to simultaneously select discriminative features guided by latent cluster centers. When class labels are available, the target matrix is also able to find latent class labels by regarding the ground-truth class labels as an approximate guide. Hence, supervised feature selection is realized using these latent class labels, which may differ from the ground-truth class labels. Experimental results demonstrate the effectiveness of the proposed method. Specifically, the proposed method outperforms the state-of-the-art methods on diverse real-world data sets for both the supervised and the unsupervised feature selection. [ABSTRACT FROM AUTHOR]

Deep convolutional neural networks (CNNs) have become a key element in the recent breakthrough of salient object detection. However, existing CNN-based methods are based on either patchwise (regionwise) training and inference or fully convolutional networks. Methods in the former category are generally time-consuming due to severe storage and computational redundancies among overlapping patches. To overcome this deficiency, methods in the second category attempt to directly map a raw input image to a predicted dense saliency map in a single network forward pass. Though being very efficient, it is arduous for these methods to detect salient objects of different scales or salient regions with weak semantic information. In this paper, we develop hybrid contrast-oriented deep neural networks to overcome the aforementioned limitations. Each of our deep networks is composed of two complementary components, including a fully convolutional stream for dense prediction and a segment-level spatial pooling stream for sparse saliency inference. We further propose an attentional module that learns weight maps for fusing the two saliency predictions from these two streams. A tailored alternate scheme is designed to train these deep networks by fine-tuning pretrained baseline models. Finally, a customized fully connected conditional random field model incorporating a salient contour feature embedding can be optionally applied as a postprocessing step to improve spatial coherence and contour positioning in the fused result from these two streams. Extensive experiments on six benchmark data sets demonstrate that our proposed model can significantly outperform the state of the art in terms of all popular evaluation metrics. [ABSTRACT FROM AUTHOR]

Nowadays, the control technology of the robotic manipulator with flexible joints (RMFJ) is not mature enough. The flexible-joint manipulator dynamic system possesses many uncertainties, which brings a great challenge to the controller design. This paper is motivated by this problem. In order to deal with this and enhance the system robustness, the full-state feedback neural network (NN) control is proposed. Moreover, output constraints of the RMFJ are achieved, which improve the security of the robot. Through the Lyapunov stability analysis, we identify that the proposed controller can guarantee not only the stability of flexible-joint manipulator system but also the boundedness of system state variables by choosing appropriate control gains. Then, we make some necessary simulation experiments to verify the rationality of our controllers. Finally, a series of control experiments are conducted on the Baxter. By comparing with the proportional–derivative control and the NN control with the rigid manipulator model, the feasibility and the effectiveness of NN control based on flexible-joint manipulator model are verified. [ABSTRACT FROM AUTHOR]

In this paper, we present an approach for minimizing the computational complexity of the trained convolutional neural networks (ConvNets). The idea is to approximate all elements of a given ConvNet and replace the original convolutional filters and parameters (pooling and bias coefficients; and activation function) with an efficient approximations capable of extreme reductions in computational complexity. Low-complexity convolution filters are obtained through a binary (zero and one) linear programming scheme based on the Frobenius norm over sets of dyadic rationals. The resulting matrices allow for multiplication-free computations requiring only addition and bit-shifting operations. Such low-complexity structures pave the way for low power, efficient hardware designs. We applied our approach on three use cases of different complexities: 1) a “light” but efficient ConvNet for face detection (with around 1000 parameters); 2) another one for hand-written digit classification (with more than 180 000 parameters); and 3) a significantly larger ConvNet: AlexNet with $\approx 1.2$ million matrices. We evaluated the overall performance on the respective tasks for different levels of approximations. In all considered applications, very low-complexity approximations have been derived maintaining an almost equal classification performance. [ABSTRACT FROM AUTHOR]

This paper focuses on the event-triggered guaranteed cost control design of nonlinear systems via a self-learning technique. In brief, an event-based guaranteed cost control strategy of nonlinear systems subjects to matched uncertainties is developed, thereby balancing the performance of guaranteed cost and the actuality of limited communication resource. The original control design is transformed into an optimal control problem with an event-based mechanism, where the relationship of guaranteed cost performance compared to the time-based formulation is discussed. A critic neural network is constructed for implementing the event-based optimal control design with stability guarantee. Simulation experiments are carried out to verify the theoretical results in detail. [ABSTRACT FROM AUTHOR]

The robust Huber’s M-estimator is widely used in signal and image processing, classification, and regression. From an optimization point of view, Huber’s M-estimation problem is often formulated as a large-sized quadratic programming (QP) problem in view of its nonsmooth cost function. This paper presents a generalized regression estimator which minimizes a reduced-sized QP problem. The generalized regression estimator may be viewed as a significant generalization of several robust regression estimators including Huber’s M-estimator. The performance of the generalized regression estimator is analyzed in terms of robustness and approximation accuracy. Furthermore, two low-dimensional recurrent neural networks (RNNs) are introduced for robust estimation. The two RNNs have low model complexity and enhanced computational efficiency. Finally, the experimental results of two examples and an application to image restoration are presented to substantiate superior performance of the proposed method over conventional algorithms for robust regression estimation in terms of approximation accuracy and convergence rate. [ABSTRACT FROM AUTHOR]

An intelligent data-driven predictive control strategy is proposed in this paper. The predictive controller is designed by combining predictive control and local weighted projection regression. The presented control strategy needs less prior knowledge and has fewer parameters that are hard to determine compared to other data-driven predictive controller, e.g., the one in dynamic partial least square (PLS) framework. Furthermore, the proposed predictive controller performs better in the control of nonlinear processes and is able to update its parameters based on the online data. The predictive model validity and intelligence of the control strategy are guaranteed by the online updating strategy to a certain degree. The control performance of the proposed predictive controller against the model predictive control (MPC) in dynamic PLS framework is illustrated through the simulation of a typical numerical example and the benchmark of a continuous stirred tank heater system. It can be observed from the simulation that the proposed MPC strategy has higher prediction precision and stronger ability in coping with nonlinear dynamic processes which are quite common in practical applications, for instance, the industrial process. [ABSTRACT FROM AUTHOR]

Actor-critic based on the policy gradient (PG-based AC) methods have been widely studied to solve learning control problems. In order to increase the data efficiency of learning prediction in the critic of PG-based AC, studies on how to use recursive least-squares temporal difference (RLS-TD) algorithms for policy evaluation have been conducted in recent years. In such contexts, the critic RLS-TD evaluates an unknown mixed policy generated by a series of different actors, but not one fixed policy generated by the current actor. Therefore, this AC framework with RLS-TD critic cannot be proved to converge to the optimal fixed point of learning problem. To address the above problem, this paper proposes a new AC framework named critic-iteration PG (CIPG), which learns the state-value function of current policy in an on-policy way and performs gradient ascent in the direction of improving discounted total reward. During each iteration, CIPG keeps the policy parameters fixed and evaluates the resulting fixed policy by $\ell _{2}$ -regularized RLS-TD critic. Our convergence analysis extends previous convergence analysis of PG with function approximation to the case of RLS-TD critic. The simulation results demonstrate that the $\ell _{2}$ -regularization term in the critic of CIPG is undamped during the learning process, and CIPG has better learning efficiency and faster convergence rate than conventional AC learning control methods. [ABSTRACT FROM AUTHOR]

Image captioning aims to generate natural language sentences to describe the salient parts of a given image. Although neural networks have recently achieved promising results, a key problem is that they can only describe concepts seen in the training image-sentence pairs. Efficient learning of novel concepts has thus been a topic of recent interest to alleviate the expensive manpower of labeling data. In this paper, we propose a novel method,Image-Text Surgery, to synthesize pseudoimage-sentence pairs. The pseudopairs are generated under the guidance of a knowledge base, with syntax from a seed data set (i.e., MSCOCO) and visual information from an existing large-scale image base (i.e., ImageNet). Via pseudodata, the captioning model learns novel concepts without any corresponding human-labeled pairs. We further introduce adaptive visual replacement, which adaptively filters unnecessary visual features in pseudodata with an attention mechanism. We evaluate our approach on a held-out subset of the MSCOCO data set. The experimental results demonstrate that the proposed approach provides significant performance improvements over state-of-the-art methods in terms of F1 score and sentence quality. An ablation study and the qualitative results further validate the effectiveness of our approach. [ABSTRACT FROM AUTHOR]

Deep learning (DL) methods and architectures have been the state-of-the-art classification algorithms for computer vision and natural language processing problems. However, the successful application of these methods in motor imagery (MI) brain–computer interfaces (BCIs), in order to boost classification performance, is still limited. In this paper, we propose a classification framework for MI data by introducing a new temporal representation of the data and also utilizing a convolutional neural network (CNN) architecture for classification. The new representation is generated from modifying the filter-bank common spatial patterns method, and the CNN is designed and optimized accordingly for the representation. Our framework outperforms the best classification method in the literature on the BCI competition IV-2a 4-class MI data set by 7% increase in average subject accuracy. Furthermore, by studying the convolutional weights of the trained networks, we gain an insight into the temporal characteristics of EEG. [ABSTRACT FROM AUTHOR]

There is a biological evidence to prove information is coded through precise timing of spikes in the brain. However, training a population of spiking neurons in a multilayer network to fire at multiple precise times remains a challenging task. Delay learning and the effect of a delay on weight learning in a spiking neural network (SNN) have not been investigated thoroughly. This paper proposes a novel biologically plausible supervised learning algorithm for learning precisely timed multiple spikes in a multilayer SNNs. Based on the spike-timing-dependent plasticity learning rule, the proposed learning method trains an SNN through the synergy between weight and delay learning. The weights of the hidden and output neurons are adjusted in parallel. The proposed learning method captures the contribution of synaptic delays to the learning of synaptic weights. Interaction between different layers of the network is realized through biofeedback signals sent by the output neurons. The trained SNN is used for the classification of spatiotemporal input patterns. The proposed learning method also trains the spiking network not to fire spikes at undesired times which contribute to misclassification. Experimental evaluation on benchmark data sets from the UCI machine learning repository shows that the proposed method has comparable results with classical rate-based methods such as deep belief network and the autoencoder models. Moreover, the proposed method can achieve higher classification accuracies than single layer and a similar multilayer SNN. [ABSTRACT FROM AUTHOR]

We are witnessing an explosive development and widespread application of deep neural networks (DNNs) in various fields. However, DNN models, especially a convolutional neural network (CNN), usually involve massive parameters and are computationally expensive, making them extremely dependent on high-performance hardware. This prohibits their further extensions, e.g., applications on mobile devices. In this paper, we present a quantized CNN, a unified approach to accelerate and compress convolutional networks. Guided by minimizing the approximation error of individual layer’s response, both fully connected and convolutional layers are carefully quantized. The inference computation can be effectively carried out on the quantized network, with much lower memory and storage consumption. Quantitative evaluation on two publicly available benchmarks demonstrates the promising performance of our approach: with comparable classification accuracy, it achieves 4 to $6 \times $ acceleration and 15 to $20\times $ compression. With our method, accurate image classification can even be directly carried out on mobile devices within 1 s. [ABSTRACT FROM AUTHOR]

Learning discriminative feature representations has shown remarkable importance due to its promising performance for machine learning problems. This paper presents a discriminative data representation learning framework by employing a simple yet powerful marginal regression function with probabilistic graphical structure adaptation. A marginally structured representation learning (MSRL) method is proposed by seamlessly incorporating distinguishable regression targets analysis, graph structure adaptation, and robust linear structural learning into a joint framework. Specifically, MSRL learns marginal regression targets from data rather than exploiting the conventional zero–one matrix that greatly hinders the freedom of regression fitness and degrades the performance of regression results. Meanwhile, an optimized graph regularization term with self-improving adaptation is constructed based on probabilistic connection knowledge to improve the compactness of the learned representation. Additionally, the regression targets are further predicted by utilizing the explanatory factors from the latent subspace of data, which can uncover the underlying feature correlations to enhance the reliability. The resulting optimization problem can be elegantly solved by an efficient iterative algorithm. Finally, the proposed method is evaluated by eight diverse but related tasks, including object, face, texture, and scene, categorization data sets. The encouraging experimental results and the explicit theoretical analysis demonstrate the efficacy of the proposed representation learning method in comparison with state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Dimension reduction for high-order tensors is a challenging problem. In conventional approaches, dimension reduction for higher order tensors is implemented via Tucker decomposition to obtain lower dimensional tensors. This paper introduces a probabilistic vectorial dimension reduction model for tensorial data. The model represents a tensor by using a linear combination of the same order basis tensors, thus it offers a learning approach to directly reduce a tensor to a vector. Under this expression, the projection base of the model is based on the tensor CandeComp/PARAFAC (CP) decomposition and the number of free parameters in the model only grows linearly with the number of modes rather than exponentially. A Bayesian inference has been established via the variational Expectation Maximization (EM) approach. A criterion to set the parameters (a factor number of CP decomposition and the number of extracted features) is empirically given. The model outperforms several existing principal component analysis-based methods and CP decomposition on several publicly available databases in terms of classification and clustering accuracy. [ABSTRACT FROM AUTHOR]

In this paper, we propose an online learning algorithm for supervised learning in multilayer spiking neural networks (SNNs). It is found that the spike timings of neurons in an SNN can be exploited to estimate the gradients that are associated with each synapse. With the proposed method of estimating gradients, learning similar to the stochastic gradient descent process employed in a conventional artificial neural network (ANN) can be achieved. In addition to the conventional layer-by-layer backpropagation, a one-pass direct backpropagation is possible using the proposed learning algorithm. Two neural networks, with one and two hidden layers, are employed as examples to demonstrate the effectiveness of the proposed learning algorithms. Several techniques for more effective learning are discussed, including utilizing a random refractory period to avoid saturation of spikes, employing a quantization noise injection technique and pseudorandom initial conditions to decorrelate spike timings, in addition to leveraging the progressive precision in an SNN to reduce the inference latency and energy. Extensive parametric simulations are conducted to examine the aforementioned techniques. The learning algorithm is developed with the considerations of ease of hardware implementation and relative compatibility with the classic ANN-based learning. Therefore, the proposed algorithm not only enjoys the high energy efficiency and good scalability of an SNN in its specialized hardware but also benefits from the well-developed theory and techniques of conventional ANN-based learning. The Modified National Institute of Standards and Technology database benchmark test is conducted to verify the newly proposed learning algorithm. Classification correct rates of 97.2% and 97.8% are achieved for the one-hidden-layer and two-hidden-layer neural networks, respectively. Moreover, a brief discussion of the hardware implementations is presented for two mainstream architectures. [ABSTRACT FROM AUTHOR]

In a real brain, the act of perception is a bidirectional process, depending on both feedforward sensory pathways and feedback pathways that carry expectations. We are interested in how such a neural network might emerge from a biologically plausible learning rule. Other neural network learning methods either only apply to feedforward networks, or employ assumptions (such as weight copying) that render them unlikely in a real brain. Predictive estimators (PEs) offer a better solution to this bidirectional learning scenario. However, PEs also depend on weight copying. In this paper, we propose the symmetric PE (SPE), an architecture that can learn both feedforward and feedback connection weights individually using only locally available information. We demonstrate that the SPE can learn complicated mappings without the use of weight copying. The SPE networks also show promise in deeper architectures. [ABSTRACT FROM AUTHOR]

Learning deep representations have been applied in action recognition widely. However, there have been a few investigations on how to utilize the structural manifold information among different action videos to enhance the recognition accuracy and efficiency. In this paper, we propose to incorporate the manifold of training samples into deep learning, which is defined as deep manifold learning (DML). The proposed DML framework can be adapted to most existing deep networks to learn more discriminative features for action recognition. When applied to a convolutional neural network, DML embeds the previous convolutional layer’s manifold into the next convolutional layer; thus, the discriminative capacity of the next layer can be promoted. We also apply the DML on a restricted Boltzmann machine, which can alleviate the overfitting problem. Experimental results on four standard action databases (i.e., UCF101, HMDB51, KTH, and UCF sports) show that the proposed method outperforms the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Many real-world classification problems are characterized by samples of a complex distribution in the input space. The classification accuracy is determined by intrinsic properties of all samples in subspaces of features. This paper proposes a novel algorithm for the construction of radial basis function neural network (RBFNN) classifier based on subspace learning. In this paper, feature subspaces are obtained for every hidden node of the RBFNN during the learning process. The connection weights between the input layer and the hidden layer are adjusted to produce various subspaces with dominative features for different hidden nodes. The network structure and dominative features are encoded in two subpopulations that are cooperatively coevolved using the coevolutionary algorithm to achieve a better global optimality for the estimated RBFNN. Experimental results illustrate that the proposed algorithm is able to obtain RBFNN models with both better classification accuracy and simpler network structure when compared with other learning algorithms. Thus, the proposed model provides a more flexible and efficient approach to complex classification tasks by employing the local characteristics of samples in subspaces. [ABSTRACT FROM PUBLISHER]

An extreme learning machine (ELM) can be regarded as a two-stage feed-forward neural network (FNN) learning system that randomly assigns the connections with and within hidden neurons in the first stage and tunes the connections with output neurons in the second stage. Therefore, ELM training is essentially a linear learning problem, which significantly reduces the computational burden. Numerous applications show that such a computation burden reduction does not degrade the generalization capability. It has, however, been open that whether this is true in theory. The aim of this paper is to study the theoretical feasibility of ELM by analyzing the pros and cons of ELM. In the previous part of this topic, we pointed out that via appropriately selected activation functions, ELM does not degrade the generalization capability in the sense of expectation. In this paper, we launch the study in a different direction and show that the randomness of ELM also leads to certain negative consequences. On one hand, we find that the randomness causes an additional uncertainty problem of ELM, both in approximation and learning. On the other hand, we theoretically justify that there also exist activation functions such that the corresponding ELM degrades the generalization capability. In particular, we prove that the generalization capability of ELM with Gaussian kernel is essentially worse than that of FNN with Gaussian kernel. To facilitate the use of ELM, we also provide a remedy to such a degradation. We find that the well-developed coefficient regularization technique can essentially improve the generalization capability. The obtained results reveal the essential characteristic of ELM in a certain sense and give theoretical guidance concerning how to use ELM. [ABSTRACT FROM AUTHOR]

In this paper, we consider a modified error-correction learning rule for the multilayer neural network with multivalued neurons (MLMVN). This modification is based on the soft margins technique, which leads to the minimization of the distance between a cluster center and the learning samples belonging to this cluster. MLMVN has a derivative-free learning algorithm based on the error-correction learning rule and demonstrate a higher functionality and better generalization capability than a number of other machine learning techniques. The discrete $k$ -valued multivalued neuron activation function divides a complex plane into $k$ equal sectors. For more efficient and reliable solving of classification problems it is possible to modify the MLMVN learning algorithm in such a way that learning samples belonging to different classes (clusters) will be located as close as possible to the bisector of a desired sector (the cluster center) and as far as possible from each other, respectively. Such a modification based on the soft margins learning technique is considered in this paper. This modified learning algorithm improves the generalization capability of MLMVN when solving classification problems. [ABSTRACT FROM AUTHOR]

Deep neural networks (NNs) are the state-of-the-art models for understanding the content of images and videos. However, implementing deep NNs in embedded systems is a challenging task, e.g., a typical deep belief network could exhaust gigabytes of memory and result in bandwidth and computational bottlenecks. To address this challenge, this paper presents an algorithm and hardware codesign for efficient deep neural computation. A hardware-oriented deep learning algorithm, named the deep adaptive network, is proposed to explore the sparsity of neural connections. By adaptively removing the majority of neural connections and robustly representing the reserved connections using binary integers, the proposed algorithm could save up to 99.9% memory utility and computational resources without undermining classification accuracy. An efficient sparse-mapping-memory-based hardware architecture is proposed to fully take advantage of the algorithmic optimization. Different from traditional Von Neumann architecture, the deep-adaptive network on chip (DANoC) brings communication and computation in close proximity to avoid power-hungry parameter transfers between on-board memory and on-chip computational units. Experiments over different image classification benchmarks show that the DANoC system achieves competitively high accuracy and efficiency comparing with the state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Reinforcement learning has shown great success in helping learning agents accomplish tasks autonomously from environment interactions. Meanwhile in many real-world applications, an agent needs to accomplish not only a fixed task but also a range of tasks. For this goal, an agent can learn a metapolicy over a set of training tasks that are drawn from an underlying distribution. By maximizing the total reward summed over all the training tasks, the metapolicy can then be reused in accomplishing test tasks from the same distribution. However, in practice, we face two major obstacles to train and reuse metapolicies well. First, how to identify tasks that are unrelated or even opposite with each other, in order to avoid their mutual interference in the training. Second, how to characterize task features, according to which a metapolicy can be reused. In this paper, we propose the MetA-Policy LEarning (MAPLE) approach that overcomes the two difficulties by introducing the shallow trail. It probes a task by running a roughly trained policy. Using the rewards of the shallow trail, MAPLE automatically groups similar tasks. Moreover, when the task parameters are unknown, the rewards of the shallow trail also serve as task features. Empirical studies on several controlling tasks verify that MAPLE can train metapolicies well and receives high reward on test tasks. [ABSTRACT FROM AUTHOR]

Sequence transfer learning is of interest in both academia and industry with the emergence of numerous new text domains from Twitter and other social media tools. In this paper, we put forward the data-sensitive granularity for transfer learning, and then, a novel substructural regularization transfer learning model (STLM) is proposed to preserve target domain features at substructural granularity in the light of the condition of labeled data set size. Our model is underpinned by hidden Markov model and regularization theory, where the substructural representation can be integrated as a penalty after measuring the dissimilarity of substructures between target domain and STLM with relative entropy. STLM can achieve the competing goals of preserving the target domain substructure and utilizing the observations from both the target and source domains simultaneously. The estimation of STLM is very efficient since an analytical solution can be derived as a necessary and sufficient condition. The relative usability of substructures to act as regularization parameters and the time complexity of STLM are also analyzed and discussed. Comprehensive experiments of part-of-speech tagging with both Brown and Twitter corpora fully justify that our model can make improvements on all the combinations of source and target domains. [ABSTRACT FROM AUTHOR]