Search

A residual networks family with hundreds or even thousands of layers dominates major image recognition tasks, but building a network by simply stacking residual blocks inevitably limits its optimization ability. This paper proposes a novel residual network architecture, residual networks of residual networks (RoR), to dig the optimization ability of residual networks. RoR substitutes optimizing residual mapping of residual mapping for optimizing original residual mapping. In particular, RoR adds levelwise shortcut connections upon original residual networks to promote the learning capability of residual networks. More importantly, RoR can be applied to various kinds of residual networks (ResNets, Pre-ResNets, and WRN) and significantly boost their performance. Our experiments demonstrate the effectiveness and versatility of RoR, where it achieves the best performance in all residual-network-like structures. Our RoR-3-WRN58-4 + SD models achieve new state-of-the-art results on CIFAR-10, CIFAR-100, and SVHN, with the test errors of 3.77%, 19.73%, and 1.59%, respectively. RoR-3 models also achieve state-of-the-art results compared with ResNets on the ImageNet data set. [ABSTRACT FROM AUTHOR]

In this paper, we seek to improve deep neural networks by generalizing the pooling operations that play a central role in the current architectures. We pursue a careful exploration of approaches to allow pooling to learn and to adapt to complex and variable patterns. The two primary directions lie in: (1) learning a pooling function via (two strategies of) combining of max and average pooling, and (2) learning a pooling function in the form of a tree-structured fusion of pooling filters that are themselves learned. In our experiments every generalized pooling operation we explore improves performance when used in place of average or max pooling. We experimentally demonstrate that the proposed pooling operations provide a boost in invariance properties relative to conventional pooling and set the state of the art on several widely adopted benchmark datasets. These benefits come with only a light increase in computational overhead during training (ranging from additional 5 to 15 percent in time complexity) and a very modest increase in the number of model parameters (e.g., additional 1, 9, and 27 parameters for mixed, gated, and 2-level tree pooling operators, respectively). To gain more insights about our proposed pooling methods, we also visualize the learned pooling masks and the embeddings of the internal feature responses for different pooling operations. Our proposed pooling operations are easy to implement and can be applied within various deep neural network architectures. [ABSTRACT FROM PUBLISHER]

Virtual network embedding arranges virtual network services onto substrate network components. The performance of embedding algorithms determines the effectiveness and efficiency of a virtualized network, making it a critical part of the network virtualization technology. To achieve better performance, the algorithm needs to automatically detect the network status which is complicated and changes in a time-varying manner, and to dynamically provide solutions that can best fit the current network status. However, most existing algorithms fail to provide automatic embedding solutions in an acceptable running time. In this paper, we combine deep reinforcement learning with a novel neural network structure based on graph convolutional networks, and propose a new and efficient algorithm for automatic virtual network embedding. In addition, a parallel reinforcement learning framework is used in training along with a newly-designed multi-objective reward function, which has proven beneficial to the proposed algorithm for automatic embedding of virtual networks. Extensive simulation results under different scenarios show that our algorithm achieves best performance on most metrics compared with the existing state-of-the-art solutions, with upto 39.6% and 70.6% improvement on acceptance ratio and average revenue, respectively. Moreover, the results also demonstrate that the proposed solution possesses good robustness. [ABSTRACT FROM AUTHOR]

In multilabel learning, each instance in the training set is associated with a set of labels and the task is to output a label set whose size is unknown a priori for each unseen instance. In this paper, this problem is addressed in the way that a neural network algorithm named BP-MLL, i.e., Backpropagation for Multilabel Learning, is proposed. It is derived from the popular Backpropagation algorithm through employing a novel error function capturing the characteristics of multilabel learning, i.e., the labels belonging to an instance should be ranked higher than those not belonging to that instance. Applications to two real-world multilabel learning problems, i.e., functional genomics and text categorization, show that the performance of BP-MLL is superior to that of some well-established multilabel learning algorithms. [ABSTRACT FROM AUTHOR]

Model-based iterative reconstruction algorithms for low-dose X-ray computed tomography (CT) are computationally expensive. To address this problem, we recently proposed a deep convolutional neural network (CNN) for low-dose X-ray CT and won the second place in 2016 AAPM Low-Dose CT Grand Challenge. However, some of the textures were not fully recovered. To address this problem, here we propose a novel framelet-based denoising algorithm using wavelet residual network which synergistically combines the expressive power of deep learning and the performance guarantee from the framelet-based denoising algorithms. The new algorithms were inspired by the recent interpretation of the deep CNN as a cascaded convolution framelet signal representation. Extensive experimental results confirm that the proposed networks have significantly improved performance and preserve the detail texture of the original images. [ABSTRACT FROM AUTHOR]

Following the great success of deep convolutional neural networks (CNNs) in computer vision, this paper proposes a complex-valued CNN (CV-CNN) specifically for synthetic aperture radar (SAR) image interpretation. It utilizes both amplitude and phase information of complex SAR imagery. All elements of CNN including input-output layer, convolution layer, activation function, and pooling layer are extended to the complex domain. Moreover, a complex backpropagation algorithm based on stochastic gradient descent is derived for CV-CNN training. The proposed CV-CNN is then tested on the typical polarimetric SAR image classification task which classifies each pixel into known terrain types via supervised training. Experiments with the benchmark data sets of Flevoland and Oberpfaffenhofen show that the classification error can be further reduced if employing CV-CNN instead of conventional real-valued CNN with the same degrees of freedom. The performance of CV-CNN is comparable to that of existing state-of-the-art methods in terms of overall classification accuracy. [ABSTRACT FROM PUBLISHER]

The paper addresses the problem of predicting hourly load demand using adaptive artificial neural networks (ANNs). A particle swarm optimization (PSO) algorithm is employed to adjust the network's weights in the training phase of the ANNs. The advantage of using a PSO algorithm over other conventional training algorithms such as the back-propagation (BP) is that potential solutions will be flown through the problem hyperspace with accelerated movement towards the best solution. Thus the training phase should result in obtaining the weights configuration associated with the minimum output error. Data are wavelet transformed during the preprocessing stage and then inserted into the neural network to extract redundant information from the load curve. This results in better load characterization which creates a more reliable forecasting model. The transformed data of historical load and weather information were trained and tested over various periods of time. The generalized error estimation is done by using the reverse part of the data as a "test" set. The results were compared with traditional BP algorithm and offered a high forecasting precision. [ABSTRACT FROM AUTHOR]

The process of learning the behavior of a given pro- gram by using machine-learning techniques (based on system-call audit data) is effective to detect intrusions. Rule learning, neural networks, statistics, and hidden Markov models (HMMs) are some of the kinds of representative methods for intrusion detection. Among them, neural networks are known for good performance in learning system-call sequences. In order to apply this knowledge to real-world problems successfully, it is important to determine the structures and weights of these call sequences. However, finding the appropriate structures requires very long time periods be- cause there are no suitable analytical solutions. In this paper, a novel intrusion-detection technique based on evolutionary neural networks (ENNs) is proposed. One advantage of using ENNs is that it takes less time to obtain superior neural networks than when using conventional approaches. This is because they discover the structures and weights of the neural networks simultaneously. Experimental results with the 1999 Defense Advanced Research Projects Agency (DARPA) Intrusion Detection Evaluation (IDEVAL) data confirm that ENNs are promising tools for intrusion detection. [ABSTRACT FROM AUTHOR]

The Levenberg–Marquardt (LM) algorithm is a nonlinear parameter learning algorithm that converges accurately and quickly. This paper demonstrates for the first time to our knowledge, a real-time implementation of the LM algorithm on field programmable gate arrays (FPGAs). It was used to train neural networks to solve the eXclusive Or function (XOR), and for 3D-to-2D camera calibration parameter estimation. A Xilinx Virtex-5 ML506 was used to implement the LMA as a hardware-in-the-loop system. The XOR function was approximated in only 13 iterations from zero initial conditions, usually the same function is approximated in thousands of iterations using the error backpropagation algorithm. Also, this type of training not only reduced the number of iterations but also achieved a speed up in excess of 3 \times 10^6 when compared to the software implementation. A real-time camera calibration and parameter estimation was performed successfully on FPGAs. Compared to the software implementation the FPGA implementation led to an increase in the mean squared error and standard deviation by only 17.94% and 8.04% respectively. The FPGA increased the calibration speed by a factor of 1.41 \times 10^6. There are a wide range of systems problems solved via nonlinear parameter optimization, this study demonstrated that a hardware solution for systems such as automated optical inspection systems or systems dealing with projective geometry estimation and motion compensation systems in robotic vision systems is possible in real time. [ABSTRACT FROM PUBLISHER]

Radial basis function (RBF) networks have advantages of easy design, good generalization, strong tolerance to input noise, and online learning ability. The properties of RBF networks make it very suitable to design flexible control systems. This paper presents a review on different approaches of designing and training RBF networks. The recently developed algorithm is introduced for designing compact RBF networks and performing efficient training process. At last, several problems are applied to test the main properties of RBF networks, including their generalization ability, tolerance to input noise, and online learning ability. RBF networks are also compared with traditional neural networks and fuzzy inference systems. [ABSTRACT FROM AUTHOR]

An NNC-Tree is a decision tree (DT) with each nonterminal node containing a nearest neighbor classifier (NNC). Compared with the conventional axis-parallel DTs (APDTs), the NNC-Trees can be more efficient, because the decision boundary made by an NNC is more complex than an axis- parallel hyperplane. Compared with single-layer NNCs, the NNC-Trees can classify given data in a hierarchical structure that is often useful for many applications. This paper proposes an algorithm for inducing NNC-Trees based on the R4-rule, which was proposed by the author for finding the smallest nearest neighbor based multilayer perceptrons (NN-MLPs). There are mainly two contributions here. 1) A heuristic but effective method is given to define the teacher signals (group labels) for the data assigned to each nonterminal node. 2) The R4-rule is modified so that an NNC with proper size can be designed automatically in each nonterminal node. Experiments with several public databases show that the proposed algorithm can produce NNC-Trees effectively and efficiently. [ABSTRACT FROM AUTHOR]

An ever-increasing number of computer vision and image/video processing challenges are being approached using deep convolutional neural networks, obtaining state-of-the-art results in object recognition and detection, semantic segmentation, action recognition, optical flow, and super resolution. Hardware acceleration of these algorithms is essential to adopt these improvements in embedded and mobile computer vision systems. We present a new architecture, design, and implementation, as well as the first reported silicon measurements of such an accelerator, outperforming previous work in terms of power, area, and I/O efficiency. The manufactured device provides up to 196 GOp/s on 3.09 \text mm^2 of silicon in UMC 65-nm technology and can achieve a power efficiency of 803 GOp/s/W. The massively reduced bandwidth requirements make it the first architecture scalable to TOp/s performance. [ABSTRACT FROM AUTHOR]

The estimation of nested functions (i.e., functions of functions) is one of the central reasons for the success and popularity of machine learning. Today, artificial neural networks are the predominant class of algorithms in this area, known as representational learning. Here, we introduce Representational Gradient Boosting (RGB), a nonparametric algorithm that estimates functions with multi-layer architectures obtained using backpropagation in the space of functions. RGB does not need to assume a functional form in the nodes or output (e.g., linear models or rectified linear units), but rather estimates these transformations. RGB can be seen as an optimized stacking procedure where a meta algorithm learns how to combine different classes of functions (e.g., Neural Networks (NN) and Gradient Boosting (GB)), while building and optimizing them jointly in an attempt to compensate each other’s weaknesses. This highlights a stark difference with current approaches to meta-learning that combine models only after they have been built independently. We showed that providing optimized stacking is one of the main advantages of RGB over current approaches. Additionally, due to the nested nature of RGB we also showed how it improves over GB in problems that have several high-order interactions. Finally, we investigate both theoretically and in practice the problem of recovering nested functions and the value of prior knowledge. [ABSTRACT FROM AUTHOR]

In the backpropagation algorithm, the error calculated from the output of the neural network should backpropagate the layers to update the weights of each layer, making it difficult to parallelize the training process and requiring frequent off-chip memory access. Local learning algorithms locally generate error signals which are used for weight updates, removing the need for backpropagation of error signals. However, prior works rely on large, complex auxiliary networks for reliable training, which results in large computational overhead and undermines the advantages of local learning. In this work, we propose a local learning algorithm that significantly reduces computational complexity as well as improves training performance. Our algorithm combines multiple consecutive layers into a block and performs local learning on a block-by-block basis, while dynamically changing block boundaries during training. In experiments, our approach achieves 95.68% and 79.42% test accuracy on the CIFAR-10 and CIFAR-100 datasets, respectively, using a small fully connected layer as auxiliary networks, closely matching the performance of the backpropagation algorithm. Multiply-accumulate (MAC) operations and off-chip memory access also reduce by up to 15% and 81% compared to backpropagation. [ABSTRACT FROM AUTHOR]

Advanced Chemical-mechanical polishing (CMP) process not only needs to maintain stable run-to-run thickness control but also achieve better within wafer/within chip planarization performance. Furthermore, slurries or other consumable parts, like PAD and Disks selection are also the keys for CMP process optimization. The most difficult thing in CMP process is to have capability to predict and cover the various topologies and layout densities patterned wafers and preventing the hot spots occurrences. In this study, different Neural-Network algorithm with data pre-processing models are implemented to the in-line CMP CLC tuning and dishing/erosion prediction at various topology/pattern density test vehicle pattern wafers. Transfer learning technique is implemented on the original Neural -Network algorithm model, the behavior of real product can be simulated and predicted based on the knowledge of test vehicle database successfully. With the aid of multiple layer erosion/ dishing Neural-Network algorithm model prediction, the potential high risky hot spots can be highlighted at the initial layout design stage, then further shorten the turn-around time of design layout validation. [ABSTRACT FROM AUTHOR]

Reliable and comfortable high-speed railway (HSR) has skyrocketed in popularity as a transportation medium for traveling around the world. High-voltage direct current (HVDC) electrification system has been introduced to the HSR gradually. However, the coexistence of AC and DC systems will last for a long time because AC railway systems are still in the dominant position. A detailed HSR traction system transient model operating under the hybrid AC/DC grid was established in PSCAD/EMTDC. We proposed a real-time fault detection and isolation (FDI) method for the simulated model using neural network (NN). Hierarchical structure of the monitoring system has been employed. Low-level sub-monitors supervised the conditions of their local regions and the top-level monitor collected all the feedback from sub-monitors making the final evaluation of the entire HSR system based on a voting strategy. Both off-line and real-time experiments were conducted to validate the effectiveness of the proposed method. In the experiments, the sub-monitors were designed based on Gated Recurrent Unit (GRU) algorithm and implemented on the Xilinx VCU128 FPGA board. For the off-line experiment, the sub-monitors used the training and testing dataset both from PSCAD/EMTDC to construct the architecture of their individual GRU networks and to verify how great the networks can be. For the real-time task, the sub-monitors interfaced with a real-time HSR system emulator running on the Xilinx VCU118 FPGA board to test the performance in the real-time application. The results proved that our proposed FDI method has the capability of real-time detection and can achieve better accuracy within reasonable time and resource consumption than other NN-based methods. Moreover, the method was capable of standing against noises from measured signals to some extent. [ABSTRACT FROM AUTHOR]

Neural network algorithms such as multilayer perceptrons (MLPs) and radial basis function networks (RBFNets) have been used to construct learners which exhibit strong predictive performance. Two data related issues that can have a detrimental impact on supervised learning initiatives are class imbalance and labeling errors (or class noise). Imbalanced data can make it more difficult for the neural network learning algorithms to distinguish between examples of the various classes, and class noise can lead to the formulation of incorrect hypotheses. Both class imbalance and labeling errors are pervasive problems encountered in a wide variety of application domains. Many studies have been performed to investigate these problems in isolation, but few have focused on their combined effects. This study presents a comprehensive empirical investigation using neural network algorithms to learn from imbalanced data with labeling errors. In particular, the first component of our study investigates the impact of class noise and class imbalance on two common neural network learning algorithms, while the second component considers the ability of data sampling (which is commonly used to address the issue of class imbalance) to improve their performances. Our results, for which over two million models were trained and evaluated, show that conclusions drawn using the more commonly studied C4.5 classifier may not apply when using neural networks. [ABSTRACT FROM AUTHOR]