Search

Automatic binary classification of brain magnetic resonance (MR) images has made remarkable progress in the past decade. In comparison, a few pieces of work has been reported on multiclass classification of brain MR images. However, there exist enough scopes for improved automation and accuracy. Most of the existing schemes follow the multi-stage pipeline structure of conventional machine learning framework, where the features are designed manually or hand-crafted. In recent years, deep learning models have attracted great interest from researchers for analyzing medical images that eliminate the traditional steps of machine learning. In this paper, we present an automated method based on deep extreme learning machine (ELM) also termed as multilayer ELM (ML-ELM) for multiclass classification of the pathological brain. ML-ELM is a multilayer architecture stacked with ELM based autoencoders. The effectiveness of leaky rectified linear unit (LReLU) activation function is investigated with ML-ELM. Extensive simulations on a multiclass brain MR image dataset indicate that the ML-ELM with LReLU activation (ML-ELM+LReLU) achieves higher performance with faster training speed compared to its counterparts as well as state-of-the-art schemes. The basic purpose of employing ML-ELM+LReLU algorithm is to eliminate the need for hand-crafted feature extraction and to develop a more stable and generalized system for multiclass brain MR image classification.

During the last few years, object recognition has received a big of interest in an attempt to make use of the large scale image datasets. Object recognition allows understanding image based on the objects that it contains. The success of the deep learning based methods in recognizing objects has encouraged recent works to follow this approach. In this paper, we propose a Boosted Convolutional Neural Network approach for object recognition. Our approach uses a very deep convolutional neural network reinforced by adding Boosted Blocks. These Boosted Blocks consist of a succession of convolutional layers boosted by using a Multi-Bias Nonlinear Activation function, that enriches the expressive power of the network, as well as a Concatenated Rectified Linear Unit function which ensures the preservation of all the information after the convolutional layer. Besides, inspired by the visual system of the human brain, our Boosted Convolutional Layer is designed following a recurrent structure. In addition, rather than using the classical max-pooling, our Boosted Convolutional Neural Network is improved by applying Generalizing pooling which allows pooling to adapt to complex and variable patterns. Furthermore, the Spatial Pyramid Pooling after the last Boosted Block has been conducted after the Boosted Block in order to remove the fixed-size constraint of input image. Our approach is evaluated on four different object recognition benchmarks: Pascal VOC 2007, Pascal VOC 2012, CIFAR-10 and the larger dataset ILSVRC-2012. These datasets are widely used by the recent object recognition methods. The experimental results validate the efficiency of our method in comparison with other methods from the literature.

Rectified linear unit (ReLU) plays an important role in today's convolutional neural networks (CNNs). In this paper, we propose a novel activation function called Rectified Exponential Unit (REU). Inspired by two recently proposed activation functions: Exponential Linear Unit (ELU) and Swish, the REU is designed by introducing the advantage of flexible exponent and multiplication function form. Moreover, we propose the Parametric REU (PREU) to increase the expressive power of the REU. The experiments with three classical CNN architectures, LeNet-5, Network in Network, and Residual Network (ResNet) on scale-various benchmarks including Fashion-MNIST, CIFAR10, CIFAR100, and Tiny ImageNet demonstrate that REU and PREU achieve improvement compared with other activation functions. Our results show that our REU has relative error improvements over ReLU of 7.74% and 6.08% on CIFAR-10 and 100 with the ResNet, the improvements of PREU is 9.24% and 9.32%. Finally, we use the different PREU variants in the Residual unit to achieve more stable results.

The activation functions play increasingly important roles in deep convolutional neural networks. The traditional activation functions have some problems such as gradient disappearance, neuron death and output offset, and so on. To solve these problems, we propose a new activation function in this paper, Fast Exponentially Linear Unit (FELU), aiming to speed up exponential linear calculations and reduce the time of network running. FELU has the advantages of Rectified Linear Unit (RELU) and Exponential Linear Unit (ELU), leading to have better classification accuracy and faster calculation speed. We test five traditional activation functions such as ReLU, ELU, SLU, MPELU, TReLU, and our new activation function on the cifar10, cifar100 and GTSRB data sets. Experiments show that the proposed activation function FELU not only improves the speed of the exponential calculation, reducing the time of convolutional neural network running, but also effectively enhances the noise robustness of network to improve the accuracy of classification.

Among deep learning methods, convolutional neural networks (CNNs) are able to extract features automatically and have increasingly been used in intelligent fault diagnosis studies. However, studies seldomly concentrate on the weakness associated with a highly imbalanced distribution of fault types due to different failure rates and when multiple faults are easily confused with single faults. To solve these problems, this paper developed a stochastic discrete-time series deep convolutional neural network (SDCNN) method based on random oversampling along with a progressive method with multiple SDCNNs to improve the diagnosis performance. To assess the developed method, datasets from three avionics 24-pulse auto-transformer rectifier units (ATRUs), which are secondary electric power supplies in aircraft, were analyzed and compared with other CNN methods.

Deep Convolutional Neural Networks (CNNs) have been extremely effective in many areas of computer vision, particularly in the domain of facial recognition where many CNN models have delivered a near perfect performance on long-standing benchmarks. A major challenge being faced by CNN architectures today is their need for a large amount of memory and processing power to deliver their exceptional performance. This hampers their deployment on less powerful devices like smart-phones whose ubiquity is unmatched. This paper critically analyzes a CNN based network architecture wherein convolutions are approximated by XNOR operations and bit-counts which are primarily binary operations hence making the network much faster and more memory efficient than standard CNN architectures. The aim of this paper is to assess the performance of the aforementioned architecture for the task of facial recognition with Minimum Barrier Salient Object Detection (MBD) as a preprocessing step and to examine the affect self-normalizing properties induced by the Scaled Exponential Linear Unit (SELU) activation function on the convergence of the network and benchmark it against the convergence performance of the same network incorporating the widely used Rectified Linear Unit (RELU) as its activation function. Two widely known datasets, Faces95 and Faces96, pre-processed using Minimum Barrier Detection (MBD) for saliency detection were used benchmarking the performance of the proposed networks. It was observed that the network using SELU converged faster than RELU for the Faces95 dataset and both converged at similar rates for Faces96 dataset. The network with SELU gave better accuracy than RELU for both the datasets.

This paper proposes an efficient and noise-insensitive end-to-end lightweight deep learning method. The method synthesizes the characteristics of a frequency domain transform and a deep convolutional neural network. The former can extract multiscale information in vibration signal processing and the latter has a good classification performance, data-driven, and high transfer-learning ability. A vibration signal is decomposed into a pyramidal wavelet packet, and each sub-band coefficient is used as an input of a channel in the deep network. A deep residual convolutional network based on a separable convolution and concatenated rectified linear unit (CReLU) lightweight convolution technology is used for fault diagnosis. The proposed algorithm is compared with related deep learning algorithms using two bearing datasets produced by Case Western Reserve University (CWRU) and the Center for Intelligent Maintenance Systems (IMS), University of Cincinnati. Compared with the existing algorithms, the experimental results show that the comprehensive performance of the algorithm proposed in this paper is “small, light, and fast,” and satisfactory diagnostic results are obtained in the fault diagnosis of rotating machinery.

Deep neural networks (DNNs) trained in a supervised way suffer from two known problems. First, the minima of the objective function used in learning correspond to data points (also known as rubbish examples or fooling images) that lack semantic similarity with the training data. Second, a clean input can be changed by a small, and often imperceptible for human vision, perturbation so that the resulting deformed input is misclassified by the network. These findings emphasize the differences between the ways DNNs and humans classify patterns and raise a question of designing learning algorithms that more accurately mimic human perception compared to the existing methods. Our article examines these questions within the framework of dense associative memory (DAM) models. These models are defined by the energy function, with higher-order (higher than quadratic) interactions between the neurons. We show that in the limit when the power of the interaction vertex in the energy function is sufficiently large, these models have the following three properties. First, the minima of the objective function are free from rubbish images, so that each minimum is a semantically meaningful pattern. Second, artificial patterns poised precisely at the decision boundary look ambiguous to human subjects and share aspects of both classes that are separated by that decision boundary. Third, adversarial images constructed by models with small power of the interaction vertex, which are equivalent to DNN with rectified linear units, fail to transfer to and fool the models with higher-order interactions. This opens up the possibility of using higher-order models for detecting and stopping malicious adversarial attacks. The results we present suggest that DAMs with higher-order energy functions are more robust to adversarial and rubbish inputs than DNNs with rectified linear units.

There are a number of problems in science and technology that demand separating useful information from certain content. For many of those problems, standard techniques, as signal processing technique, shape recognition, system control theory, artificial intelligence etc., have shown as inadequate. Neural networks are a way to solve these problems in a way they are solved in human brain. Same as the human brain, neural networks are able to learn from given data, and afterwards, when they meet the same or similar data they may give the same or approximate result. There are several types of transfer functions: sigmoid, logistic sigmoid, linear, semilinear, threshold, Gauss' function. Figure 1 shows the graph for one of most used transfer functions: Fig. 1. Logistic sigmoid function

Network science possesses an unreplaceable status in solving social and scientific problems. This study focuses on investigating machine learning based graph mining of large-scale network and optimization by using the California road network dataset from Stanford as the large-scale social network. After building different neural networks with various hyperparameters and selected learning activation functions, accuracy results were compared and conclusions include: 1) Changing the activation functions does not have much effect on accuracy compared to neural network structures, 2) Changing the learning rate does not have much effect either and 3) exponential linear unit (ELU) is sensitive to the change of hidden layer size and kernel compared to rectifier linear unit (ReLU) and hyperbolic tangent (Tanh), causing decrement of accuracy after growth of hidden layer size brings overfitting. These conclusions were drawn based on numeric experiments where the final accuracy rate kept the average of every ten trials. However, given this specific dataset, the accuracy of all the models remain high so that varying neural network parameters or structures does not present much distinctive divergence.

Our proposal consists of developing two novel activation functions in time series anomaly detection, they have the capability to reduce the validation loss. The approach is based on a current activation function in Deep Learning, a very intensive field studied over time, in order to find the most suitable activation in a neural network. In order to achieve this purpose, we used an LSTM (Long Short-Term Memory) Autoencoder architecture, using these two novel functions to see the network’s behavior through introducing them. The key point in our proposal is given by the learnable parameter, assuring more flexibility within the network in weights’ updates, in fact, this property being more powerful than a predefined parameter that will bring a constraint due to its limit. We tested our proposal in comparison to other popular functions such as ReLU (Linear Rectifier Unit), hyperbolic tangent (tanh), Talu activation function. Also, the novelty of this paper consists of taking into consideration of piecewise behavior of an activation function in order to increase the performance of a neural network in Deep Learning.

With its powerful data processing ability and convenient end-to-end characteristics, fault diagnosis method based on deep neural network (DNN) has been widely used in many professional fields. However, the current network models mostly are trained based on the measurement information, and few make use of the prior knowledge of the system of interest. Therefore, a new fault diagnosis method based on an improved graph convolution network (GCN) is proposed. Specifically, this method uses the prior knowledge to construct the structural analysis (SA) method, then the SA is used to pre-diagnose the faults and construct the association graph. Next, the association graph and measurements are sent into the improved GCN model to train the network model iteratively, in which a weight coefficient ( $\theta$ ) is proposed to adjust the influence of measurements and the prior knowledge. A particle swarm optimization algorithm (PSO) is used to find the optimal $\theta$ . Finally the fault diagnosis is realized by trained GCN model. This method makes comprehensive use of measurement information and prior knowledge, and achieves better results than other existing fault diagnosis methods in the experiment. Especially, to achieve the same level of diagnosis performance, much less samples are needed in the proposed method when comparing with the state-of-the-art DNN methods.

Aiming at the problem that the solution space of mapping function from low resolution image to high resolution image is extremely large, which makes it difficult for super-resolution reconstruction models to generate detailed textures, this paper proposes a single image super resolution reconstruction network combining dual attention mechanism. The improved U-Net network model is used as the basic structure, and the data enhancement method is introduced to increase the diversity of samples. The encoder is composed of a convolution layer and an adaptive parameter linear rectifier function (Dynamic ReLU). The image size is decreased step by step through the subsampling operation. The decoder part was composed of Residual Dual Attention Module (RDAM) and Pixelshuffle Module. The image was enlarged gradually through the up-sampling operation, and dual regression loss was introduced to enhance the network constraints. The experimental results show that the proposed method can make the reconstructed image texture more detailed and reduce the possible solution space of the mapping function effectively.

Until recently, traditional machine learning techniques (TMLTs) such as multilayer perceptrons (MLPs) and support vector machines (SVMs) have been used successfully for churn prediction, but with significant efforts expended on the configuration of the training parameters. The selection of the right training parameters for supervised learning is almost always experimentally determined in an ad hoc manner. Deep neural networks (DNNs) have shown significant predictive strength over TMLTs when used for churn predictions. However, the more complex architecture of DNNs and their capacity to process huge amounts of non-linear input data demand more time and effort to configure the training hyperparameters for DNNs during churn modeling. This makes the process more challenging for inexperienced machine learning practitioners and researchers. So far, limited research has been done to establish the effects of different hyperparameters on the performance of DNNs during churn prediction. There is a lack of empirically derived heuristic knowledge to guide the selection of hyperparameters when DNNs are used for churn modeling. This paper presents an experimental analysis of the effects of different hyperparameters when DNNs are used for churn prediction in the banking sector. The results from three experiments revealed that the deep neural network (DNN) model performed better than the MLP when a rectifier function was used for activation in the hidden layers and a sigmoid function was used in the output layer. The performance of the DNN was better when the batch size was smaller than the size of the test set data, while the RemsProp training algorithm had better accuracy when compared with the stochastic gradient descent (SGD), Adam, AdaGrad, Adadelta, and AdaMax algorithms. The study provides heuristic knowledge that could guide researchers and practitioners in machine learning-based churn prediction from the tabular data for customer relationship management in the banking sector when DNNs are used.

International audience; Comprehensive two−dimensional Gas Chromatography with Vacuum Ultraviolet detection (GC×GC/VUV) results in sizable data for which noise and baseline drift ought to be corrected. As GC×GC/VUV signal is acquired from multiple channels, these pre−processing steps have to be applied to data from all channels while being robust and rather fast with respect to significant size of the GC×GC/VUV data. In this study, we describe advanced GC×GC/VUV data pre−processing techniques for noise and baseline correction that are not available in commercial softwares. Noise reduction was performed on both the spectral and the time dimension. For baseline correction, a morphological approach based on iterated convolutions and rectifier operations is proposed. On the spectral dimension, much less noisy and reliable spectra are obtained. From a quantitative point of view, mentioned pre−processing steps significantly improve signal to noise ratio for analyte detection and hence improve their limit of detection (circa 6 times in this study). These pre−processing methods were integrated into plug im! platform (https://www.plugim.fr/).

Object recognition research has made notable steps since the appearance of convolutional neural networks, and many activation functions have been proposed to enhance the classification performance of these networks. Maxout networks have achieved great success in many computer vision tasks, but there is limited information on whether an increase in the number of trainable parameters can increase the performance in Leaky Rectified Linear Unit (LReLU) networks compared to maxout networks. Our experiments compare LReLU, rectified linear unit, scaled exponential linear unit, and hyperbolic tangent to four maxout variants. We evaluate ReLU and LReLU with 2x, 3x and 6x the number of filters in each convolutional layer. We also evaluate ReLU, LReLU and maxout networks with approximately the same number of trainable parameters. Under equal conditions, we found that on average, across all datasets, LReLU performs better than any of the evaluated activation functions.

Background and aim: Diagnosis and treatment of female infertility conditions would help future reproductive planning. Although current deep learning frameworks are able to classify and separately count all types at high accuracy, these solutions suffer from a misclassification error and a high computation complexity due to a positive bias effect and an internal covariate shift. The objective of this paper is to increase the classification accuracy of OFs and to reduce the computational costs of classification via deep learning (DL). Methodology: our framework for follicle classification and counting (FCaC) uses filter-based segmentation. A new method is also proposed to accelerate learning and to normalize the input layer by adjusting and scaling the activations. Our method uses a modified activation function (MAF)- displaced rectifier linear unit (DReLU) and batch normalization (BN) in Feature Extraction and Classification. Therefore, faster and more stable training is achieved by modifying input distribution of an activation function (AF). Results: The proposed system was able to obtain a mean classification accuracy of 97.614%, which is 2.264% more accurate classification than the state-of-the-art. Furthermore, the model processes a single WSI 30% faster (in 10.23 seconds compared to 14.646 seconds processing time of the existing solutions). Conclusion: The proposed system focuses on processing histology images with an accurate classification. It is also faster, has an accelerated convergence and enhanced learning thanks to BN and the EAF. We considered a positive bias effect and internal covariate shift as the main aspects to improve the classification performance.

Computational modeling of cardiac electrophysiology (EP) has recently transitioned from a scientific research tool to clinical applications. To ensure reliability of clinical or regulatory decisions made using cardiac EP models, it is vital to evaluate the uncertainty in model predictions. Model predictions are uncertain because there is typically substantial uncertainty in model input parameters, due to measurement error or natural variability. While there has been much recent uncertainty quantification (UQ) research for cardiac EP models, all previous work has been limited by either: (i) considering uncertainty in only a subset of the full set of parameters; and/or (ii) assigning arbitrary variation to parameters (e.g., ±10 or 50% around mean value) rather than basing the parameter uncertainty on experimental data. In our recent work we overcame the first limitation by performing UQ and sensitivity analysis using a novel canine action potential model, allowing all parameters to be uncertain, but with arbitrary variation. Here, we address the second limitation by extending our previous work to use data-driven estimates of parameter uncertainty. Overall, we estimated uncertainty due to population variability in all parameters in five currents active during repolarization: inward potassium rectifier, transient outward potassium, L-type calcium, rapidly and slowly activating delayed potassium rectifier; 25 parameters in total (all model parameters except fast sodium current parameters). A variety of methods was used to estimate the variability in these parameters. We then propagated the uncertainties through the model to determine their impact on predictions of action potential shape, action potential duration (APD) prolongation due to drug block, and spiral wave dynamics. Parameter uncertainty had a significant effect on model predictions, especially L-type calcium current parameters. Correlation between physiological parameters was determined to play a role in physiological realism of action potentials. Surprisingly, even model outputs that were relative differences, specifically drug-induced APD prolongation, were heavily impacted by the underlying uncertainty. This is the first data-driven end-to-end UQ analysis in cardiac EP accounting for uncertainty in the vast majority of parameters, including first in tissue, and demonstrates how future UQ could be used to ensure model-based decisions are robust to all underlying parameter uncertainties.

Sales forecast is a key issue in pharmacy wholesales and has a direct effect in inventory cost, it also represents a difficult problem, affected by factors like promotions, price changes and seasonal preferences. Traditional sales forecast techniques use historical sales data and recursive formulas, which limit their accuracy. More recently learning-based methods have expanded data features capture capabilities. This paper presents a deep architecture which explores a few years of weekly sales data and learns to makes assertive predictions. The system is assembled with ReLU neurons, whose learning behavior makes possible the effective training of sparse coded deep sub-nets. The developed learning algorithm uses two learning stages where the first produces sparse representation of the studied time series and the second harvest one week ahead predictions using this sparse data. In both cases the reward system is future-focused and favors search for future capacities. To achieving successful network training we develop an algorithms that deals with exploding gradient problem, typical of ReLU networks. The fully assembled predictor automatically learn features from structured data and produces inventories with improved dollar cost. The system has been tested in real time with real data.

The amount of computation and the number of parameters of neural networks are increasing rapidly as the depth of convolutional neural networks (CNNs) is increasing. Therefore, it is very crucial to reduce both the amount of computation and that of memory usage. The pruning method, which compresses a neural network, has been actively studied. Depending on the layer characteristics, the sparsity level of each layer varies significantly after the pruning is conducted. If weights are sparse, most results of convolution operations will be zeroes. Although several studies have proposed methods to utilize the weight sparsity to avoid carrying out meaningless operations, those studies lack consideration that input activations may also have a high sparsity level. The Rectified Linear Unit (ReLU) function is one of the most popular activation functions because it is simple and yet pretty effective. Due to properties of the ReLU function, it is often observed that the input activation sparsity level is high (up to 85%). Therefore, it is important to consider both the input activation sparsity and the weight one to accelerate CNN to minimize carrying out meaningless computation. In this paper, we propose a new acceleration method called Direct Conversion that considers the weight sparsity under the sparse input activation condition. The Direct Conversion method converts a 3D input tensor directly into a compressed format. This method selectively applies one of two different methods: a method called image to Compressed Sparse Row (im2CSR) when input activations are sparse and weights are dense; the other method called image to Compressed Sparse Overlapped Activations (im2CSOA) when both input activations and weights are sparse. Our experimental results show that Direct Conversion improves the inference speed up to 2.82× compared to the conventional method.

Scene text recognition is the task of recognizing character sequences in images of natural scenes. The considerable diversity in the appearance of text in a scene image and potentially highly complex backgrounds make text recognition challenging. Previous approaches employ character sequence generators to analyze text regions and, subsequently, compare the candidate character sequences against a language model. In this work, we propose a bimodal framework that simultaneously utilizes visual and linguistic information to enhance recognition performance. Our linguistically aware learning (LAL) method effectively learns visual embeddings using a rectifier, encoder, and attention decoder approach, and linguistic embeddings, using a deep next-character prediction model. We present an innovative way of combining these two embeddings effectively. Our experiments on eight standard benchmarks show that our method outperforms previous methods by large margins, particularly on rotated, foreshortened, and curved text. We show that the bimodal approach has a statistically significant impact. We also contribute a new dataset, and show robust performance when LAL is combined with a text detector in a pipelined text spotting framework.

We design a ReLU-based multilayer neural network by mapping the feature vectors to a higher dimensional space in every layer. We design the weight matrices in every layer to ensure a reduction of the training cost as the number of layers increases. Linear projection to the target in the higher dimensional space leads to a lower training cost if a convex cost is minimized. An $\ell_2$-norm convex constraint is used in the minimization to reduce the generalization error and avoid overfitting. The regularization hyperparameters of the network are derived analytically to guarantee a monotonic decrement of the training cost, and therefore, it eliminates the need for cross-validation to find the regularization hyperparameter in each layer. We show that the proposed architecture is norm-preserving and provides an invertible feature vector, and therefore, can be used to reduce the training cost of any other learning method which employs linear projection to estimate the target., Comment: 2020 EURASIP Journal on Advances in Signal Processing

Enhancement of the up-to-date computing systems in performance and memory capacity stimulates development of new mathematical models and methods for numerical simulation. Machine learning methods are widely used nowadays in the electronic design automation. New mathematical entities are focused onto increasing the design quality and reducing a time cost. A method of constructing the neuromorphic functional models (NFM) for analog components and functional blocks is proposed. An approach to improvement of the NFM accuracy by partitioning the domain of definition for output characteristics according to the threshold coefficient and using the parallel artificial neural network (ANN) architecture is offered. The automated synthesis route of the NFM is represented. The results of experimental study for semiconductor diode and the voltage rectifier circuit are demonstrated. The accuracy increasing of the synthesized NFM and circuit simulation results shown high efficiency of the proposed method.

While having achieved great success in rich real-life applications, deep neural network (DNN) models have long been criticized for their vulnerability to adversarial attacks. Tremendous research efforts have been dedicated to mitigating the threats of adversarial attacks, but the essential trait of adversarial examples is not yet clear, and most existing methods are yet vulnerable to hybrid attacks and suffer from counterattacks. In light of this, in this paper, we first reveal a gradient-based correlation between sensitivity analysis-based DNN interpreters and the generation process of adversarial examples, which indicates the Achilles's heel of adversarial attacks and sheds light on linking together the two long-standing challenges of DNN: fragility and unexplainability. We then propose an interpreter-based ensemble framework called X-Ensemble for robust adversary defense. X-Ensemble adopts a novel detection-rectification process and features in building multiple sub-detectors and a rectifier upon various types of interpretation information toward target classifiers. Moreover, X-Ensemble employs the Random Forests (RF) model to combine sub-detectors into an ensemble detector for adversarial hybrid attacks defense. The non-differentiable property of RF further makes it a precious choice against the counterattack of adversaries. Extensive experiments under various types of state-of-the-art attacks and diverse attack scenarios demonstrate the advantages of X-Ensemble to competitive baseline methods., Comment: 10 pages, accepted to KDD'20

An ultra-low power integrate-and-fire neuron array transceiver with a multi-modal neuron architecture is presented. The design features an array of 16 × 16 charge-mode mixed-signal neurons that can be configured to implement a variety of activation functions, including step, sigmoid and Rectified Linear Unit (ReLU), through reconfiguration of clocking waveforms through partial reset in charge accumulation and additive stochastic noise by Linear Feedback Shift Register (LFSR) coupling. The neuron outputs spike-based sparse synchronous events, which are either binary (event/no event) or ternary (positive/negative/no events). The reconfigurable energy-efficient design makes this architecture suitable for deep learning and neuromorphic applications like Restricted Boltzmann Machines, Convolutional Neural Networks and general event-driven computing. The 1.796 mm2 chip fabricated in 130nm CMOS technology consumes 140.6 μW from a 1.8V supply at 92.5 MSpikes/s achieving an energy efficiency Figure-of-Merit (FoM) of 1.52 pJ/Spike. A CNN architecture implemented on the chip using sigmoid and ReLU activation achieves MNIST prediction accuracy of 94.8% and 96.9%.

In the artificial intelligence, machine learning and neural network field, many practical use models have been proposed. However, these models are based on a digital Von Neumann model’s computer. There are minority studies in the field of analog learning neural networks. Early analog hardware neural network models were configured with the operational amplifier and the resistance element. It is difficult to change the value of solid resistance for the learning process. In the present paper, proposed is a learning neural network using electronic alternating current (AC) circuits including a voltage-frequency (V-F) converter. These circuits are composed of an amplifier, additional circuit, inverter, subtract circuit, rectifier circuit and V-F converter. The input voltage value was described as the input signal and the input frequency considered the connecting weights. It is easy to change the connecting weights with a V-F converter. Finally, the input frequency converges to a constant value after only several learning process. The learning count time is extremely small. The learning time is quite fast in this AC transmission circuit. The model works using pure analog electronic circuits. The learning time is quite short compare to with a digital process computer.

For highly nonlinear systems, using deep learning models to capture complex dynamics is a promising feature for advanced control applications. Recently it has been shown that a particular class of deep learning models can be exactly recast in a mixed-integer linear programming formulation. Recasting a deep learning model as a set of piecewise linear functions enables the incorporation of advanced predictive models in model-based control strategies such as model predictive control. To alleviate the computational burden of solving the piecewise linear optimization problem online, multiparametric programming is utilized to obtain the full, offline, explicit solution of the optimal control problem. In this work, a strategy is presented for the integration of deep learning models, specifically neural networks with rectified linear units, and explicit model predictive control. The proposed strategy is demonstrated on the advanced control of a benchmark chemical process involving multiple reactors, a flash separator, and a recycle stream. The positive results showcase the relevance and strength of the proposed methodology.

Deep Convolutional Neural Network (DCNN) is widely used as state-of-the art models for image classification. There is a variety of applications related to image classification such as object classification, facial expression classification, and scene classification. In this paper, different activation functions are evaluated for facial expression recognition (FER). The activation functions used for evaluation are Rectified Linear Unit (ReLU), Leaky Rectified Linear Unit (Leaky ReLU), Hyperbolic Tangent Tanh, and Sigmoid. Experiments are conducted on a benchmark dataset known as Fer2013 which is publicly available on Kaggle. Our experiments show that Tanh achieved better performance compared to other activation functions.

Piecewise affine (PWA) control laws often arise in automatic control. A popular example is explicit model predictive control (MPC). On the other hand, it is well known that artificial neural neutworks (ANNs) with rectifier or maxout activations lead to PWA input-output relations. Against this background, it is natural to investigate whether ANNs can be parametrized in such a way that they exactly reproduce the input-output relation of a given PWA function, e.g., a given explicit MPC law. We briefly summarize a known but numerically demanding approach based on maxout networks and provide a novel and efficient method using rectifier networks for the special case of one-dimensional inputs.

Previous literature on unsupervised learning focused on designing structural priors with the aim of learning meaningful features. However, this was done without considering the description length of the learned representations which is a direct and unbiased measure of the model complexity. In this paper, first we introduce the $\varphi$ metric that evaluates unsupervised models based on their reconstruction accuracy and the degree of compression of their internal representations. We then present and define two activation functions (Identity, ReLU) as base of reference and three sparse activation functions (top-k absolutes, Extrema-Pool indices, Extrema) as candidate structures that minimize the previously defined $\varphi$. We lastly present Sparsely Activated Networks (SANs) that consist of kernels with shared weights that, during encoding, are convolved with the input and then passed through a sparse activation function. During decoding, the same weights are convolved with the sparse activation map and subsequently the partial reconstructions from each weight are summed to reconstruct the input. We compare SANs using the five previously defined activation functions on a variety of datasets (Physionet, UCI-epilepsy, MNIST, FMNIST) and show that models that are selected using $\varphi$ have small description representation length and consist of interpretable kernels., Comment: 10 pages, 5 figures, 4 algorithms, 4 tables, submission to IEEE Transactions on Neural Networks and Learning Systems

Training neural networks is recently a hot topic in machine learning due to its great success in many applications. Since the neural networks’ training usually involves a highly nonconvex optimization problem, it is difficult to design optimization algorithms with perfect convergence guarantees to derive a neural network estimator of high quality. In this article, we borrow the well-known random sketching strategy from kernel methods to transform the training of shallow rectified linear unit (ReLU) nets into a linear least-squares problem. Using the localized approximation property of shallow ReLU nets and a recently developed dimensionality-leveraging scheme, we succeed in equipping shallow ReLU nets with a specific random sketching scheme. The efficiency of the suggested random sketching strategy is guaranteed by theoretical analysis and also verified via a series of numerical experiments. Theoretically, we show that the proposed random sketching is almost optimal in terms of both approximation capability and learning performance. This implies that random sketching does not degenerate the performance of shallow ReLU nets. Numerically, we show that random sketching can significantly reduce the computational burden of numerous backpropagation (BP) algorithms while maintaining their learning performance.

The distance relay has come into wide use in the protection of long transmission linea becausenit offers good selectivity without the need for a pilotnchannel, yet can offer a high speed of operation. By itsnvery nature it is often difficult for a student of protection to obtain a good understanding of the operationnand characteristics of a distance relay.This project develops an experiment for power students which deals with the basic operation of a distancenrelay, so that this understanding may be more readily obtained.Only the essential theory relevant to thisnexperiment is presented, as there are many good referencesnalready available on this subject.The construction of the relay built for thisnexperiment is dealt with, and important leatures of thendesign mentioned, such as the design of the summingntransformers, rectifier bridge phase and amplitudencomparators, and level detecting circuitry.Finally a discussion of the goals, practicalndetails, and procedure of the experiment is included,nand limitations and fields of error mentioned.n

The choice of activation function can have a large effect on the performance of a neural network. While there have been some attempts to hand-engineer novel activation functions, the Rectified Linear Unit (ReLU) remains the most commonly-used in practice. This paper shows that evolutionary algorithms can discover novel activation functions that outperform ReLU. A tree-based search space of candidate activation functions is defined and explored with mutation, crossover, and exhaustive search. Experiments on training wide residual networks on the CIFAR-10 and CIFAR-100 image datasets show that this approach is effective. Replacing ReLU with evolved activation functions results in statistically significant increases in network accuracy. Optimal performance is achieved when evolution is allowed to customize activation functions to a particular task; however, these novel activation functions are shown to generalize, achieving high performance across tasks. Evolutionary optimization of activation functions is therefore a promising new dimension of metalearning in neural networks., 8 pages; 9 figures/tables; GECCO 2020

Spiking neural networks (SNNs) are considered as the third generation of artificial neural networks, having the potential to improve the energy efficiency of conventional computing systems. Although the firing rate of a spiking neuron is an approximation of rectified linear unit (ReLU) activation in an analog-valued neural network (ANN), there remain many challenges to be overcome owing to differences in operation between ANNs and SNNs. Unlike actual biological and biophysical processes, various hardware implementations of neurons and SNNs do not allow the membrane potential to fall below the resting potential—in other words, neurons must allow the sub-resting membrane potential. Because there occur an excitatory post-synaptic potential (EPSP) as well as an inhibitory post-synaptic potential (IPSP), negatively valued synaptic weights in SNNs induce the sub-resting membrane potential at some time point. If a membrane is not allowed to hold the sub-resting potential, errors will accumulate over time, resulting in inaccurate inference operations. This phenomenon is not observed in ANNs given their use of only spatial synaptic integration, but it can cause serious performance degradation in SNNs. In this paper, we demonstrate the impact of the sub-resting membrane potential on accurate inference operations in SNNs. Moreover, several important considerations for a hardware SNN that can maintain the sub-resting membrane potential are discussed. All of the results in this paper indicate that it is essential for neurons to allow the sub-resting membrane potential in order to realize high-performance SNNs.

In this paper, we present a spatial rectifier to estimate surface normals of tilted images. Tilted images are of particular interest as more visual data are captured by arbitrarily oriented sensors such as body-/robot-mounted cameras. Existing approaches exhibit bounded performance on predicting surface normals because they were trained using gravity-aligned images. Our two main hypotheses are: (1) visual scene layout is indicative of the gravity direction; and (2) not all surfaces are equally represented by a learned estimator due to the structured distribution of the training data, thus, there exists a transformation for each tilted image that is more responsive to the learned estimator than others. We design a spatial rectifier that is learned to transform the surface normal distribution of a tilted image to the rectified one that matches the gravity-aligned training data distribution. Along with the spatial rectifier, we propose a novel truncated angular loss that offers a stronger gradient at smaller angular errors and robustness to outliers. The resulting estimator outperforms the state-of-the-art methods including data augmentation baselines not only on ScanNet and NYUv2 but also on a new dataset called Tilt-RGBD that includes considerable roll and pitch camera motion.

In this paper, we consider the problem of automatically designing a Rectified Linear Unit (ReLU) Neural Network (NN) architecture (number of layers and number of neurons per layer) with the guarantee that it is sufficiently parametrized to control a nonlinear system. Whereas current state-of-the-art techniques are based on hand-picked architectures or heuristic-based search to find such NN architectures, our approach exploits a given model of the system to design an architecture; as a result, we provide a guarantee that the resulting NN architecture is sufficient to implement a controller that satisfies an achievable specification. Our approach exploits two basic ideas. First, we assume that the system can be controlled by a Lipschitz-continuous state-feedback controller that is unknown but whose Lipschitz constant is upper-bounded by a known constant; then using this assumption, we bound the number of affine functions needed to construct a Continuous Piecewise Affine (CPWA) function that can approximate the unknown Lipschitz-continuous controller. Second, we utilize the authors’ recent results on the Two-Level Lattice (TLL) NN architecture, a novel NN architecture that was shown to be parameterized directly by the number of affine functions that comprise the CPWA function it realizes. We also evaluate our method by designing a NN architecture to control an inverted pendulum.

Data driven learning models have not been successfully implemented due to higher rate of false positive in the detection of breast cancer lesions using mammograms. This research aims to decrease the false positive and increase the accuracy and to keep the independent features as they are. The intelligent system designed uses Fuzzy C as the morphological operators to eliminate the undesired elements of a mammogram. The embedded intelligence for breast cancer in this work has increased the accuracy rate of ~96% compared to existing rate of ~91% leading to the average processing time 0.459S with respect to current timing of 0.598S. Data driven model introduced targets on improving the detection of mass lesions by forcing to ignore undesired texture patterns. Hence, the intelligent system improves the positive rate of mammogram screening by decreasing the false positive rate. Data driven point of care system has been built using convolutional neural networks by applying the Rectifier Liner Unit (ReLU) activation function based on the personalized features extracted for early detection and classification of breast cancer.

Many existing learning algorithms suffer from limited architectural depth and the locality of estimators, making it difficult to generalize from the test set and providing inefficient and biased estimators. Deep architectures have been shown to appropriately learn correlation structures in time series data. This paper compares the effectiveness of a deep feedforward Neural Network (DNN) and shallow architectures (e.g., Support Vector Machine (SVM) and one-layer NN) when predicting a broad cross-section of stock price indices in both developed and emerging markets. An extensive evaluation is undertaken, using daily, hourly, minute and tick level data related to thirty-four financial indices from 32 countries across six years. Our evaluation results show a considerable advantage from training deep (cf. shallow) architectures, using a rectifier linear (RELU) activation function, across all thirty-four markets when ‘minute’ data is used. However, the predictive performance of DNN was not significantly better than that of shallower architectures when using tick level data. This result suggests that when training a DNN algorithm, the predictive accuracy peaks, regardless of training size. We also examine which activation function works best for stock price index data. Our results demonstrate that the RELU activation function performs better than TANH across all markets and time horizons when using DNN to predict stock price indices.

Spiking Neural Networks (SNNs) have recently attracted significant research interest as the third generation of artificial neural networks that can enable low-power event-driven data analytics. The best performing SNNs for image recognition tasks are obtained by converting a trained Analog Neural Network (ANN), consisting of Rectified Linear Units (ReLU), to SNN composed of integrate-and-fire neurons with "proper" firing thresholds. The converted SNNs typically incur loss in accuracy compared to that provided by the original ANN and require sizable number of inference time-steps to achieve the best accuracy. We find that performance degradation in the converted SNN stems from using "hard reset" spiking neuron that is driven to fixed reset potential once its membrane potential exceeds the firing threshold, leading to information loss during SNN inference. We propose ANN-SNN conversion using "soft reset" spiking neuron model, referred to as Residual Membrane Potential (RMP) spiking neuron, which retains the "residual" membrane potential above threshold at the firing instants. We demonstrate near loss-less ANN-SNN conversion using RMP neurons for VGG-16, ResNet-20, and ResNet-34 SNNs on challenging datasets including CIFAR-10 (93.63% top-1), CIFAR-100 (70.93% top-1), and ImageNet (73.09% top-1 accuracy). Our results also show that RMP-SNN surpasses the best inference accuracy provided by the converted SNN with "hard reset" spiking neurons using 2-8 times fewer inference time-steps across network architectures and datasets., Comment: to be published in CVPR'20

Fine-grained domain adaptation is an emerging yet very challenging task in representation learning. In this paper, we analyze a possible reason for the poor performance in fine-grained domain adaptation, which is the difficulty in striking a balance between distribution alignment and fine-grained variations elimination. Furthermore, we propose an adversarial fine-grained domain adaptation framework as a step towards alleviating the underlying conflict between fine-grained variations elimination and domain adaptation. Specifically, our adversarial framework consists of two key modules: a joint label predictor for conditional distribution alignment and a rectifier for fine-grained variations elimination. The key balance can be achieved through the adversarial learning. Besides, experiments on domain adaptation benchmark and fine-grained dataset validate the effectiveness of our framework and show that our framework consistently outperforms the state-of-the-art methods including RTN, MADA, Multi-Task, and DASA.

Neural networks can be used as approximations of several complex control schemes such as model predictive control. We show in this paper which properties deep neural networks with rectifier linear units as activation functions need to satisfy to guarantee constraint satisfaction and asymptotic stability of the closed-loop system. To do so, we introduce a parametric description of the neural network controller and use a mixed-integer linear programming formulation to perform output range analysis of neural networks. We also propose a novel method to modify a neural network controller such that it performs optimally in the LQR sense in a region surrounding the equilibrium. The proposed method enables the analysis and design of neural network controllers with formal safety guarantees as we illustrate with simulation results., Comment: 8 pages, 4 figures

This paper is concerned with the open-circuit fault diagnosis of phase-controlled three-phase full-bridge rectifier by using a sparse autoencoder-based deep neural network (SAE-based DNN). Firstly, some preliminaries on SAE-based DNN are briefly introduced to automatically learn the representative fault features from the raw fault signals. Then, a novel strategy is developed to design the structure of the SAE-based DNN, by which the depth and hidden neurons of the SAE-based DNN could be regularly determined to extract the features of input signals. Furthermore, the fault model and system framework are presented to diagnose the open-circuit fault of the three-phase full-bridge rectifier. Finally, the effectiveness of the developed novel strategy is verified by the results of simulation experiments, and the superiority of the novel SAE-based DNN is evaluated by comparing with other frequently used approaches.

In this paper, we apply an improved deep convolutional neural network (CNN) in fruit category classification, which is a hotspot in computer vision field. We created an 8-layer deep convolutional neural network, and utilized parametric rectified linear unit to take the place of plain rectified linear unit, and placed dropout layer before each fully-connected layer. Data augmentation was used to help avoid overfitting. Our 8-layer deep convolutional neural network secured an overall accuracy of 95.67%. This proposed 8-layer method performs better than five state-of-the-art methods using traditional machine learning methods and one state-of-the-art CNN method.

By introducing sign constraints on the weights, this paper proposes sign constrained rectifier networks (SCRNs), whose training can be solved efficiently by the well known majorization-minimization (MM) algorithms. We prove that the proposed two-hidden-layer SCRNs, which exhibit negative weights in the second hidden layer and negative weights in the output layer, are capable of separating any two (or more) disjoint pattern sets. Furthermore, the proposed two-hidden-layer SCRNs can decompose the patterns of each class into several clusters so that each cluster is convexly separable from all the patterns from the other classes. This provides a means to learn the pattern structures and analyse the discriminant factors between different classes of patterns., Comment: 11 pages

Data transforms, parameter re-normalization, and activation functions have gained significant attention in the neural network community over the past several years for improving convergence speed. The results in the literature are for computer vision applications, with batch-normalization (BN) and the Rectified Linear Unit (ReLU) activation attracting attention. In this paper, we present a new approach in data transformation in the context of regression during the synthesis of Head-related Transfer Functions (HRTFs) in the field of audio. The encoding technique whitens the real-valued input data delivered to the first hidden layer of a fully-connected neural network (FCNN) thereby providing the training speedup. The experimental results demonstrate, in a statistically significant way, that the presented data encoding approach outperforms other forms of normalization in terms of convergence speed, lower mean-square error, and robustness to network parameter initialization. Towards this, we used some popular first-and second-order gradient techniques such as scaled conjugate gradient, Extreme Learning Machine (ELM), and stochastic gradient descent with momentum and batch normalization. The improvements, as shown through t-SNE based depiction and analysis on the input covariance matrix, confirm the reduction in the condition number of the input covariance matrix (a process similar to whitening).

Compact semiconductor device models are essential for efficiently designing and analyzing large circuits. However, traditional compact model development requires a large amount of manual effort and can span many years. Moreover, inclusion of new physics (eg, radiation effects) into an existing compact model is not trivial and may require redevelopment from scratch. Machine Learning (ML) techniques have the potential to automate and significantly speed up the development of compact models. In addition, ML provides a range of modeling options that can be used to develop hierarchies of compact models tailored to specific circuit design stages. In this paper, we explore three such options: (1) table-based interpolation, (2)Generalized Moving Least-Squares, and (3) feed-forward Deep Neural Networks, to develop compact models for a p-n junction diode. We evaluate the performance of these "data-driven" compact models by (1) comparing their voltage-current characteristics against laboratory data, and (2) building a bridge rectifier circuit using these devices, predicting the circuit's behavior using SPICE-like circuit simulations, and then comparing these predictions against laboratory measurements of the same circuit.

Recently, research in machine learning has become more reliant on data-driven approaches. However, understanding the general theory behind optimal neural network architecture is, arguably, just as important. With the proliferation of deep learning and neural networks, finding optimal neural network architecture is vital for both accuracy and performance. Recently, extensive research on neural network architecture has been performed [3],[4]. Additionally, while there has been plenty of research on hidden layer neural network architecture [2], activation functions are often not considered. In a network, an activation function defines the output of a neuron and introduces non-linearities into the neural network, enabling it to be a universal function approximator [12]. In terms of activation functions, one significant paper is Krizhevsky’s seminole work on ImageNet classification and the creation of the ReLU activation function [1]. In the paper, Krizhevsky outlines the construction of an image recognition model using the Rectified Linear Unit activation function (ReLU) for the ImageNET LSVRC-2010 competition which outperformed the state-of-the-art image recognition systems at the time [1]. Since then, ReLU has increased in popularity. In their 2018 conference paper, Bircanoglu and Arica, with the assistance of 231 distinct training procedures, found ReLU to be the best general activation function [12]. In addition to comparisons of activation functions, Nwankpa, Ijomah, Gachagan, and Marshall conducted a meta analysis of the field of research centered around activation functions and found ReLU to be the most popular activation function choice [5]. In terms of optimizers, gradient descent has historically been the most popular loss optimization algorithm, but with Kingma and Ba’s 2014 paper [8], Adam: A Method for Stochastic Optimization, Adam optimizer is slowly becoming the industry standard [11]. In their paper, Kingma and Ba cleverly combine momentum descent, RMSprop, and Adagrad optimization into one algorithm, Adam (or adaptive moment estimation) [8]. In addition to Adam, there are plenty of other optimizers to choose from, including gradient descent, RMSprop [9], Adagrad [10], and Adadelta [7]. Recently, many breakthroughs have been made in terms of neural network performance, improved GPU performance and adaptation to deep learning tasks has created massive efficiency increases for the whole field of machine learning. Furthermore, as machine learning becomes increasingly optimized, the importance of efficiency improvements will continue to rise. Thus, understanding the optimal activation function and optimizer choice for a neural network is relevant. The goal of this paper is to make comparisons between activation functions, optimizers, and, more generally, entire neural network architectures, through measured error in a training environment. In this paper, we examine the performance of a wide variety of neural network configurations on randomly generated polynomial data sets of fixed degree. To do this, we compare various neural network activation functions and optimizers while controlling for hidden layer configurations and degree of the underlying polynomial dataset. Curiously, we find that the Sigmoid activation function is more accurate than ReLU and Tanh for regression tasks on low-featured polynomial data. We also reach the same conclusion regarding Stochastic Gradient Descent (SGD) in comparison to the Adam optimization function and Root Mean Square Propagation (RMSprop). Additionally, we observe that SGD is more efficient in the short term for finding local minimums than Adam or RMSprop; however, after sufficiently many epochs, performance differences between the optimizers vanished.

This study presents a method based on an artificial neural network (ANN) for protecting voltage source converters (VSC)-high-voltage direct current (HVDC) systems. The proposed method takes advantage of ANNs' excellent capabilities in pattern recognition and classification. The solution presented here is able to detect a fault condition in the whole HVDC system based only on voltage waveforms measured at the rectifier substation, which is an important advantage over other fault detection methods. For any given fault condition, the faulty zone is identified and then the fault is classified concerning the phases or poles involved. The proposed method is shown to be both robust against false operations and very reliable with respect to fault detection and classification. A large number of simulations were performed and results show the effectiveness of the proposed scheme. Different operational conditions and fault cases were considered to evaluate the proposed algorithm, ensuring excellent performance concerning a wide range of possible situations in VSC-HVDC systems.

Activation function is crucial to the recent successes of deep neural networks. In this paper, we first propose a new activation function, Multiple Parametric Exponential Linear Units (MPELU), aiming to generalize and unify the rectified and exponential linear units. As the generalized form, MPELU shares the advantages of Parametric Rectified Linear Unit (PReLU) and Exponential Linear Unit (ELU), leading to better classification performance and convergence property. In addition, weight initialization is very important to train very deep networks. The existing methods laid a solid foundation for networks using rectified linear units but not for exponential linear units. This paper complements the current theory and extends it to the wider range. Specifically, we put forward a way of initialization, enabling training of very deep networks using exponential linear units. Experiments demonstrate that the proposed initialization not only helps the training process but leads to better generalization performance. Finally, utilizing the proposed activation function and initialization, we present a deep MPELU residual architecture that achieves state-of-the-art performance on the CIFAR-10/100 datasets. The code is available at https://github.com/Coldmooon/Code-for-MPELU .

In order to obtain prominent recognition accuracy convolutional neural networks (CNNs) need large amounts of labeled data to initialize network parameters. However, there exist two open problems, i.e., the uncertainties of the initialized effects and the limited labeled data To address the problems, we propose a novel method named UPSCNNs, which uses unlabeled data to perform Principal Component Analysis (PCA) and shuffling initialization for CNNs, composed of four steps, i.e. sampling the input images, calculating the sampling sets with PCA and initializing and shuffling the convolutional kernels. In cases with the same network architecture and activation function, i.e., Rectified Linear Units, we conduct the comparative experiments on three image datasets, i.e., STL-10, CIFAR-10(I) and CIFAR-10(II). In terms of accuracy, we find (1) the novel method increases by 4-20 percent in comparison to other weight initialization methods, e.g., Msra initialization, Xavier initialization and Random initialization and (2) an increase of 1-3 percent is obtained with unlabeled data than with only labeled data The results indicate that our method can make full use of unlabeled data for initializing CNNs to achieve good recognition effectiveness.