Search

Despite being cost-effective, seven-level Modified Packed U-Cell (MPUC7) active rectifier tends to be unstable due to unequal dc-links. Thus, a multiobjective controller is required to stabilize voltages and currents besides preserving efficiency and power quality. While conventional finite-set model predictive control (FSMPC) can deal with the multiobjective problem, it cannot assure the system stability, and its weighing factors tuning significantly becomes tiresome as the number of objectives increases. This article presents a low-frequency adaptive FSMPC (AMPC) stabilized based on Lyapunov stability theory to overcome the design problems of FSMPC. AMPC handles four control objectives and a decoupled stability objective. The control objectives assure the standard performance of MPUC7 in terms of switching losses, d v /d t , THD, and capacitors ripple. The stability objective guarantees the rectifier reliability under unstable conditions. The weighting factors in AMPC are floating to tackle the tuning challenges where a radial basis function neural network controller (RBFC) adjusts their variations. RBFC is trained by a novel self-training method including particle swarm optimization (PSO) algorithm and some mathematical analyses without using any training data. Experimental and simulation tests also evaluate AMPC in different conditions to confirm its reliability in fulfilling the desired objectives.

There have been 350,000 tweets generated by the interaction of social networks with different cultures and educational backgrounds in the last ten years. Various sentiments are expressed in the user comments, from support to hatred. The sentiments regarded the United States General Election in 2020. This dataset has 3,000 data gotten from previous research. We augment it becomes 15,000 data to facilitate training and increase the required data. Sentiment detection is carried out using the CNN-BiLSTM architecture. It is chosen because CNN can filter essential words, and BiLSTM can remember memory in two directions. By utilizing both, the training process becomes maximum. However, this method has disadvantages in the activation. The drawback of the existing activation method, i.e., "Zero-hard Rectifier" and "ReLU Dropout" problem to become the cause of training stopped in the ReLU activation, and the exponential function cannot be set become the activation function still rigid towards output value in the SERLU activation. To overcome this problem, we propose a novel activation method to repair activation in CNN-BiLSTM architecture. It is namely the ASERLU activation function. It can adjust positive value output, negative value output, and exponential value by the setter variables. So, it adapts more conveniently to the output value and becomes a flexible activation function because it can be increased and decreased as needed. It is the first research applied in architecture. Compared with ReLU and SERLU, our proposed method gives higher accuracy based on the experiment results.

Bottom-up-fabricated crossbars promise superior circuit density and 3-D integrability compared with the traditional CMOS-based implementations. However, their inherent stochasticity presents difficulties in building complex circuits from components that demand precise patterning and high registration accuracies. With fewer terminals than active devices, passive components offer higher device densities and registration tolerances, making them amenable to bottom-up synthesized nanocrossbars. Motivated by this preference for passivity, we explore, in this article, neuromorphic classifiers based on passive neurons and passive synapses. We demonstrate via SPICE simulations how a shallow network of the diode-resistor-based passive rectifier neurons and resistive voltage summers, despite its inherent inability to buffer, amplify, and negate signals, can recognize MNIST digits with 95.4% accuracy. We introduce weight-to-conductance mappings that enable negative weights to be implemented in hardware without excessive memory overheads. The influences of soft and hard defects on the classification performance are evaluated, and the methods to boost fault-tolerance are proposed. The first-order evaluation of the area, speed, and power consumption of the passive multilayer perceptron classifiers is undertaken, and the results are compared with a benchmark study in neuromorphic hardware.

Radioactive particle tracking (RPT) is a minimally invasive nuclear technique that tracks a radioactive particle inside a volume of interest by means of a mathematical location algorithm. During the past decades, many algorithms have been developed including ones based on artificial intelligence techniques. In this study, RPT technique is applied in a simulated test section that employs a simplified mixer filled with concrete, six scintillator detectors and a137Cs radioactive particle emitting gamma rays of 662 keV. The test section was developed using MCNPX code, which is a mathematical code based on Monte Carlo simulation, and 3516 different radioactive particle positions (x,y,z) were simulated. Novelty of this paper is the use of a location algorithm based on a deep learning model, more specifically a 6-layers deep rectifier neural network (DRNN), in which hyperparameters were defined using a Bayesian optimization method. DRNN is a type of deep feedforward neural network that substitutes the usual sigmoid based activation functions, traditionally used in vanilla Multilayer Perceptron Networks, for rectified activation functions. Results show the great accuracy of the DRNN in a RPT tracking system. Root mean squared error for x, y and coordinates of the radioactive particle is, respectively, 0.03064, 0.02523 and 0.07653.

Nowadays, networked controls using cyber-physical systems (CPSs) necessitate engineers considering “faulty signals” into the control from the beginning of the design process. Therefore, synthesizing control methods, which are able to deal with faulty signals and tolerate them, must be thoroughly investigated and integrated into the design process from the commencement. This article proposes an innovative, reliable control based on a sliding mode faulty signal rectifier for active-/reactive-power-controlled, grid-connected voltage-source inverters (named GC-VSIs hereinafter). It is called “faulty-signal-tolerant” control in this article. Those faulty signals can reach the GC-VSI's controls from any sources; for example, they may arise provided that the CPSs malfunction or fail to prevent data-integrity-related issues, cyber threats, and more. The sliding mode algorithm provides the proposed controller with resilient performance via rectifying faulty signals. Besides, the proposed structure is enhanced by an adaptive mechanism, which makes it more robust against the “unknown” nature of faulty signals. The adaptation rule is able to find the unknown bounds of faulty signals (which externally impact control feedback) and incorporate them into the control by the sliding-mode-based faulty signal rectifier to form a faulty-signal-tolerant methodology. Thorough theoretical analyses, including stability assessment using the Lyapunov criterion, are provided in order to design the proposed controller. Comprehensive simulations and experimental results (associated with a GC-VSI) show the effectiveness and reliability of the faulty-signal-tolerant controller, which is proposed in this research.

We analyze the learning problem of one-hidden-layer nonoverlapping convolutional neural networks with the rectified linear unit (ReLU) activation function from the perspective of model estimation. The training outputs are assumed to be generated by the neural network with the unknown ground-truth parameters plus some additive noise, and the objective is to estimate the model parameters by minimizing a nonconvex squared loss function of the training data. Assuming that the training set contains a finite number of samples generated from the Gaussian distribution, we prove that the accelerated gradient descent (GD) algorithm with a proper initialization converges to the ground-truth parameters (up to the noise level) with a linear rate even though the learning problem is nonconvex. Moreover, the convergence rate is proved to be faster than the vanilla GD. The initialization can be achieved by the existing tensor initialization method. In contrast to the existing works that assume an infinite number of samples, we theoretically establish the sample complexity of the required number of training samples. Although the neural network considered here is not deep, this is the first work to show that accelerated GD algorithms can find the global optimizer of the nonconvex learning problem of neural networks. This is also the first work that characterizes the sample complexity of gradient-based methods in learning convolutional neural networks with the nonsmooth ReLU activation function. This work also provides the tightest bound so far of the estimation error with respect to the output noise.

The existing intelligent fault diagnosis methods of rotor-bearing system mainly focus on vibration analysis under steady operation, which has low adaptability to new scenes. In this article, a new framework for rotor-bearing system fault diagnosis under varying working conditions is proposed by using modified convolutional neural network (CNN) with transfer learning. First, infrared thermal images are collected and used to characterize the health condition of rotor-bearing system. Second, modified CNN is developed by introducing stochastic pooling and Leaky rectified linear unit to overcome the training problems in classical CNN. Finally, parameter transfer is used to enable the source modified CNN to adapt to the target domain, which solves the problem of limited available training data in the target domain. The proposed method is applied to analyze thermal images of rotor-bearing system collected under different working conditions. The results show that the proposed method outperforms other cutting edge methods in fault diagnosis of rotor-bearing system.

Machine learning can be a technique of nursing lysis that automatically develops an analytical model. It is a branch of synthetic intelligence that believes that systems are going to learn information, determine patterns of information and decide with degraded human intervention. Machine learning addresses the question of how computers can be constructed that improve mechanically through knowledge. It lies at the intersection of technology and statistics and at the center of artificial data and information science, one in all the quickest increasing technical fields of nowadays. Recent advances in machine learning were driven by the event of latest learning and theories also as by the constant explosion. The event of latest learning algorithms and also theory and the in-progress growth within the accessibility of on-line information also as low-priced computation crystal rectifier to recent progress within the field of machine learning. Additional evidence-based decision-making could be carried out in science, technology and trade, including healthcare, production, education and monetary modelling, enforcement and promotion, with adoption of mechanical learning techniques based on data-intensive methods. The results are also available. The infection can be a life-threatening disease. The bite of a nursing partner is often transmitted in dipterous Anopheles. In infected mosquitoes, plasmodium parasite is a gift. The parasite is discharged into your blood after you bite this dipterous insect once it bites you. Once your body is composed of the parasites, they mature into the liver. The mature parasites enter the blood for several days when red blood cells start to infect. In red blood cells, parasites increase over 48-72 hours, causing infected cells to divide. The parasites still infect red blood cells, which last 2 to 3 days in cycles. This paper is used for observation of protozoan infection with a deep learning idea.

Vibration signals under the same health state often have large differences due to changes in operating conditions. Likewise, the differences among vibration signals under different health states can be small under some operating conditions. Traditional deep learning methods apply fixed nonlinear transformations to all the input signals, which have a negative impact on the discriminative feature learning ability, i.e., projecting the intraclass signals into the same region and the interclass signals into distant regions. Aiming at this issue, this article develops a new activation function, i.e., adaptively parametric rectifier linear units, and inserts the activation function into deep residual networks to improve the feature learning ability, so that each input signal is trained to have its own set of nonlinear transformations. To be specific, a subnetwork is inserted as an embedded module to learn slopes to be used in the nonlinear transformation. The slopes are dependent on the input signal, and thereby the developed method has more flexible nonlinear transformations than the traditional deep learning methods. Finally, the improved performance of the developed method in learning discriminative features has been validated through fault diagnosis applications.

Purpose To improve the performance of a denoising convolutional neural network (DnCNN) and to make it applicable to images with inhomogeneous noise, a refinement involving an activation function (AF) and an application of the refined method for inhomogeneous-noise images was examined in combination with parallelized image denoising. Methods Improvements in the DnCNN were performed by three approaches. One is refinement of the AF of each neural network that constructs the DnCNN. Swish was used in the DnCNN instead of rectifier linear unit. Second, blind noise removal was introduced to the DnCNN in order to adapt spatially variant noises. Third, blind noise removal was applied to parallelized image denoising, referred to herein as ParBID. The ParBID procedure is as follows: (1) adjacent 2D slice images are linearly combined to obtained higher peak SNR (PSNR) images, (2) combined images with different weight coefficients are denoised using the blind DnCNN, and (3) denoised combined images are separated into original position images by algebraic calculation. Results Experimental studies showed that the PSNR and the structural similarity index (SSIM) were improved by using Swish for all noise levels, from 2.5% to 7.5%, as compared to the conventional DnCNN. It was also shown that a well-trained CNN could remove spatially variant noises superimposed on images. Experimental studies with ParBID showed that the greatest PSNR and SSIM improvements were obtained at the middle slice when three slice images were used for linear image combination. More fine structures of images and image contrast remained when the proposed ParBID procedure was used. Conclusion Swish can improve the denoising performance of the DnCNN, and the denoising performance and effectiveness were further improved by ParBID.

Cardiovascular diseases (CVDs) are consider the main cause of death today According to World Health Organization (WHO),and because that ECG signal is very important tool in monitoring and diagnosis of these disease , different automatic methods were proposed based on this signal. [1]. The manual analysis of ECG signals is suffered different challenges such as differeculty of detecting and classify waveform of this signal, So, many machine learning methods are explored to describe the anomalies ECG signal accurately . Deep learning (DL) can be used in ECG classification, it can improve the quality of the automatic classification system. In this paper , we have proposed a deep learning classification system by using different layers of convolution, rectifier and pooling operations that can be used to increase feature extraction of ECG signal. We have proposed two models, one is used for input signal of 1-D, in which we designed model for classification csv type of data for ECG signal, while in the second proposed system, we used model for 2-D signal after convert it from its csv type . 2-D signal (ECG image) is used in order to augment the two dimensional signal with different methods to increase the accuracy of the model by training it with geometric transformation of the original input images such as rotation, shearing etc.The results are compared with AlexNet and other models based on the metrics, which are used to measure the performance of the proposed work, the result show that, the proposed models improve the efficiency of the classification in the two systems.

Activation functions play important roles in determining the depth and non-linearity of deep learning models. Since the Rectified Linear Unit (ReLU) was introduced, many modifications, in which noise is intentionally injected, have been proposed to avoid overfitting. Exponential Linear Unit (ELU) and their variants, with trainable parameters, have been proposed to reduce the bias shift effects which are often observed in ReLU-type activation functions. In this paper, we propose a novel activation function, called the Elastic Exponential Linear Unit (EELU), which combines the advantages of both types of activation functions in a generalized form. EELU has an elastic slope in the positive part, and preserves the negative signal by using a small non-zero gradient. We also present a new strategy to insert neuronal noise using a Gaussian distribution in the activation function to improve generalization. We demonstrated how EELU can represent a wider variety of features with random noise than other activation functions, by visualizing the latent features of convolutional neural networks. We evaluated the effectiveness of the EELU approach through extensive experiments with image classification using the CIFAR-10/CIFAR-100, ImageNet, and Tiny ImageNet datasets. Our experimental results show that EELU achieved better generalization performance and improved classification accuracy over conventional activation functions, such as ReLU, ELU, ReLU- and ELU-like variants, Scaled ELU, and Swish. EELU produced performance improvements in image classification using a smaller number of training samples, owing to its noise injection strategy, which allows significant variation in function outputs, including deactivation.

Maize kernel traits such as kernel length, kernel width, and kernel number determine the total kernel weight and, consequently, maize yield. Therefore, the measurement of kernel traits is important for maize breeding and the evaluation of maize yield. There are a few methods that allow the extraction of ear and kernel features through image processing. We evaluated the potential of deep convolutional neural networks and binary machine learning (ML) algorithms (logistic regression (LR), support vector machine (SVM), AdaBoost (ADB), Classification tree (CART), and the K-Neighbor (kNN)) for accurate maize kernel abortion detection and classification. The algorithms were trained using 75% of 66 total images, and the remaining 25% was used for testing their performance. Confusion matrix, classification accuracy, and precision were the major metrics in evaluating the performance of the algorithms. The SVM and LR algorithms were highly accurate and precise (100%) under all the abortion statuses, while the remaining algorithms had a performance greater than 95%. Deep convolutional neural networks were further evaluated using different activation and optimization techniques. The best performance (100% accuracy) was reached using the rectifier linear unit (ReLu) activation procedure and the Adam optimization technique. Maize ear with abortion were accurately detected by all tested algorithms with minimum training and testing time compared to ear without abortion. The findings suggest that deep convolutional neural networks can be used to detect the maize ear abortion status supplemented with the binary machine learning algorithms in maize breading programs. By using a convolution neural network (CNN) method, more data (big data) can be collected and processed for hundreds of maize ears, accelerating the phenotyping process.

This article focuses on a problem important to automatic machine learning: the automatic processing of a nonpreprocessed time series. The convolutional neural network (CNN) is one of the most popular neural network (NN) algorithms for pattern recognition. Seasonal time series with trends are the most common data sets used in forecasting. Both the convolutional layer and the pooling layer of a CNN can be used to extract important features and patterns that reflect the seasonality, trends, and time lag correlation coefficients in the data. The ability to identify such features and patterns makes CNN a good candidate algorithm for analyzing seasonal time-series data with trends. This article reports our experimental findings using a fully connected NN (FNN), a nonpooling CNN (NPCNN), and a CNN to study both simulated and real time-series data with seasonality and trends. We found that convolutional layers tend to improve the performance, while pooling layers tend to introduce too many negative effects. Therefore, we recommend using an NPCNN when processing seasonal time-series data with trends. Moreover, we suggest using the Adam optimizer and selecting either a rectified linear unit (ReLU) function or a linear activation function. Using an NN to analyze seasonal time series with trends has become popular in the NN community. This article provides an approach for building a network that fits time-series data with seasonality and trends automatically.

In order to realize the unsupervised extraction and identification of fault features in power electronic circuits, we proposed a fault diagnosis method based on sparse autoencoder (SAE) and broad learning system (BLS). Firstly, the feature is extracted by the sparse autoencoder, and the fault samples and feature vectors are combined as the input of the broad learning system. The broad learning system is trained based on the error precision step update method, and the system is used to the fault type identification. The simulation results of the thyristor fault diagnosis of the three-phase bridge rectifier circuit show that the method is effective and has better performance than other traditional methods.

An activation function has crucial role in a deep neural network. A simple rectified linear unit (ReLU) are widely used for the activation function. In this paper, a weighted sigmoid gate unit (WiG) is proposed as the activation function. The proposed WiG consists of a multiplication of inputs and the weighted sigmoid gate. It is shown that the WiG includes the ReLU and same activation functions as a special case. Many activation functions have been proposed to overcome the performance of the ReLU. In the literature, the performance is mainly evaluated with an object recognition task. The proposed WiG is evaluated with the object recognition task and the image restoration task. Then, the expeirmental comparisons demonstrate the proposed WiG overcomes the existing activation functions including the ReLU.

Rectified linear units (ReLUs) are currently the most popular activation function used in neural networks. Although ReLUs can solve the gradient vanishing problem and accelerate training convergence, it suffers from the dying ReLU problem in which some neurons are never activated if the weights are not updated properly. In this work, we propose a novel activation function, known as the adaptive convolutional ReLU (ConvReLU), that can better mimic brain neuron activation behaviors and overcome the dying ReLU problem. With our novel parameter sharing scheme, ConvReLUs can be applied to convolution layers that allow each input neuron to be activated by different trainable thresholds without involving a large number of extra parameters. We employ the zero initialization scheme in ConvReLU to encourage trainable thresholds to be close to zero. Finally, we develop a partial replacement strategy that only replaces the ReLUs in the early layers of the network. This resolves the dying ReLU problem and retains sparse representations for linear classifiers. Experimental results demonstrate that our proposed ConvReLU has consistently better performance compared to ReLU, LeakyReLU, and PReLU. In addition, the partial replacement strategy is shown to be effective not only for our ConvReLU but also for LeakyReLU and PReLU.

A power electronic converter is a switching system; the key to modeling it is how to describe the interaction between the discrete switch variables and the continuous state variables in a unified model. A power electronic converter modeling method based on sigmoid functions is proposed. Since the sigmoid function is continuous, smooth, differentiable, and saturated at a certain value, a sigmoid function with a large steepness factor is used to approximate the switching process of the switch; the converter is transformed from a switching system to a continuous system so that nonlinear continuous system analysis and design methods can be directly applied to the converter control system. Sigmoid function equivalent circuit models for several typical inverter and rectifier circuits are provided. A half-bridge LLC resonant converter is selected as an example, and its sigmoid function model is constructed and subsequently validated through simulation and experimentation. The results show that the constructed model is in good agreement with the circuit simulation and experimental results. The mathematical expression of the model has a simple and unified structure and definite physical meaning and is a high-accuracy large-signal continuous model with universal significance.

A key strategy to enable training of deep neural networks is to use non-saturating activation functions to reduce the vanishing gradient problem. Popular choices that saturate only in the negative domain are the rectified linear unit (ReLU), its smooth, non-linear variant, Softplus, and the exponential linear units (ELU and SELU). Other functions are non-saturating across the entire real domain, like the linear parametric ReLU (PReLU). Here we introduce a nonlinear activation function called Soft++ that extends PReLU and Softplus, parametrizing the slope in the negative domain and the exponent. We test identical network architectures with ReLU, PReLU, Softplus, ELU, SELU, and Soft++ on several machine learning problems and find that: i) convergence of networks with any activation function depends critically on the particular dataset and network architecture, emphasizing the need for parametrization, which allows to adapt the activation function to the particular problem; ii) non-linearity around the origin improves learning and generalization; iii) in many cases, non-saturation across the entire real domain further improves performance. On very difficult learning problems with deep fully-connected and convolutional networks, Soft++ outperforms all other activation functions, accelerating learning and improving generalization. Its main advantage lies in its dual parametrization, offering flexible control of the shape and gradient of the function.

CryptoNets and subsequent work have demonstrated the capability of homomorphic encryption (HE) in the applications of private artificial intelligence (AI). In convolutional neural networks (CNNs), many computations are linear functions such as the convolution layer which can be homomorphically evaluated. However, there are layers such as the activation layer which is comprised of non-linear functions that cannot be homomorphically evaluated. One of the most commonly used methods is approximating these non-linear functions using low-degree polynomials. However, using the approximated polynomials as activation functions introduces errors which could have a significant impact on accuracy in classification tasks. In this paper, we present a systematic method to construct HE-friendly activation functions for CNNs. We first determine what properties in a good activation function contribute to performance by analyzing commonly used functions such as Rectified Linear Units (ReLU) and Sigmoid. Then, we compare polynomial approximation methods and search for an optimal range of approximation for the polynomial activation. We also propose a novel weighted polynomial approximation method tailored to the output distribution of a batch normalization layer. Finally, we demonstrate the effectiveness of our method using several datasets such as MNIST, FMNIST, CIFAR-10.

The ever-increasing computation cost of Convolutional Neural Network (CNN) makes it imperative for real-world applications to accelerate the key steps especially the inference. In this work, we propose an efficient yet general scheme called Sparse Prediction Layer (SPL) which can predict and skip the trivial elements in the CNN layer. Pruned weights are used to predict the locations of maximum values in max-pooling kernels and those of positive values before Rectified Linear Units (ReLUs). Thereafter, the precise values of these predicted important elements are calculated selectively and the complete outputs are restored from them. Our experiments on ImageNet Large Scale Visual Recognition Competition (ILSVRC) 2012 show that SPL can reduce 68.3%, 58.6% and 59.5% Floating-point Operations (FLOPs) on AlexNet, VGG-16 and ResNet-50, respectively, within an accuracy loss of less than 1% without retraining. The proposed SPL scheme can further accelerate these networks pruned by other pruning-based methods, such as a FLOP reduction of 50.2% on the ResNet-50 which has been pruned by Channel Pruning (CP) before being applied with SPLs. A special matrix multiplication called Sparse Result Matrix Multiplication (SRMM) is proposed to support the implementation of SPL, and its acceleration effect is in line with expectations.

The ascent of Industry 4.0 and smart manufacturing has emphasized the use of intelligent manufacturing techniques, tools, and methods such as predictive maintenance. The predictive maintenance function facilitates the early detection of faults and errors in machinery before they reach critical stages. This study suggests the design of an experimental predictive maintenance framework, for conveyor motors, that efficiently detects a conveyor system's impairments and considerably reduces the risk of incorrect faults diagnosis in the plant; We achieve this remarkable task by developing a machine learning model that classifies whether the abnormalities observed are production-threatening or not. We build a classification model using a combination of time-series imaging and convolutional neural network (CNN) for better accuracy. In this research, time-series represent different observations recorded from the machine over time. Our framework is designed to accommodate both univariate and multivariate time-series as inputs of the model, offering more flexibility to prepare for an Industry 4.0 environment. Because multivariate time-series are challenging to manipulate and visualize, we apply a feature extraction approach called principal component analysis (PCA) to reduce their dimensions to a maximum of two channels. The time-series are encoded into images via the Gramian Angular Field (GAF) method and used as inputs to a CNN model. We added a parameterized rectifier linear unit (PReLU) activation function option to the CNN model to improve the performance of more extensive networks. All the features listed added together contribute to the creation of a robust future proof predictive maintenance framework. The experimental results achieved in this study show the advantages of our predictive maintenance framework over traditional classification approaches.

Rotating machinery has been developed with high complexity and precision, and bearings and gears are crucial components in the machinery system. Deep learning has attracted considerable attention from researchers in this area. The convolutional neural network (CNN) is a typical deep learning model that has a strong capability for automatically extracting features from raw data. This capability minimizes dependence on expert knowledge during feature extraction and selection. In CNN, hyperparameters, such as activation functions, can directly influence the performance of the model. In this study, the improved rectified linear units (ReLU)-CNNs are proposed for machinery fault diagnosis. The model's input are raw vibration signals without feature extraction and selection. It is experimentally validated for fault diagnosis using bearing and gearbox datasets. Results show that the proposed method can obtain satisfactory accuracy with enhanced convergence speed. For both datasets, the proposed method gives better diagnosis accrues than the other compared models. The proposed model can take advantages of standard ReLU-CNN, and these advantages can overcome traditional activation functions' vanishing gradient problems. Meanwhile, the improved ReLU-CNN has a new property that makes it perform better than the standard ReLU-CNN.

A method based on a convolutional neural network for the automatic classification of odontocete echolocation clicks is presented. The proposed convolutional neural network comprises six layers: three one-dimensional convolutional layers, two fully connected layers, and a softmax classification layer. Rectified linear units were chosen as the activation function for each convolutional layer. The input to the first convolutional layer is the raw time signal of an echolocation click. Species prediction was performed for groups of m clicks, and two strategies for species label prediction were explored: the majority vote and maximum posterior. Two datasets were used to evaluate the classification performance of the proposed algorithm. Experiments showed that the convolutional neural network can model odontocete species from the raw time signal of echolocation clicks. With the increase in m, the classification accuracy of the proposed method improved. The proposed method can be employed in passive acoustic monitoring to classify different delphinid species and facilitate future studies on odontocetes.

It is of great importance to construct a convolutional neural network architecture in the frequency domain to explore the theory of deep learning in the frequency domain. However, due to the complexity of the construction mechanism of the forward and backward pipelines needed to train the convolutional neural network in the frequency domain, higher requirements are put forward for the representation strategy of the frequency-domain activation function and pooling method in the forward and backward pipelines. Therefore, to construct a full frequency-domain convolutional neural network architecture, it is necessary to construct a frequency-domain representation strategy with a high classification accuracy and excellent time performance. In this paper, based on a chunk decomposition mechanism and the construction principle of the frequency-domain unsaturated activation function, a frequency-domain convolutional neural network architecture is proposed. Two important representation strategies are introduced into the frequency-domain forward/backward pipeline: a frequency-domain randomized offset rectified linear unit and a frequency-domain chunk max pooling method. The former can alleviate the vanishing and exploding gradient phenomena in the frequency-domain forward/backward pipeline and ensure the convergence of the convolutional neural network architecture in the frequency-domain training stage; the latter can capture the partial location information and characteristic strength of the frequency-domain neurons and improve the classification performance of the convolutional neural network in the frequency domain. This full frequency-domain convolutional neural network architecture improves the training accuracy of the convolutional neural network in the frequency-domain pipeline. The results show that on the basis of ResNet-50 as the backbone framework, an NVIDIA GeForce CUDA(Compute Unified Device Architecture) as the training pipeline, and $4\times 4$ as the activation block size of the third-level output neuron’s characteristic parameter matrix, the convolutional neural network architecture proposed in this paper can lower the top-1 error from 24.90% to 17.95%, the top-5 error from 12.85% to 9.23%. Furthermore, when the batch size is equal to 128 (in the worst-case bandwidth usage scenario), the acceleration ratio of the proposed architecture can still reach 13.0375 by selecting cuDNN as the reference model. Under the same backbone framework, the proposed architecture is tested on the MetData-1 dataset, and the classification accuracy can reach the maximum value; that is, the average difference is merely 0.18. This finding shows that the proposed architecture can improve the accuracy of the deep learning-based frequency-domain convolutional neural network model without reducing the time performance and expand the frequency-domain representation strategy of the frequency-domain activation function and pooling method.

In recent years, residual learning has shown excellent performance on convolutional neural network (CNN)-based single-image super-resolution (SISR) tasks. However, CNN-based SISR approaches have focused mainly on the design of deep architectures, and the rectified linear units (ReLUs) used in these networks hinder shallow-to-deep information transfer. As a result, these methods are unable to utilize some shallow information, and improving model performance is difficult. To solve the above issues, this paper proposes an image SR reconstruction method based on a generative adversarial network with a residual dense architecture. First, before ReLU activation, the number of feature channels is expanded by a factor of 6~9 using a $1\times 1$ convolutional layer, which improves the utilization of shallow information. Next, the original discriminator is replaced with a relativistic average discriminator, thereby improving the authenticity of the discriminative network. Finally, preactivation features are used to improve the perceptual loss, thus providing stronger monitoring for brightness consistency and texture restoration. Experimental results show that the proposed algorithm improves the utilization of shallow information in a deep network. Structural similarity (SSIM) index evaluations show that the overall utilization of shallow information is increased by 105.52%. In addition, the average runtime is 0.42 sec/frame, nearly 3.6 times faster than those of traditional methods. Moreover, the recovered images have an average natural image quality evaluator value of 3.4 and high perceptual quality, showing that the proposed method is suitable for image reconstruction applications in fields such as agriculture and medicine.

Penelitian ini bertujuan untuk mengetahui pengaruh National Instruments Multisim (NI-Multisim) Software terhadap penguasaan rangkaian penyearah gelombang ditinjau dari kemampuan visual spasial (KVS) mahasiswa. Penelitian ini menggunakan pra-eksperimental design. Sampel penelitian adalah mahasiswa calon guru fisika yang mengambil mata kuliah Elektronika Dasar I yang dipilih melalui teknik sampling jenuh. Alat pengumpul data berupa instrumen tes kemampuan visual spasial dan instrumen tes rangkaian penyearah gelombang. Uji korelasi, regresi berganda, dan uji independent sample t-test digunakan untuk mengetahui pengaruh kemampuan spasial-visual terhadap hasil belajar rangkaian penyearah gelombang menggunakan NI-Multisim Software. Hasil menunjukkan bahwa: 1) Hasil belajar materi rangkaian penyearah gelombang menggunakan NI-Multisim Software berada pada kategori cukup; 2) Kemampuan visual spasial mahasiswa pada indikator KVS-intepretasi berkategori sangat tinggi, KVS-pola seri berkategori tinggi, KVS-refleksi berkategori tinggi, KVS-rotasi berkategori tinggi, KVS-proyeksi berkategori tinggi, dan KVS-menggambar berkategori sedang; 3) Kemampuan visul spasial mahasiswa memiliki korelasi yang positif dan signifikan terhadap hasil belajar mahasiswa dan memiliki kontribusi pengaruh sebesar 76,20% terhadap hasil belajar; 4) Hasil belajar mahasiswa yang memiliki KVS tinggi, lebih baik daripada rerata hasil belajar mahasiswa yang memiliki KVS rendah dalam mempelajari materi rangakaian penyearah gelombang menggunakan NI-Multisim Software.

Deep convolutional neural networks (DCNNs) are one of the most promising deep learning techniques and have been recognized as the dominant approach for almost all recognition and detection tasks. The computation of DCNNs is memory intensive due to large feature maps and neuron connections, and the performance highly depends on the capability of hardware resources. With the recent trend of wearable devices and Internet of Things, it becomes desirable to integrate the DCNNs onto embedded and portable devices that require low power and energy consumptions and small hardware footprints. Recently stochastic computing (SC)-DCNN demonstrated that SC as a low-cost substitute to binary-based computing radically simplifies the hardware implementation of arithmetic units and has the potential to satisfy the stringent power requirements in embedded devices. In SC, many arithmetic operations that are resource-consuming in binary designs can be implemented with very simple hardware logic, alleviating the extensive computational complexity. It offers a colossal design space for integration and optimization due to its reduced area and soft error resiliency. In this paper, we present HEIF, a highly efficient SC-based inference framework of the large-scale DCNNs, with broad applications including (but not limited to) LeNet-5 and AlexNet , that achieves high energy efficiency and low area/hardware cost. Compared to SC-DCNN, HEIF features: 1) the first (to the best of our knowledge) SC-based rectified linear unit activation function to catch up with the recent advances in software models and mitigate degradation in application-level accuracy; 2) the redesigned approximate parallel counter and optimized stochastic multiplication using transmission gates and inverse mirror adders; and 3) the new optimization of weight storage using clustering. Most importantly, to achieve maximum energy efficiency while maintaining acceptable accuracy, HEIF considers holistic optimizations on cascade connection of function blocks in DCNN, pipelining technique, and bit-stream length reduction. Experimental results show that in large-scale applications HEIF outperforms previous SC-DCNN by the throughput of $4.1\times $ , by area efficiency of up to $6.5\times $ , and achieves up to ${5.6\times }$ energy improvement.

Pedestrian detection and tracking remains a highlight research topic due to its paramount importance in the fields of video surveillance, human-machine interaction, and tracking analysis. At present time, pedestrian detection is still an open problem because of many challenges of image representation in the outdoor and indoor scenes. In recent years, deep learning, in particular Convolutional Neural Networks (CNNs) became the state-of-the-art in terms of accuracy in many computer vision tasks. The unsupervised learning of CNNs is still an open issue. In this paper, we study a matter of feature extraction using a special activation function. Most of CNNs share the same architecture, when each convolutional layer is followed by a nonlinear activation layer. The activation function Rectified Linear Unit (ReLU) is the most widely used as a fast alternative to sigmoid function. We propose a bounded randomized leaky ReLU working in such manner that the angle of linear part with the highest input values is tuned during learning stage, and this linear part can be directed not only upward but also downward using a variable bias for its starting point. The bounded randomized leaky ReLU was tested on Caltech Pedestrian Dataset with promising results.

Neural networks with REctified Linear Unit (ReLU) activation functions (a.k.a. ReLU networks) have achieved great empirical success in various domains. Nonetheless, existing results for learning ReLU networks either pose assumptions on the underlying data distribution being e.g. Gaussian, or require the network size and/or training size to be sufficiently large. In this context, the problem of learning a two-layer ReLU network is approached in a binary classification setting, where the data are linearly separable and a hinge loss criterion is adopted. Leveraging the power of random noise perturbation, this paper presents a novel stochastic gradient descent (SGD) algorithm, which can \emph{provably} train any single-hidden-layer ReLU network to attain global optimality, despite the presence of infinitely many bad local minima, maxima, and saddle points in general. This result is the first of its kind, requiring no assumptions on the data distribution, training/network size, or initialization. Convergence of the resultant iterative algorithm to a global minimum is analyzed by establishing both an upper bound and a lower bound on the number of non-zero updates to be performed. Moreover, generalization guarantees are developed for ReLU networks trained with the novel SGD leveraging classic compression bounds. These guarantees highlight a key difference (at least in the worst case) between reliably learning a ReLU network as well as a leaky ReLU network in terms of sample complexity. Numerical tests using both synthetic data and real images validate the effectiveness of the algorithm and the practical merits of the theory., 23 pages, 7 figures, work in progress

A convolutional neural network (CNN) has recently demonstrated its outstanding capability for the classification of hyperspectral images (HSIs). Typical CNN-based methods usually adopt image patches as inputs to the network. However, a fixed-size image patch in HSI with complex spatial contexts may contain multiple ground objects of different classes, which will deteriorate the classification performance of the CNN. In addition, traditional convolutional layers adopted in the CNN have a huge amount of parameters needed to be tuned, which will cause high computational cost. To address the above-mentioned issues, a novel squeeze multibias network (SMBN) is proposed for HSI classification. Specifically, the proposed SMBN first introduces the multibias module (MBM), which incorporates multibias into the rectified linear unit layers. The MBM can decouple the feature maps of input patches into multiple response maps (corresponding to different ground objects) and adaptively select the meaningful maps for classification. Furthermore, the proposed SMBN replaces the traditional convolutional layer with a squeeze convolution module, which can greatly reduce the number of parameters in the network, thus saving the running time, while still maintaining high classification accuracy. Experimental results on three real HSIs demonstrate the superiority of the proposed SMBN method over several state-of-the-art classification approaches.

Rectified linear unit (ReLU) plays an important role in today's convolutional neural networks (CNNs). In this paper, we propose a novel activation function called Rectified Exponential Unit (REU). Inspired by two recently proposed activation functions: Exponential Linear Unit (ELU) and Swish, the REU is designed by introducing the advantage of flexible exponent and multiplication function form. Moreover, we propose the Parametric REU (PREU) to increase the expressive power of the REU. The experiments with three classical CNN architectures, LeNet-5, Network in Network, and Residual Network (ResNet) on scale-various benchmarks including Fashion-MNIST, CIFAR10, CIFAR100, and Tiny ImageNet demonstrate that REU and PREU achieve improvement compared with other activation functions. Our results show that our REU has relative error improvements over ReLU of 7.74% and 6.08% on CIFAR-10 and 100 with the ResNet, the improvements of PREU is 9.24% and 9.32%. Finally, we use the different PREU variants in the Residual unit to achieve more stable results.

The activation functions play increasingly important roles in deep convolutional neural networks. The traditional activation functions have some problems such as gradient disappearance, neuron death and output offset, and so on. To solve these problems, we propose a new activation function in this paper, Fast Exponentially Linear Unit (FELU), aiming to speed up exponential linear calculations and reduce the time of network running. FELU has the advantages of Rectified Linear Unit (RELU) and Exponential Linear Unit (ELU), leading to have better classification accuracy and faster calculation speed. We test five traditional activation functions such as ReLU, ELU, SLU, MPELU, TReLU, and our new activation function on the cifar10, cifar100 and GTSRB data sets. Experiments show that the proposed activation function FELU not only improves the speed of the exponential calculation, reducing the time of convolutional neural network running, but also effectively enhances the noise robustness of network to improve the accuracy of classification.

Among deep learning methods, convolutional neural networks (CNNs) are able to extract features automatically and have increasingly been used in intelligent fault diagnosis studies. However, studies seldomly concentrate on the weakness associated with a highly imbalanced distribution of fault types due to different failure rates and when multiple faults are easily confused with single faults. To solve these problems, this paper developed a stochastic discrete-time series deep convolutional neural network (SDCNN) method based on random oversampling along with a progressive method with multiple SDCNNs to improve the diagnosis performance. To assess the developed method, datasets from three avionics 24-pulse auto-transformer rectifier units (ATRUs), which are secondary electric power supplies in aircraft, were analyzed and compared with other CNN methods.

Deep Convolutional Neural Networks (CNNs) have been extremely effective in many areas of computer vision, particularly in the domain of facial recognition where many CNN models have delivered a near perfect performance on long-standing benchmarks. A major challenge being faced by CNN architectures today is their need for a large amount of memory and processing power to deliver their exceptional performance. This hampers their deployment on less powerful devices like smart-phones whose ubiquity is unmatched. This paper critically analyzes a CNN based network architecture wherein convolutions are approximated by XNOR operations and bit-counts which are primarily binary operations hence making the network much faster and more memory efficient than standard CNN architectures. The aim of this paper is to assess the performance of the aforementioned architecture for the task of facial recognition with Minimum Barrier Salient Object Detection (MBD) as a preprocessing step and to examine the affect self-normalizing properties induced by the Scaled Exponential Linear Unit (SELU) activation function on the convergence of the network and benchmark it against the convergence performance of the same network incorporating the widely used Rectified Linear Unit (RELU) as its activation function. Two widely known datasets, Faces95 and Faces96, pre-processed using Minimum Barrier Detection (MBD) for saliency detection were used benchmarking the performance of the proposed networks. It was observed that the network using SELU converged faster than RELU for the Faces95 dataset and both converged at similar rates for Faces96 dataset. The network with SELU gave better accuracy than RELU for both the datasets.

This paper proposes an efficient and noise-insensitive end-to-end lightweight deep learning method. The method synthesizes the characteristics of a frequency domain transform and a deep convolutional neural network. The former can extract multiscale information in vibration signal processing and the latter has a good classification performance, data-driven, and high transfer-learning ability. A vibration signal is decomposed into a pyramidal wavelet packet, and each sub-band coefficient is used as an input of a channel in the deep network. A deep residual convolutional network based on a separable convolution and concatenated rectified linear unit (CReLU) lightweight convolution technology is used for fault diagnosis. The proposed algorithm is compared with related deep learning algorithms using two bearing datasets produced by Case Western Reserve University (CWRU) and the Center for Intelligent Maintenance Systems (IMS), University of Cincinnati. Compared with the existing algorithms, the experimental results show that the comprehensive performance of the algorithm proposed in this paper is “small, light, and fast,” and satisfactory diagnostic results are obtained in the fault diagnosis of rotating machinery.

Deep neural networks (DNNs) trained in a supervised way suffer from two known problems. First, the minima of the objective function used in learning correspond to data points (also known as rubbish examples or fooling images) that lack semantic similarity with the training data. Second, a clean input can be changed by a small, and often imperceptible for human vision, perturbation so that the resulting deformed input is misclassified by the network. These findings emphasize the differences between the ways DNNs and humans classify patterns and raise a question of designing learning algorithms that more accurately mimic human perception compared to the existing methods. Our article examines these questions within the framework of dense associative memory (DAM) models. These models are defined by the energy function, with higher-order (higher than quadratic) interactions between the neurons. We show that in the limit when the power of the interaction vertex in the energy function is sufficiently large, these models have the following three properties. First, the minima of the objective function are free from rubbish images, so that each minimum is a semantically meaningful pattern. Second, artificial patterns poised precisely at the decision boundary look ambiguous to human subjects and share aspects of both classes that are separated by that decision boundary. Third, adversarial images constructed by models with small power of the interaction vertex, which are equivalent to DNN with rectified linear units, fail to transfer to and fool the models with higher-order interactions. This opens up the possibility of using higher-order models for detecting and stopping malicious adversarial attacks. The results we present suggest that DAMs with higher-order energy functions are more robust to adversarial and rubbish inputs than DNNs with rectified linear units.

There are a number of problems in science and technology that demand separating useful information from certain content. For many of those problems, standard techniques, as signal processing technique, shape recognition, system control theory, artificial intelligence etc., have shown as inadequate. Neural networks are a way to solve these problems in a way they are solved in human brain. Same as the human brain, neural networks are able to learn from given data, and afterwards, when they meet the same or similar data they may give the same or approximate result. There are several types of transfer functions: sigmoid, logistic sigmoid, linear, semilinear, threshold, Gauss' function. Figure 1 shows the graph for one of most used transfer functions: Fig. 1. Logistic sigmoid function

Until recently, traditional machine learning techniques (TMLTs) such as multilayer perceptrons (MLPs) and support vector machines (SVMs) have been used successfully for churn prediction, but with significant efforts expended on the configuration of the training parameters. The selection of the right training parameters for supervised learning is almost always experimentally determined in an ad hoc manner. Deep neural networks (DNNs) have shown significant predictive strength over TMLTs when used for churn predictions. However, the more complex architecture of DNNs and their capacity to process huge amounts of non-linear input data demand more time and effort to configure the training hyperparameters for DNNs during churn modeling. This makes the process more challenging for inexperienced machine learning practitioners and researchers. So far, limited research has been done to establish the effects of different hyperparameters on the performance of DNNs during churn prediction. There is a lack of empirically derived heuristic knowledge to guide the selection of hyperparameters when DNNs are used for churn modeling. This paper presents an experimental analysis of the effects of different hyperparameters when DNNs are used for churn prediction in the banking sector. The results from three experiments revealed that the deep neural network (DNN) model performed better than the MLP when a rectifier function was used for activation in the hidden layers and a sigmoid function was used in the output layer. The performance of the DNN was better when the batch size was smaller than the size of the test set data, while the RemsProp training algorithm had better accuracy when compared with the stochastic gradient descent (SGD), Adam, AdaGrad, Adadelta, and AdaMax algorithms. The study provides heuristic knowledge that could guide researchers and practitioners in machine learning-based churn prediction from the tabular data for customer relationship management in the banking sector when DNNs are used.

International audience; Comprehensive two−dimensional Gas Chromatography with Vacuum Ultraviolet detection (GC×GC/VUV) results in sizable data for which noise and baseline drift ought to be corrected. As GC×GC/VUV signal is acquired from multiple channels, these pre−processing steps have to be applied to data from all channels while being robust and rather fast with respect to significant size of the GC×GC/VUV data. In this study, we describe advanced GC×GC/VUV data pre−processing techniques for noise and baseline correction that are not available in commercial softwares. Noise reduction was performed on both the spectral and the time dimension. For baseline correction, a morphological approach based on iterated convolutions and rectifier operations is proposed. On the spectral dimension, much less noisy and reliable spectra are obtained. From a quantitative point of view, mentioned pre−processing steps significantly improve signal to noise ratio for analyte detection and hence improve their limit of detection (circa 6 times in this study). These pre−processing methods were integrated into plug im! platform (https://www.plugim.fr/).

Computational modeling of cardiac electrophysiology (EP) has recently transitioned from a scientific research tool to clinical applications. To ensure reliability of clinical or regulatory decisions made using cardiac EP models, it is vital to evaluate the uncertainty in model predictions. Model predictions are uncertain because there is typically substantial uncertainty in model input parameters, due to measurement error or natural variability. While there has been much recent uncertainty quantification (UQ) research for cardiac EP models, all previous work has been limited by either: (i) considering uncertainty in only a subset of the full set of parameters; and/or (ii) assigning arbitrary variation to parameters (e.g., ±10 or 50% around mean value) rather than basing the parameter uncertainty on experimental data. In our recent work we overcame the first limitation by performing UQ and sensitivity analysis using a novel canine action potential model, allowing all parameters to be uncertain, but with arbitrary variation. Here, we address the second limitation by extending our previous work to use data-driven estimates of parameter uncertainty. Overall, we estimated uncertainty due to population variability in all parameters in five currents active during repolarization: inward potassium rectifier, transient outward potassium, L-type calcium, rapidly and slowly activating delayed potassium rectifier; 25 parameters in total (all model parameters except fast sodium current parameters). A variety of methods was used to estimate the variability in these parameters. We then propagated the uncertainties through the model to determine their impact on predictions of action potential shape, action potential duration (APD) prolongation due to drug block, and spiral wave dynamics. Parameter uncertainty had a significant effect on model predictions, especially L-type calcium current parameters. Correlation between physiological parameters was determined to play a role in physiological realism of action potentials. Surprisingly, even model outputs that were relative differences, specifically drug-induced APD prolongation, were heavily impacted by the underlying uncertainty. This is the first data-driven end-to-end UQ analysis in cardiac EP accounting for uncertainty in the vast majority of parameters, including first in tissue, and demonstrates how future UQ could be used to ensure model-based decisions are robust to all underlying parameter uncertainties.

Scene text recognition is the task of recognizing character sequences in images of natural scenes. The considerable diversity in the appearance of text in a scene image and potentially highly complex backgrounds make text recognition challenging. Previous approaches employ character sequence generators to analyze text regions and, subsequently, compare the candidate character sequences against a language model. In this work, we propose a bimodal framework that simultaneously utilizes visual and linguistic information to enhance recognition performance. Our linguistically aware learning (LAL) method effectively learns visual embeddings using a rectifier, encoder, and attention decoder approach, and linguistic embeddings, using a deep next-character prediction model. We present an innovative way of combining these two embeddings effectively. Our experiments on eight standard benchmarks show that our method outperforms previous methods by large margins, particularly on rotated, foreshortened, and curved text. We show that the bimodal approach has a statistically significant impact. We also contribute a new dataset, and show robust performance when LAL is combined with a text detector in a pipelined text spotting framework.

We design a ReLU-based multilayer neural network by mapping the feature vectors to a higher dimensional space in every layer. We design the weight matrices in every layer to ensure a reduction of the training cost as the number of layers increases. Linear projection to the target in the higher dimensional space leads to a lower training cost if a convex cost is minimized. An $\ell_2$-norm convex constraint is used in the minimization to reduce the generalization error and avoid overfitting. The regularization hyperparameters of the network are derived analytically to guarantee a monotonic decrement of the training cost, and therefore, it eliminates the need for cross-validation to find the regularization hyperparameter in each layer. We show that the proposed architecture is norm-preserving and provides an invertible feature vector, and therefore, can be used to reduce the training cost of any other learning method which employs linear projection to estimate the target., Comment: 2020 EURASIP Journal on Advances in Signal Processing

While having achieved great success in rich real-life applications, deep neural network (DNN) models have long been criticized for their vulnerability to adversarial attacks. Tremendous research efforts have been dedicated to mitigating the threats of adversarial attacks, but the essential trait of adversarial examples is not yet clear, and most existing methods are yet vulnerable to hybrid attacks and suffer from counterattacks. In light of this, in this paper, we first reveal a gradient-based correlation between sensitivity analysis-based DNN interpreters and the generation process of adversarial examples, which indicates the Achilles's heel of adversarial attacks and sheds light on linking together the two long-standing challenges of DNN: fragility and unexplainability. We then propose an interpreter-based ensemble framework called X-Ensemble for robust adversary defense. X-Ensemble adopts a novel detection-rectification process and features in building multiple sub-detectors and a rectifier upon various types of interpretation information toward target classifiers. Moreover, X-Ensemble employs the Random Forests (RF) model to combine sub-detectors into an ensemble detector for adversarial hybrid attacks defense. The non-differentiable property of RF further makes it a precious choice against the counterattack of adversaries. Extensive experiments under various types of state-of-the-art attacks and diverse attack scenarios demonstrate the advantages of X-Ensemble to competitive baseline methods., Comment: 10 pages, accepted to KDD'20

An ultra-low power integrate-and-fire neuron array transceiver with a multi-modal neuron architecture is presented. The design features an array of 16 × 16 charge-mode mixed-signal neurons that can be configured to implement a variety of activation functions, including step, sigmoid and Rectified Linear Unit (ReLU), through reconfiguration of clocking waveforms through partial reset in charge accumulation and additive stochastic noise by Linear Feedback Shift Register (LFSR) coupling. The neuron outputs spike-based sparse synchronous events, which are either binary (event/no event) or ternary (positive/negative/no events). The reconfigurable energy-efficient design makes this architecture suitable for deep learning and neuromorphic applications like Restricted Boltzmann Machines, Convolutional Neural Networks and general event-driven computing. The 1.796 mm2 chip fabricated in 130nm CMOS technology consumes 140.6 μW from a 1.8V supply at 92.5 MSpikes/s achieving an energy efficiency Figure-of-Merit (FoM) of 1.52 pJ/Spike. A CNN architecture implemented on the chip using sigmoid and ReLU activation achieves MNIST prediction accuracy of 94.8% and 96.9%.

Previous literature on unsupervised learning focused on designing structural priors with the aim of learning meaningful features. However, this was done without considering the description length of the learned representations which is a direct and unbiased measure of the model complexity. In this paper, first we introduce the $\varphi$ metric that evaluates unsupervised models based on their reconstruction accuracy and the degree of compression of their internal representations. We then present and define two activation functions (Identity, ReLU) as base of reference and three sparse activation functions (top-k absolutes, Extrema-Pool indices, Extrema) as candidate structures that minimize the previously defined $\varphi$. We lastly present Sparsely Activated Networks (SANs) that consist of kernels with shared weights that, during encoding, are convolved with the input and then passed through a sparse activation function. During decoding, the same weights are convolved with the sparse activation map and subsequently the partial reconstructions from each weight are summed to reconstruct the input. We compare SANs using the five previously defined activation functions on a variety of datasets (Physionet, UCI-epilepsy, MNIST, FMNIST) and show that models that are selected using $\varphi$ have small description representation length and consist of interpretable kernels., Comment: 10 pages, 5 figures, 4 algorithms, 4 tables, submission to IEEE Transactions on Neural Networks and Learning Systems

Training neural networks is recently a hot topic in machine learning due to its great success in many applications. Since the neural networks’ training usually involves a highly nonconvex optimization problem, it is difficult to design optimization algorithms with perfect convergence guarantees to derive a neural network estimator of high quality. In this article, we borrow the well-known random sketching strategy from kernel methods to transform the training of shallow rectified linear unit (ReLU) nets into a linear least-squares problem. Using the localized approximation property of shallow ReLU nets and a recently developed dimensionality-leveraging scheme, we succeed in equipping shallow ReLU nets with a specific random sketching scheme. The efficiency of the suggested random sketching strategy is guaranteed by theoretical analysis and also verified via a series of numerical experiments. Theoretically, we show that the proposed random sketching is almost optimal in terms of both approximation capability and learning performance. This implies that random sketching does not degenerate the performance of shallow ReLU nets. Numerically, we show that random sketching can significantly reduce the computational burden of numerous backpropagation (BP) algorithms while maintaining their learning performance.

The choice of activation function can have a large effect on the performance of a neural network. While there have been some attempts to hand-engineer novel activation functions, the Rectified Linear Unit (ReLU) remains the most commonly-used in practice. This paper shows that evolutionary algorithms can discover novel activation functions that outperform ReLU. A tree-based search space of candidate activation functions is defined and explored with mutation, crossover, and exhaustive search. Experiments on training wide residual networks on the CIFAR-10 and CIFAR-100 image datasets show that this approach is effective. Replacing ReLU with evolved activation functions results in statistically significant increases in network accuracy. Optimal performance is achieved when evolution is allowed to customize activation functions to a particular task; however, these novel activation functions are shown to generalize, achieving high performance across tasks. Evolutionary optimization of activation functions is therefore a promising new dimension of metalearning in neural networks., 8 pages; 9 figures/tables; GECCO 2020

Spiking neural networks (SNNs) are considered as the third generation of artificial neural networks, having the potential to improve the energy efficiency of conventional computing systems. Although the firing rate of a spiking neuron is an approximation of rectified linear unit (ReLU) activation in an analog-valued neural network (ANN), there remain many challenges to be overcome owing to differences in operation between ANNs and SNNs. Unlike actual biological and biophysical processes, various hardware implementations of neurons and SNNs do not allow the membrane potential to fall below the resting potential—in other words, neurons must allow the sub-resting membrane potential. Because there occur an excitatory post-synaptic potential (EPSP) as well as an inhibitory post-synaptic potential (IPSP), negatively valued synaptic weights in SNNs induce the sub-resting membrane potential at some time point. If a membrane is not allowed to hold the sub-resting potential, errors will accumulate over time, resulting in inaccurate inference operations. This phenomenon is not observed in ANNs given their use of only spatial synaptic integration, but it can cause serious performance degradation in SNNs. In this paper, we demonstrate the impact of the sub-resting membrane potential on accurate inference operations in SNNs. Moreover, several important considerations for a hardware SNN that can maintain the sub-resting membrane potential are discussed. All of the results in this paper indicate that it is essential for neurons to allow the sub-resting membrane potential in order to realize high-performance SNNs.

In this paper, we present a spatial rectifier to estimate surface normals of tilted images. Tilted images are of particular interest as more visual data are captured by arbitrarily oriented sensors such as body-/robot-mounted cameras. Existing approaches exhibit bounded performance on predicting surface normals because they were trained using gravity-aligned images. Our two main hypotheses are: (1) visual scene layout is indicative of the gravity direction; and (2) not all surfaces are equally represented by a learned estimator due to the structured distribution of the training data, thus, there exists a transformation for each tilted image that is more responsive to the learned estimator than others. We design a spatial rectifier that is learned to transform the surface normal distribution of a tilted image to the rectified one that matches the gravity-aligned training data distribution. Along with the spatial rectifier, we propose a novel truncated angular loss that offers a stronger gradient at smaller angular errors and robustness to outliers. The resulting estimator outperforms the state-of-the-art methods including data augmentation baselines not only on ScanNet and NYUv2 but also on a new dataset called Tilt-RGBD that includes considerable roll and pitch camera motion.