Search

Distributed fiber-optical acoustic sensing (DAS) is a new and booming technology in seismic exploration. DAS technology has been gradually applied to the exploration of vertical seismic profile (VSP) due to its strong resistance to high temperature and pressure, high sensitivity, high precision (trace interval can be accurate to about 1 m), and so on. However, real DAS-VSP data are always contaminated by both random and coherent noises, which greatly affects the quality of DAS-VSP data. In order to suppress the background noise and increase the signal-to-noise ratio (SNR), a convolutional neural network (CNN) based on leaky rectifier linear unit (ReLU) and forward modeling is proposed and named L-FM-CNN. In terms of network architecture, Leaky ReLU is adopted as the activation function of CNN, which can enhance the recovery ability of trained CNN denoising model to the weak effective signals. As for the training data set, we construct a high-authenticity theoretical pure seismic data set for DAS-VSP data through the complexity of forward models and the diversification of physical parameters. In addition, we propose a new mean square error (MSE) loss function combined with an energy ratio matrix (ERM). The ERM can adjust the SNR between the signal patch and noise patch during the network training and thus increase the robustness of trained CNN denoising model for the DAS-VSP data with different SNRs, especially the DAS-VSP data with extremely low SNR. Both synthetic and real experiments prove the effectiveness of the proposed L-FM-CNN.

There have been 350,000 tweets generated by the interaction of social networks with different cultures and educational backgrounds in the last ten years. Various sentiments are expressed in the user comments, from support to hatred. The sentiments regarded the United States General Election in 2020. This dataset has 3,000 data gotten from previous research. We augment it becomes 15,000 data to facilitate training and increase the required data. Sentiment detection is carried out using the CNN-BiLSTM architecture. It is chosen because CNN can filter essential words, and BiLSTM can remember memory in two directions. By utilizing both, the training process becomes maximum. However, this method has disadvantages in the activation. The drawback of the existing activation method, i.e., "Zero-hard Rectifier" and "ReLU Dropout" problem to become the cause of training stopped in the ReLU activation, and the exponential function cannot be set become the activation function still rigid towards output value in the SERLU activation. To overcome this problem, we propose a novel activation method to repair activation in CNN-BiLSTM architecture. It is namely the ASERLU activation function. It can adjust positive value output, negative value output, and exponential value by the setter variables. So, it adapts more conveniently to the output value and becomes a flexible activation function because it can be increased and decreased as needed. It is the first research applied in architecture. Compared with ReLU and SERLU, our proposed method gives higher accuracy based on the experiment results.

Bottom-up-fabricated crossbars promise superior circuit density and 3-D integrability compared with the traditional CMOS-based implementations. However, their inherent stochasticity presents difficulties in building complex circuits from components that demand precise patterning and high registration accuracies. With fewer terminals than active devices, passive components offer higher device densities and registration tolerances, making them amenable to bottom-up synthesized nanocrossbars. Motivated by this preference for passivity, we explore, in this article, neuromorphic classifiers based on passive neurons and passive synapses. We demonstrate via SPICE simulations how a shallow network of the diode-resistor-based passive rectifier neurons and resistive voltage summers, despite its inherent inability to buffer, amplify, and negate signals, can recognize MNIST digits with 95.4% accuracy. We introduce weight-to-conductance mappings that enable negative weights to be implemented in hardware without excessive memory overheads. The influences of soft and hard defects on the classification performance are evaluated, and the methods to boost fault-tolerance are proposed. The first-order evaluation of the area, speed, and power consumption of the passive multilayer perceptron classifiers is undertaken, and the results are compared with a benchmark study in neuromorphic hardware.

Radioactive particle tracking (RPT) is a minimally invasive nuclear technique that tracks a radioactive particle inside a volume of interest by means of a mathematical location algorithm. During the past decades, many algorithms have been developed including ones based on artificial intelligence techniques. In this study, RPT technique is applied in a simulated test section that employs a simplified mixer filled with concrete, six scintillator detectors and a137Cs radioactive particle emitting gamma rays of 662 keV. The test section was developed using MCNPX code, which is a mathematical code based on Monte Carlo simulation, and 3516 different radioactive particle positions (x,y,z) were simulated. Novelty of this paper is the use of a location algorithm based on a deep learning model, more specifically a 6-layers deep rectifier neural network (DRNN), in which hyperparameters were defined using a Bayesian optimization method. DRNN is a type of deep feedforward neural network that substitutes the usual sigmoid based activation functions, traditionally used in vanilla Multilayer Perceptron Networks, for rectified activation functions. Results show the great accuracy of the DRNN in a RPT tracking system. Root mean squared error for x, y and coordinates of the radioactive particle is, respectively, 0.03064, 0.02523 and 0.07653.

The existing intelligent fault diagnosis methods of rotor-bearing system mainly focus on vibration analysis under steady operation, which has low adaptability to new scenes. In this article, a new framework for rotor-bearing system fault diagnosis under varying working conditions is proposed by using modified convolutional neural network (CNN) with transfer learning. First, infrared thermal images are collected and used to characterize the health condition of rotor-bearing system. Second, modified CNN is developed by introducing stochastic pooling and Leaky rectified linear unit to overcome the training problems in classical CNN. Finally, parameter transfer is used to enable the source modified CNN to adapt to the target domain, which solves the problem of limited available training data in the target domain. The proposed method is applied to analyze thermal images of rotor-bearing system collected under different working conditions. The results show that the proposed method outperforms other cutting edge methods in fault diagnosis of rotor-bearing system.

Machine learning can be a technique of nursing lysis that automatically develops an analytical model. It is a branch of synthetic intelligence that believes that systems are going to learn information, determine patterns of information and decide with degraded human intervention. Machine learning addresses the question of how computers can be constructed that improve mechanically through knowledge. It lies at the intersection of technology and statistics and at the center of artificial data and information science, one in all the quickest increasing technical fields of nowadays. Recent advances in machine learning were driven by the event of latest learning and theories also as by the constant explosion. The event of latest learning algorithms and also theory and the in-progress growth within the accessibility of on-line information also as low-priced computation crystal rectifier to recent progress within the field of machine learning. Additional evidence-based decision-making could be carried out in science, technology and trade, including healthcare, production, education and monetary modelling, enforcement and promotion, with adoption of mechanical learning techniques based on data-intensive methods. The results are also available. The infection can be a life-threatening disease. The bite of a nursing partner is often transmitted in dipterous Anopheles. In infected mosquitoes, plasmodium parasite is a gift. The parasite is discharged into your blood after you bite this dipterous insect once it bites you. Once your body is composed of the parasites, they mature into the liver. The mature parasites enter the blood for several days when red blood cells start to infect. In red blood cells, parasites increase over 48-72 hours, causing infected cells to divide. The parasites still infect red blood cells, which last 2 to 3 days in cycles. This paper is used for observation of protozoan infection with a deep learning idea.

Vibration signals under the same health state often have large differences due to changes in operating conditions. Likewise, the differences among vibration signals under different health states can be small under some operating conditions. Traditional deep learning methods apply fixed nonlinear transformations to all the input signals, which have a negative impact on the discriminative feature learning ability, i.e., projecting the intraclass signals into the same region and the interclass signals into distant regions. Aiming at this issue, this article develops a new activation function, i.e., adaptively parametric rectifier linear units, and inserts the activation function into deep residual networks to improve the feature learning ability, so that each input signal is trained to have its own set of nonlinear transformations. To be specific, a subnetwork is inserted as an embedded module to learn slopes to be used in the nonlinear transformation. The slopes are dependent on the input signal, and thereby the developed method has more flexible nonlinear transformations than the traditional deep learning methods. Finally, the improved performance of the developed method in learning discriminative features has been validated through fault diagnosis applications.

A vital necessity when employing state-of-the-art deep neural networks (DNNs) for topology optimization is to predict near-optimal structures while satisfying pre-defined optimization constraints and objective function. Existing studies, on the other hand, suffer from the structural disconnections which result in unexpected errors in the objective and constraints. In this study, a two-stage network model is proposed using convolutional encoder-decoder networks that incorporate a new way of loss functions to reduce the number of structural disconnection cases as well as to reduce pixel-wise error to enhance the predictive performance of DNNs for topology optimization without any iteration. In the first stage, a single DNN model architecture is proposed and used in two parallel networks using two different loss functions for each called the mean square error (MSE) and mean absolute error (MAE). Once the priori information is generated from the first stage, it is instantly fed into the second stage, which acts as a rectifier network over the priori predictions. Finally, the second stage is trained using the binary cross-entropy (BCE) loss to provide the final predictions. The proposed two-stage network with the proposed loss functions is implemented for both two-dimensional (2D) and three-dimensional (3D) topology optimization datasets to observe its generalization ability. The validation results showed that the proposed two-stage framework could improve network prediction ability compared to a single network while significantly reducing compliance and volume fraction errors.

Purpose To improve the performance of a denoising convolutional neural network (DnCNN) and to make it applicable to images with inhomogeneous noise, a refinement involving an activation function (AF) and an application of the refined method for inhomogeneous-noise images was examined in combination with parallelized image denoising. Methods Improvements in the DnCNN were performed by three approaches. One is refinement of the AF of each neural network that constructs the DnCNN. Swish was used in the DnCNN instead of rectifier linear unit. Second, blind noise removal was introduced to the DnCNN in order to adapt spatially variant noises. Third, blind noise removal was applied to parallelized image denoising, referred to herein as ParBID. The ParBID procedure is as follows: (1) adjacent 2D slice images are linearly combined to obtained higher peak SNR (PSNR) images, (2) combined images with different weight coefficients are denoised using the blind DnCNN, and (3) denoised combined images are separated into original position images by algebraic calculation. Results Experimental studies showed that the PSNR and the structural similarity index (SSIM) were improved by using Swish for all noise levels, from 2.5% to 7.5%, as compared to the conventional DnCNN. It was also shown that a well-trained CNN could remove spatially variant noises superimposed on images. Experimental studies with ParBID showed that the greatest PSNR and SSIM improvements were obtained at the middle slice when three slice images were used for linear image combination. More fine structures of images and image contrast remained when the proposed ParBID procedure was used. Conclusion Swish can improve the denoising performance of the DnCNN, and the denoising performance and effectiveness were further improved by ParBID.

Cardiovascular diseases (CVDs) are consider the main cause of death today According to World Health Organization (WHO),and because that ECG signal is very important tool in monitoring and diagnosis of these disease , different automatic methods were proposed based on this signal. [1]. The manual analysis of ECG signals is suffered different challenges such as differeculty of detecting and classify waveform of this signal, So, many machine learning methods are explored to describe the anomalies ECG signal accurately . Deep learning (DL) can be used in ECG classification, it can improve the quality of the automatic classification system. In this paper , we have proposed a deep learning classification system by using different layers of convolution, rectifier and pooling operations that can be used to increase feature extraction of ECG signal. We have proposed two models, one is used for input signal of 1-D, in which we designed model for classification csv type of data for ECG signal, while in the second proposed system, we used model for 2-D signal after convert it from its csv type . 2-D signal (ECG image) is used in order to augment the two dimensional signal with different methods to increase the accuracy of the model by training it with geometric transformation of the original input images such as rotation, shearing etc.The results are compared with AlexNet and other models based on the metrics, which are used to measure the performance of the proposed work, the result show that, the proposed models improve the efficiency of the classification in the two systems.

Deep learning is drawing keen attention in contemporary financial research. In this article, the authors investigate the statistical predictive power and economic significance of financial stock market data by using deep learning techniques. In particular, the authors use the equity premium as the response variable and financial variables as predictors. The deep learning techniques used in this study provide useful evidence of statistical predictability and economic significance. Considering the statistical predictive performance of the deep learning models, H2O deep learning (H2ODL) gives the smallest mean-squared forecast error (MSFE), with the corresponding highest cumulative return (CR) and Sharpe ratio (SR) in each of the out-of-sample periods. Specifically, the H2ODL with Rectifier used as the activation function outperformed the other models in this article. In the fusion results, the SAE-with-H2O using the Maxout activation function yields the smallest MSFE with the corresponding highest CR and SR in all of the out-of-sample periods. It is worth noting that the higher the CR, the higher the SR and the lower the MSFE, which concords with a rule of thumb. Overall, the empirical analysis in this study revealed that the SAE-with-H2O using the Maxout activation function produced the best statistically predictive and economically significant results with robustness across all out-of-sample periods. TOPICS:Big data/machine learning, performance measurement, quantitative methods, simulations, statistical methods Key Findings ▪ In this article, the authors use deep learning models to predict the equity premium, employing a plethora of well-known predictors. ▪ The authors employ deep learning models such as deep neural networks, a stacked autoencoder, and long short-term memory models. ▪ The statistical and economic significance of the proposed models is examined and back tested in three out-of-sample periods.

Activation functions play important roles in determining the depth and non-linearity of deep learning models. Since the Rectified Linear Unit (ReLU) was introduced, many modifications, in which noise is intentionally injected, have been proposed to avoid overfitting. Exponential Linear Unit (ELU) and their variants, with trainable parameters, have been proposed to reduce the bias shift effects which are often observed in ReLU-type activation functions. In this paper, we propose a novel activation function, called the Elastic Exponential Linear Unit (EELU), which combines the advantages of both types of activation functions in a generalized form. EELU has an elastic slope in the positive part, and preserves the negative signal by using a small non-zero gradient. We also present a new strategy to insert neuronal noise using a Gaussian distribution in the activation function to improve generalization. We demonstrated how EELU can represent a wider variety of features with random noise than other activation functions, by visualizing the latent features of convolutional neural networks. We evaluated the effectiveness of the EELU approach through extensive experiments with image classification using the CIFAR-10/CIFAR-100, ImageNet, and Tiny ImageNet datasets. Our experimental results show that EELU achieved better generalization performance and improved classification accuracy over conventional activation functions, such as ReLU, ELU, ReLU- and ELU-like variants, Scaled ELU, and Swish. EELU produced performance improvements in image classification using a smaller number of training samples, owing to its noise injection strategy, which allows significant variation in function outputs, including deactivation.

Maize kernel traits such as kernel length, kernel width, and kernel number determine the total kernel weight and, consequently, maize yield. Therefore, the measurement of kernel traits is important for maize breeding and the evaluation of maize yield. There are a few methods that allow the extraction of ear and kernel features through image processing. We evaluated the potential of deep convolutional neural networks and binary machine learning (ML) algorithms (logistic regression (LR), support vector machine (SVM), AdaBoost (ADB), Classification tree (CART), and the K-Neighbor (kNN)) for accurate maize kernel abortion detection and classification. The algorithms were trained using 75% of 66 total images, and the remaining 25% was used for testing their performance. Confusion matrix, classification accuracy, and precision were the major metrics in evaluating the performance of the algorithms. The SVM and LR algorithms were highly accurate and precise (100%) under all the abortion statuses, while the remaining algorithms had a performance greater than 95%. Deep convolutional neural networks were further evaluated using different activation and optimization techniques. The best performance (100% accuracy) was reached using the rectifier linear unit (ReLu) activation procedure and the Adam optimization technique. Maize ear with abortion were accurately detected by all tested algorithms with minimum training and testing time compared to ear without abortion. The findings suggest that deep convolutional neural networks can be used to detect the maize ear abortion status supplemented with the binary machine learning algorithms in maize breading programs. By using a convolution neural network (CNN) method, more data (big data) can be collected and processed for hundreds of maize ears, accelerating the phenotyping process.

This article focuses on a problem important to automatic machine learning: the automatic processing of a nonpreprocessed time series. The convolutional neural network (CNN) is one of the most popular neural network (NN) algorithms for pattern recognition. Seasonal time series with trends are the most common data sets used in forecasting. Both the convolutional layer and the pooling layer of a CNN can be used to extract important features and patterns that reflect the seasonality, trends, and time lag correlation coefficients in the data. The ability to identify such features and patterns makes CNN a good candidate algorithm for analyzing seasonal time-series data with trends. This article reports our experimental findings using a fully connected NN (FNN), a nonpooling CNN (NPCNN), and a CNN to study both simulated and real time-series data with seasonality and trends. We found that convolutional layers tend to improve the performance, while pooling layers tend to introduce too many negative effects. Therefore, we recommend using an NPCNN when processing seasonal time-series data with trends. Moreover, we suggest using the Adam optimizer and selecting either a rectified linear unit (ReLU) function or a linear activation function. Using an NN to analyze seasonal time series with trends has become popular in the NN community. This article provides an approach for building a network that fits time-series data with seasonality and trends automatically.

In order to realize the unsupervised extraction and identification of fault features in power electronic circuits, we proposed a fault diagnosis method based on sparse autoencoder (SAE) and broad learning system (BLS). Firstly, the feature is extracted by the sparse autoencoder, and the fault samples and feature vectors are combined as the input of the broad learning system. The broad learning system is trained based on the error precision step update method, and the system is used to the fault type identification. The simulation results of the thyristor fault diagnosis of the three-phase bridge rectifier circuit show that the method is effective and has better performance than other traditional methods.

The ascent of Industry 4.0 and smart manufacturing has emphasized the use of intelligent manufacturing techniques, tools, and methods such as predictive maintenance. The predictive maintenance function facilitates the early detection of faults and errors in machinery before they reach critical stages. This study suggests the design of an experimental predictive maintenance framework, for conveyor motors, that efficiently detects a conveyor system's impairments and considerably reduces the risk of incorrect faults diagnosis in the plant; We achieve this remarkable task by developing a machine learning model that classifies whether the abnormalities observed are production-threatening or not. We build a classification model using a combination of time-series imaging and convolutional neural network (CNN) for better accuracy. In this research, time-series represent different observations recorded from the machine over time. Our framework is designed to accommodate both univariate and multivariate time-series as inputs of the model, offering more flexibility to prepare for an Industry 4.0 environment. Because multivariate time-series are challenging to manipulate and visualize, we apply a feature extraction approach called principal component analysis (PCA) to reduce their dimensions to a maximum of two channels. The time-series are encoded into images via the Gramian Angular Field (GAF) method and used as inputs to a CNN model. We added a parameterized rectifier linear unit (PReLU) activation function option to the CNN model to improve the performance of more extensive networks. All the features listed added together contribute to the creation of a robust future proof predictive maintenance framework. The experimental results achieved in this study show the advantages of our predictive maintenance framework over traditional classification approaches.

Rotating machinery has been developed with high complexity and precision, and bearings and gears are crucial components in the machinery system. Deep learning has attracted considerable attention from researchers in this area. The convolutional neural network (CNN) is a typical deep learning model that has a strong capability for automatically extracting features from raw data. This capability minimizes dependence on expert knowledge during feature extraction and selection. In CNN, hyperparameters, such as activation functions, can directly influence the performance of the model. In this study, the improved rectified linear units (ReLU)-CNNs are proposed for machinery fault diagnosis. The model's input are raw vibration signals without feature extraction and selection. It is experimentally validated for fault diagnosis using bearing and gearbox datasets. Results show that the proposed method can obtain satisfactory accuracy with enhanced convergence speed. For both datasets, the proposed method gives better diagnosis accrues than the other compared models. The proposed model can take advantages of standard ReLU-CNN, and these advantages can overcome traditional activation functions' vanishing gradient problems. Meanwhile, the improved ReLU-CNN has a new property that makes it perform better than the standard ReLU-CNN.

A method based on a convolutional neural network for the automatic classification of odontocete echolocation clicks is presented. The proposed convolutional neural network comprises six layers: three one-dimensional convolutional layers, two fully connected layers, and a softmax classification layer. Rectified linear units were chosen as the activation function for each convolutional layer. The input to the first convolutional layer is the raw time signal of an echolocation click. Species prediction was performed for groups of m clicks, and two strategies for species label prediction were explored: the majority vote and maximum posterior. Two datasets were used to evaluate the classification performance of the proposed algorithm. Experiments showed that the convolutional neural network can model odontocete species from the raw time signal of echolocation clicks. With the increase in m, the classification accuracy of the proposed method improved. The proposed method can be employed in passive acoustic monitoring to classify different delphinid species and facilitate future studies on odontocetes.

It is of great importance to construct a convolutional neural network architecture in the frequency domain to explore the theory of deep learning in the frequency domain. However, due to the complexity of the construction mechanism of the forward and backward pipelines needed to train the convolutional neural network in the frequency domain, higher requirements are put forward for the representation strategy of the frequency-domain activation function and pooling method in the forward and backward pipelines. Therefore, to construct a full frequency-domain convolutional neural network architecture, it is necessary to construct a frequency-domain representation strategy with a high classification accuracy and excellent time performance. In this paper, based on a chunk decomposition mechanism and the construction principle of the frequency-domain unsaturated activation function, a frequency-domain convolutional neural network architecture is proposed. Two important representation strategies are introduced into the frequency-domain forward/backward pipeline: a frequency-domain randomized offset rectified linear unit and a frequency-domain chunk max pooling method. The former can alleviate the vanishing and exploding gradient phenomena in the frequency-domain forward/backward pipeline and ensure the convergence of the convolutional neural network architecture in the frequency-domain training stage; the latter can capture the partial location information and characteristic strength of the frequency-domain neurons and improve the classification performance of the convolutional neural network in the frequency domain. This full frequency-domain convolutional neural network architecture improves the training accuracy of the convolutional neural network in the frequency-domain pipeline. The results show that on the basis of ResNet-50 as the backbone framework, an NVIDIA GeForce CUDA(Compute Unified Device Architecture) as the training pipeline, and $4\times 4$ as the activation block size of the third-level output neuron’s characteristic parameter matrix, the convolutional neural network architecture proposed in this paper can lower the top-1 error from 24.90% to 17.95%, the top-5 error from 12.85% to 9.23%. Furthermore, when the batch size is equal to 128 (in the worst-case bandwidth usage scenario), the acceleration ratio of the proposed architecture can still reach 13.0375 by selecting cuDNN as the reference model. Under the same backbone framework, the proposed architecture is tested on the MetData-1 dataset, and the classification accuracy can reach the maximum value; that is, the average difference is merely 0.18. This finding shows that the proposed architecture can improve the accuracy of the deep learning-based frequency-domain convolutional neural network model without reducing the time performance and expand the frequency-domain representation strategy of the frequency-domain activation function and pooling method.

In recent years, residual learning has shown excellent performance on convolutional neural network (CNN)-based single-image super-resolution (SISR) tasks. However, CNN-based SISR approaches have focused mainly on the design of deep architectures, and the rectified linear units (ReLUs) used in these networks hinder shallow-to-deep information transfer. As a result, these methods are unable to utilize some shallow information, and improving model performance is difficult. To solve the above issues, this paper proposes an image SR reconstruction method based on a generative adversarial network with a residual dense architecture. First, before ReLU activation, the number of feature channels is expanded by a factor of 6~9 using a $1\times 1$ convolutional layer, which improves the utilization of shallow information. Next, the original discriminator is replaced with a relativistic average discriminator, thereby improving the authenticity of the discriminative network. Finally, preactivation features are used to improve the perceptual loss, thus providing stronger monitoring for brightness consistency and texture restoration. Experimental results show that the proposed algorithm improves the utilization of shallow information in a deep network. Structural similarity (SSIM) index evaluations show that the overall utilization of shallow information is increased by 105.52%. In addition, the average runtime is 0.42 sec/frame, nearly 3.6 times faster than those of traditional methods. Moreover, the recovered images have an average natural image quality evaluator value of 3.4 and high perceptual quality, showing that the proposed method is suitable for image reconstruction applications in fields such as agriculture and medicine.

In this article, a novel edge computing system is proposed for image recognition via memristor-based blaze block circuit, which includes a memristive convolutional neural network (MCNN) layer, two single-memristive blaze blocks (SMBBs), four double-memristive blaze blocks (DMBBs), a global Avg-pooling (GAP) layer, and a memristive full connected (MFC) layer. SMBBs and DMBBs mainly utilize the depthwise separable convolution neural network (DwCNN) that can be implemented with a much smaller memristor crossbar (MC). In the backward propagation, we use batch normalization (BN) layers to accelerate the convergence. In the forward propagation, this circuit combines DwCNN layers/CNN layers with nonseparate BN layers, which means that the required number of operational amplifiers is cut by half as long as the greatly reduced power consumption. A diode is added after the rectified linear unit (ReLU) layer to limit the output of the circuit below the threshold voltage Vt of the memristor; thus, the circuit is more stable. Experiments show that the proposed memristor-based circuit achieves an accuracy of 84.38% on the CIFAR-10 data set with advantages in computing resources, calculation time, and power consumption. Experiments also show that, when the number of multistate conductance is 2⁸ and the quantization bit of the data is 8, the circuit can achieve its best balance between power consumption and production cost.

Training of deep neural networks is difficult due to vanishing gradients. Therefore, a pre-training procedure based on restricted Boltzmann machines is suggested to resolve this problem. However, new developments in deep learning aim to resolve the problem with vanishing gradients by using rectifier linear units (ReLU). This study compares the performance of a RBM pre-trained auto-encoder with sigmoid activations to the performance of auto-encoder with ReLU activation. The results showed that the ReLU auto-encoder achieved better reconstruction and saved training time, since it doesn't require pre-training .

Digital frauds get a dramatic increase over the years and lead to considerable losses. Detecting fraudulent attempts is valuable to many industries and especially to the banking and financial sectors. To help in anticipating and accurately identifying whether a transaction is fraudulent, machine learning-based models are the key solution for banking and financial institutions. In this paper, an artificial intelligence-based model was built using deep learning and was trained using stochastic gradient descent and feedforward neural networks. The dropout regularization has been utilized to enhance the generalization capabilities of the digital transaction classification model. Different activation functions were used and explored such as the max-out, the hyperbolic tangent, the rectifier linear unit, and the exponential rectifier linear unit. The impact of the learning rate on the model performance was analyzed. For the evaluation of the model, we did use of different metrics such as the accuracy, the precision, and the recall. The obtained results are promising, and the developed model can be used effectively to defend the banking sector against digital frauds.

Penelitian ini bertujuan untuk mengetahui pengaruh National Instruments Multisim (NI-Multisim) Software terhadap penguasaan rangkaian penyearah gelombang ditinjau dari kemampuan visual spasial (KVS) mahasiswa. Penelitian ini menggunakan pra-eksperimental design. Sampel penelitian adalah mahasiswa calon guru fisika yang mengambil mata kuliah Elektronika Dasar I yang dipilih melalui teknik sampling jenuh. Alat pengumpul data berupa instrumen tes kemampuan visual spasial dan instrumen tes rangkaian penyearah gelombang. Uji korelasi, regresi berganda, dan uji independent sample t-test digunakan untuk mengetahui pengaruh kemampuan spasial-visual terhadap hasil belajar rangkaian penyearah gelombang menggunakan NI-Multisim Software. Hasil menunjukkan bahwa: 1) Hasil belajar materi rangkaian penyearah gelombang menggunakan NI-Multisim Software berada pada kategori cukup; 2) Kemampuan visual spasial mahasiswa pada indikator KVS-intepretasi berkategori sangat tinggi, KVS-pola seri berkategori tinggi, KVS-refleksi berkategori tinggi, KVS-rotasi berkategori tinggi, KVS-proyeksi berkategori tinggi, dan KVS-menggambar berkategori sedang; 3) Kemampuan visul spasial mahasiswa memiliki korelasi yang positif dan signifikan terhadap hasil belajar mahasiswa dan memiliki kontribusi pengaruh sebesar 76,20% terhadap hasil belajar; 4) Hasil belajar mahasiswa yang memiliki KVS tinggi, lebih baik daripada rerata hasil belajar mahasiswa yang memiliki KVS rendah dalam mempelajari materi rangakaian penyearah gelombang menggunakan NI-Multisim Software.

Deep convolutional neural networks (DCNNs) are one of the most promising deep learning techniques and have been recognized as the dominant approach for almost all recognition and detection tasks. The computation of DCNNs is memory intensive due to large feature maps and neuron connections, and the performance highly depends on the capability of hardware resources. With the recent trend of wearable devices and Internet of Things, it becomes desirable to integrate the DCNNs onto embedded and portable devices that require low power and energy consumptions and small hardware footprints. Recently stochastic computing (SC)-DCNN demonstrated that SC as a low-cost substitute to binary-based computing radically simplifies the hardware implementation of arithmetic units and has the potential to satisfy the stringent power requirements in embedded devices. In SC, many arithmetic operations that are resource-consuming in binary designs can be implemented with very simple hardware logic, alleviating the extensive computational complexity. It offers a colossal design space for integration and optimization due to its reduced area and soft error resiliency. In this paper, we present HEIF, a highly efficient SC-based inference framework of the large-scale DCNNs, with broad applications including (but not limited to) LeNet-5 and AlexNet , that achieves high energy efficiency and low area/hardware cost. Compared to SC-DCNN, HEIF features: 1) the first (to the best of our knowledge) SC-based rectified linear unit activation function to catch up with the recent advances in software models and mitigate degradation in application-level accuracy; 2) the redesigned approximate parallel counter and optimized stochastic multiplication using transmission gates and inverse mirror adders; and 3) the new optimization of weight storage using clustering. Most importantly, to achieve maximum energy efficiency while maintaining acceptable accuracy, HEIF considers holistic optimizations on cascade connection of function blocks in DCNN, pipelining technique, and bit-stream length reduction. Experimental results show that in large-scale applications HEIF outperforms previous SC-DCNN by the throughput of $4.1\times $ , by area efficiency of up to $6.5\times $ , and achieves up to ${5.6\times }$ energy improvement.

Pedestrian detection and tracking remains a highlight research topic due to its paramount importance in the fields of video surveillance, human-machine interaction, and tracking analysis. At present time, pedestrian detection is still an open problem because of many challenges of image representation in the outdoor and indoor scenes. In recent years, deep learning, in particular Convolutional Neural Networks (CNNs) became the state-of-the-art in terms of accuracy in many computer vision tasks. The unsupervised learning of CNNs is still an open issue. In this paper, we study a matter of feature extraction using a special activation function. Most of CNNs share the same architecture, when each convolutional layer is followed by a nonlinear activation layer. The activation function Rectified Linear Unit (ReLU) is the most widely used as a fast alternative to sigmoid function. We propose a bounded randomized leaky ReLU working in such manner that the angle of linear part with the highest input values is tuned during learning stage, and this linear part can be directed not only upward but also downward using a variable bias for its starting point. The bounded randomized leaky ReLU was tested on Caltech Pedestrian Dataset with promising results.

Neural networks with REctified Linear Unit (ReLU) activation functions (a.k.a. ReLU networks) have achieved great empirical success in various domains. Nonetheless, existing results for learning ReLU networks either pose assumptions on the underlying data distribution being e.g. Gaussian, or require the network size and/or training size to be sufficiently large. In this context, the problem of learning a two-layer ReLU network is approached in a binary classification setting, where the data are linearly separable and a hinge loss criterion is adopted. Leveraging the power of random noise perturbation, this paper presents a novel stochastic gradient descent (SGD) algorithm, which can \emph{provably} train any single-hidden-layer ReLU network to attain global optimality, despite the presence of infinitely many bad local minima, maxima, and saddle points in general. This result is the first of its kind, requiring no assumptions on the data distribution, training/network size, or initialization. Convergence of the resultant iterative algorithm to a global minimum is analyzed by establishing both an upper bound and a lower bound on the number of non-zero updates to be performed. Moreover, generalization guarantees are developed for ReLU networks trained with the novel SGD leveraging classic compression bounds. These guarantees highlight a key difference (at least in the worst case) between reliably learning a ReLU network as well as a leaky ReLU network in terms of sample complexity. Numerical tests using both synthetic data and real images validate the effectiveness of the algorithm and the practical merits of the theory., 23 pages, 7 figures, work in progress

Packet classification in software-defined network has become more important with the rapid growth of Internet. Existing approaches focused on the data structure algorithms to classify the packets. But the existing algorithms lead to the problem of time budget and fails to accommodate large rule sets. Thus the key task is to design an algorithm for packet classification that inflicts process overhead, and the algorithm should handle large databases of classification rule. These challenging issues are achieved by proposing rectified linear unit deep neural network. The aim of this work is twofold. First various hyper-parameter values are analyzed in order to examine how they affect the packet classification performance of deep neural network; and their performance is compared with that of popular methods, e.g., K-nearest neighbor and support vector machines. The open-source TensorFlow deep learning framework with the support of NVidia GPU units is used to carry out this work, thus allowing a large number of rules to predict the exact flow. The result shows that the proposed method performs well, and hence, this model is more suitable for large classification rules.

Most state-of-the-art methods in recognizing rotated, scaled, and translated (RST) palmprint image have difficulty to achieve the tradeoff between accuracy and robustness. Inspired by vision processing mechanic, we propose a double biologically inspired transform (DBIT) network. DBIT involves two stages, and each stage has several layers. In the first stage, we first build a convolution layer to highlight the edges at all direction and interval. Then, we construct another two convolution layers to measure local spatial frequency (LSF), which can improve the selectivity of feature maps. Since there is no negative spike in biological neurons, a rectified linear unit and sum pooling are employed to produce a neuron spike. Then, each spike is normalized by their original orientation edge maps, and finally, all spikes are combined to a feature vector. In the second stage, two convolution layers including orientation edge detection and single LSF followed by several subsequent operations are performed once again, which transforms image shift to an invariant spike on final feature maps. Experiments on PolyU multispectral, PolyU II, CASIA, IITD, and COEP database illustrate the effectiveness and robustness of the proposed approach. Remark progresses in feature invariance and selectivity are achieved by the proposed method. Comparing with several existing methods, our approach can achieve better recognition performance under rotation, scaling, translation, and even noise interference.

A convolutional neural network (CNN) has recently demonstrated its outstanding capability for the classification of hyperspectral images (HSIs). Typical CNN-based methods usually adopt image patches as inputs to the network. However, a fixed-size image patch in HSI with complex spatial contexts may contain multiple ground objects of different classes, which will deteriorate the classification performance of the CNN. In addition, traditional convolutional layers adopted in the CNN have a huge amount of parameters needed to be tuned, which will cause high computational cost. To address the above-mentioned issues, a novel squeeze multibias network (SMBN) is proposed for HSI classification. Specifically, the proposed SMBN first introduces the multibias module (MBM), which incorporates multibias into the rectified linear unit layers. The MBM can decouple the feature maps of input patches into multiple response maps (corresponding to different ground objects) and adaptively select the meaningful maps for classification. Furthermore, the proposed SMBN replaces the traditional convolutional layer with a squeeze convolution module, which can greatly reduce the number of parameters in the network, thus saving the running time, while still maintaining high classification accuracy. Experimental results on three real HSIs demonstrate the superiority of the proposed SMBN method over several state-of-the-art classification approaches.

Automatic binary classification of brain magnetic resonance (MR) images has made remarkable progress in the past decade. In comparison, a few pieces of work has been reported on multiclass classification of brain MR images. However, there exist enough scopes for improved automation and accuracy. Most of the existing schemes follow the multi-stage pipeline structure of conventional machine learning framework, where the features are designed manually or hand-crafted. In recent years, deep learning models have attracted great interest from researchers for analyzing medical images that eliminate the traditional steps of machine learning. In this paper, we present an automated method based on deep extreme learning machine (ELM) also termed as multilayer ELM (ML-ELM) for multiclass classification of the pathological brain. ML-ELM is a multilayer architecture stacked with ELM based autoencoders. The effectiveness of leaky rectified linear unit (LReLU) activation function is investigated with ML-ELM. Extensive simulations on a multiclass brain MR image dataset indicate that the ML-ELM with LReLU activation (ML-ELM+LReLU) achieves higher performance with faster training speed compared to its counterparts as well as state-of-the-art schemes. The basic purpose of employing ML-ELM+LReLU algorithm is to eliminate the need for hand-crafted feature extraction and to develop a more stable and generalized system for multiclass brain MR image classification.

During the last few years, object recognition has received a big of interest in an attempt to make use of the large scale image datasets. Object recognition allows understanding image based on the objects that it contains. The success of the deep learning based methods in recognizing objects has encouraged recent works to follow this approach. In this paper, we propose a Boosted Convolutional Neural Network approach for object recognition. Our approach uses a very deep convolutional neural network reinforced by adding Boosted Blocks. These Boosted Blocks consist of a succession of convolutional layers boosted by using a Multi-Bias Nonlinear Activation function, that enriches the expressive power of the network, as well as a Concatenated Rectified Linear Unit function which ensures the preservation of all the information after the convolutional layer. Besides, inspired by the visual system of the human brain, our Boosted Convolutional Layer is designed following a recurrent structure. In addition, rather than using the classical max-pooling, our Boosted Convolutional Neural Network is improved by applying Generalizing pooling which allows pooling to adapt to complex and variable patterns. Furthermore, the Spatial Pyramid Pooling after the last Boosted Block has been conducted after the Boosted Block in order to remove the fixed-size constraint of input image. Our approach is evaluated on four different object recognition benchmarks: Pascal VOC 2007, Pascal VOC 2012, CIFAR-10 and the larger dataset ILSVRC-2012. These datasets are widely used by the recent object recognition methods. The experimental results validate the efficiency of our method in comparison with other methods from the literature.

Among deep learning methods, convolutional neural networks (CNNs) are able to extract features automatically and have increasingly been used in intelligent fault diagnosis studies. However, studies seldomly concentrate on the weakness associated with a highly imbalanced distribution of fault types due to different failure rates and when multiple faults are easily confused with single faults. To solve these problems, this paper developed a stochastic discrete-time series deep convolutional neural network (SDCNN) method based on random oversampling along with a progressive method with multiple SDCNNs to improve the diagnosis performance. To assess the developed method, datasets from three avionics 24-pulse auto-transformer rectifier units (ATRUs), which are secondary electric power supplies in aircraft, were analyzed and compared with other CNN methods.

Deep Convolutional Neural Networks (CNNs) have been extremely effective in many areas of computer vision, particularly in the domain of facial recognition where many CNN models have delivered a near perfect performance on long-standing benchmarks. A major challenge being faced by CNN architectures today is their need for a large amount of memory and processing power to deliver their exceptional performance. This hampers their deployment on less powerful devices like smart-phones whose ubiquity is unmatched. This paper critically analyzes a CNN based network architecture wherein convolutions are approximated by XNOR operations and bit-counts which are primarily binary operations hence making the network much faster and more memory efficient than standard CNN architectures. The aim of this paper is to assess the performance of the aforementioned architecture for the task of facial recognition with Minimum Barrier Salient Object Detection (MBD) as a preprocessing step and to examine the affect self-normalizing properties induced by the Scaled Exponential Linear Unit (SELU) activation function on the convergence of the network and benchmark it against the convergence performance of the same network incorporating the widely used Rectified Linear Unit (RELU) as its activation function. Two widely known datasets, Faces95 and Faces96, pre-processed using Minimum Barrier Detection (MBD) for saliency detection were used benchmarking the performance of the proposed networks. It was observed that the network using SELU converged faster than RELU for the Faces95 dataset and both converged at similar rates for Faces96 dataset. The network with SELU gave better accuracy than RELU for both the datasets.

This paper proposes an efficient and noise-insensitive end-to-end lightweight deep learning method. The method synthesizes the characteristics of a frequency domain transform and a deep convolutional neural network. The former can extract multiscale information in vibration signal processing and the latter has a good classification performance, data-driven, and high transfer-learning ability. A vibration signal is decomposed into a pyramidal wavelet packet, and each sub-band coefficient is used as an input of a channel in the deep network. A deep residual convolutional network based on a separable convolution and concatenated rectified linear unit (CReLU) lightweight convolution technology is used for fault diagnosis. The proposed algorithm is compared with related deep learning algorithms using two bearing datasets produced by Case Western Reserve University (CWRU) and the Center for Intelligent Maintenance Systems (IMS), University of Cincinnati. Compared with the existing algorithms, the experimental results show that the comprehensive performance of the algorithm proposed in this paper is “small, light, and fast,” and satisfactory diagnostic results are obtained in the fault diagnosis of rotating machinery.

Deep neural networks (DNNs) trained in a supervised way suffer from two known problems. First, the minima of the objective function used in learning correspond to data points (also known as rubbish examples or fooling images) that lack semantic similarity with the training data. Second, a clean input can be changed by a small, and often imperceptible for human vision, perturbation so that the resulting deformed input is misclassified by the network. These findings emphasize the differences between the ways DNNs and humans classify patterns and raise a question of designing learning algorithms that more accurately mimic human perception compared to the existing methods. Our article examines these questions within the framework of dense associative memory (DAM) models. These models are defined by the energy function, with higher-order (higher than quadratic) interactions between the neurons. We show that in the limit when the power of the interaction vertex in the energy function is sufficiently large, these models have the following three properties. First, the minima of the objective function are free from rubbish images, so that each minimum is a semantically meaningful pattern. Second, artificial patterns poised precisely at the decision boundary look ambiguous to human subjects and share aspects of both classes that are separated by that decision boundary. Third, adversarial images constructed by models with small power of the interaction vertex, which are equivalent to DNN with rectified linear units, fail to transfer to and fool the models with higher-order interactions. This opens up the possibility of using higher-order models for detecting and stopping malicious adversarial attacks. The results we present suggest that DAMs with higher-order energy functions are more robust to adversarial and rubbish inputs than DNNs with rectified linear units.

There are a number of problems in science and technology that demand separating useful information from certain content. For many of those problems, standard techniques, as signal processing technique, shape recognition, system control theory, artificial intelligence etc., have shown as inadequate. Neural networks are a way to solve these problems in a way they are solved in human brain. Same as the human brain, neural networks are able to learn from given data, and afterwards, when they meet the same or similar data they may give the same or approximate result. There are several types of transfer functions: sigmoid, logistic sigmoid, linear, semilinear, threshold, Gauss' function. Figure 1 shows the graph for one of most used transfer functions: Fig. 1. Logistic sigmoid function

Network science possesses an unreplaceable status in solving social and scientific problems. This study focuses on investigating machine learning based graph mining of large-scale network and optimization by using the California road network dataset from Stanford as the large-scale social network. After building different neural networks with various hyperparameters and selected learning activation functions, accuracy results were compared and conclusions include: 1) Changing the activation functions does not have much effect on accuracy compared to neural network structures, 2) Changing the learning rate does not have much effect either and 3) exponential linear unit (ELU) is sensitive to the change of hidden layer size and kernel compared to rectifier linear unit (ReLU) and hyperbolic tangent (Tanh), causing decrement of accuracy after growth of hidden layer size brings overfitting. These conclusions were drawn based on numeric experiments where the final accuracy rate kept the average of every ten trials. However, given this specific dataset, the accuracy of all the models remain high so that varying neural network parameters or structures does not present much distinctive divergence.

Our proposal consists of developing two novel activation functions in time series anomaly detection, they have the capability to reduce the validation loss. The approach is based on a current activation function in Deep Learning, a very intensive field studied over time, in order to find the most suitable activation in a neural network. In order to achieve this purpose, we used an LSTM (Long Short-Term Memory) Autoencoder architecture, using these two novel functions to see the network’s behavior through introducing them. The key point in our proposal is given by the learnable parameter, assuring more flexibility within the network in weights’ updates, in fact, this property being more powerful than a predefined parameter that will bring a constraint due to its limit. We tested our proposal in comparison to other popular functions such as ReLU (Linear Rectifier Unit), hyperbolic tangent (tanh), Talu activation function. Also, the novelty of this paper consists of taking into consideration of piecewise behavior of an activation function in order to increase the performance of a neural network in Deep Learning.

Until recently, traditional machine learning techniques (TMLTs) such as multilayer perceptrons (MLPs) and support vector machines (SVMs) have been used successfully for churn prediction, but with significant efforts expended on the configuration of the training parameters. The selection of the right training parameters for supervised learning is almost always experimentally determined in an ad hoc manner. Deep neural networks (DNNs) have shown significant predictive strength over TMLTs when used for churn predictions. However, the more complex architecture of DNNs and their capacity to process huge amounts of non-linear input data demand more time and effort to configure the training hyperparameters for DNNs during churn modeling. This makes the process more challenging for inexperienced machine learning practitioners and researchers. So far, limited research has been done to establish the effects of different hyperparameters on the performance of DNNs during churn prediction. There is a lack of empirically derived heuristic knowledge to guide the selection of hyperparameters when DNNs are used for churn modeling. This paper presents an experimental analysis of the effects of different hyperparameters when DNNs are used for churn prediction in the banking sector. The results from three experiments revealed that the deep neural network (DNN) model performed better than the MLP when a rectifier function was used for activation in the hidden layers and a sigmoid function was used in the output layer. The performance of the DNN was better when the batch size was smaller than the size of the test set data, while the RemsProp training algorithm had better accuracy when compared with the stochastic gradient descent (SGD), Adam, AdaGrad, Adadelta, and AdaMax algorithms. The study provides heuristic knowledge that could guide researchers and practitioners in machine learning-based churn prediction from the tabular data for customer relationship management in the banking sector when DNNs are used.

International audience; Comprehensive two−dimensional Gas Chromatography with Vacuum Ultraviolet detection (GC×GC/VUV) results in sizable data for which noise and baseline drift ought to be corrected. As GC×GC/VUV signal is acquired from multiple channels, these pre−processing steps have to be applied to data from all channels while being robust and rather fast with respect to significant size of the GC×GC/VUV data. In this study, we describe advanced GC×GC/VUV data pre−processing techniques for noise and baseline correction that are not available in commercial softwares. Noise reduction was performed on both the spectral and the time dimension. For baseline correction, a morphological approach based on iterated convolutions and rectifier operations is proposed. On the spectral dimension, much less noisy and reliable spectra are obtained. From a quantitative point of view, mentioned pre−processing steps significantly improve signal to noise ratio for analyte detection and hence improve their limit of detection (circa 6 times in this study). These pre−processing methods were integrated into plug im! platform (https://www.plugim.fr/).

Object recognition research has made notable steps since the appearance of convolutional neural networks, and many activation functions have been proposed to enhance the classification performance of these networks. Maxout networks have achieved great success in many computer vision tasks, but there is limited information on whether an increase in the number of trainable parameters can increase the performance in Leaky Rectified Linear Unit (LReLU) networks compared to maxout networks. Our experiments compare LReLU, rectified linear unit, scaled exponential linear unit, and hyperbolic tangent to four maxout variants. We evaluate ReLU and LReLU with 2x, 3x and 6x the number of filters in each convolutional layer. We also evaluate ReLU, LReLU and maxout networks with approximately the same number of trainable parameters. Under equal conditions, we found that on average, across all datasets, LReLU performs better than any of the evaluated activation functions.

Background and aim: Diagnosis and treatment of female infertility conditions would help future reproductive planning. Although current deep learning frameworks are able to classify and separately count all types at high accuracy, these solutions suffer from a misclassification error and a high computation complexity due to a positive bias effect and an internal covariate shift. The objective of this paper is to increase the classification accuracy of OFs and to reduce the computational costs of classification via deep learning (DL). Methodology: our framework for follicle classification and counting (FCaC) uses filter-based segmentation. A new method is also proposed to accelerate learning and to normalize the input layer by adjusting and scaling the activations. Our method uses a modified activation function (MAF)- displaced rectifier linear unit (DReLU) and batch normalization (BN) in Feature Extraction and Classification. Therefore, faster and more stable training is achieved by modifying input distribution of an activation function (AF). Results: The proposed system was able to obtain a mean classification accuracy of 97.614%, which is 2.264% more accurate classification than the state-of-the-art. Furthermore, the model processes a single WSI 30% faster (in 10.23 seconds compared to 14.646 seconds processing time of the existing solutions). Conclusion: The proposed system focuses on processing histology images with an accurate classification. It is also faster, has an accelerated convergence and enhanced learning thanks to BN and the EAF. We considered a positive bias effect and internal covariate shift as the main aspects to improve the classification performance.

Sales forecast is a key issue in pharmacy wholesales and has a direct effect in inventory cost, it also represents a difficult problem, affected by factors like promotions, price changes and seasonal preferences. Traditional sales forecast techniques use historical sales data and recursive formulas, which limit their accuracy. More recently learning-based methods have expanded data features capture capabilities. This paper presents a deep architecture which explores a few years of weekly sales data and learns to makes assertive predictions. The system is assembled with ReLU neurons, whose learning behavior makes possible the effective training of sparse coded deep sub-nets. The developed learning algorithm uses two learning stages where the first produces sparse representation of the studied time series and the second harvest one week ahead predictions using this sparse data. In both cases the reward system is future-focused and favors search for future capacities. To achieving successful network training we develop an algorithms that deals with exploding gradient problem, typical of ReLU networks. The fully assembled predictor automatically learn features from structured data and produces inventories with improved dollar cost. The system has been tested in real time with real data.

Scene text recognition is the task of recognizing character sequences in images of natural scenes. The considerable diversity in the appearance of text in a scene image and potentially highly complex backgrounds make text recognition challenging. Previous approaches employ character sequence generators to analyze text regions and, subsequently, compare the candidate character sequences against a language model. In this work, we propose a bimodal framework that simultaneously utilizes visual and linguistic information to enhance recognition performance. Our linguistically aware learning (LAL) method effectively learns visual embeddings using a rectifier, encoder, and attention decoder approach, and linguistic embeddings, using a deep next-character prediction model. We present an innovative way of combining these two embeddings effectively. Our experiments on eight standard benchmarks show that our method outperforms previous methods by large margins, particularly on rotated, foreshortened, and curved text. We show that the bimodal approach has a statistically significant impact. We also contribute a new dataset, and show robust performance when LAL is combined with a text detector in a pipelined text spotting framework.

An ultra-low power integrate-and-fire neuron array transceiver with a multi-modal neuron architecture is presented. The design features an array of 16 × 16 charge-mode mixed-signal neurons that can be configured to implement a variety of activation functions, including step, sigmoid and Rectified Linear Unit (ReLU), through reconfiguration of clocking waveforms through partial reset in charge accumulation and additive stochastic noise by Linear Feedback Shift Register (LFSR) coupling. The neuron outputs spike-based sparse synchronous events, which are either binary (event/no event) or ternary (positive/negative/no events). The reconfigurable energy-efficient design makes this architecture suitable for deep learning and neuromorphic applications like Restricted Boltzmann Machines, Convolutional Neural Networks and general event-driven computing. The 1.796 mm2 chip fabricated in 130nm CMOS technology consumes 140.6 μW from a 1.8V supply at 92.5 MSpikes/s achieving an energy efficiency Figure-of-Merit (FoM) of 1.52 pJ/Spike. A CNN architecture implemented on the chip using sigmoid and ReLU activation achieves MNIST prediction accuracy of 94.8% and 96.9%.

In the artificial intelligence, machine learning and neural network field, many practical use models have been proposed. However, these models are based on a digital Von Neumann model’s computer. There are minority studies in the field of analog learning neural networks. Early analog hardware neural network models were configured with the operational amplifier and the resistance element. It is difficult to change the value of solid resistance for the learning process. In the present paper, proposed is a learning neural network using electronic alternating current (AC) circuits including a voltage-frequency (V-F) converter. These circuits are composed of an amplifier, additional circuit, inverter, subtract circuit, rectifier circuit and V-F converter. The input voltage value was described as the input signal and the input frequency considered the connecting weights. It is easy to change the connecting weights with a V-F converter. Finally, the input frequency converges to a constant value after only several learning process. The learning count time is extremely small. The learning time is quite fast in this AC transmission circuit. The model works using pure analog electronic circuits. The learning time is quite short compare to with a digital process computer.

For highly nonlinear systems, using deep learning models to capture complex dynamics is a promising feature for advanced control applications. Recently it has been shown that a particular class of deep learning models can be exactly recast in a mixed-integer linear programming formulation. Recasting a deep learning model as a set of piecewise linear functions enables the incorporation of advanced predictive models in model-based control strategies such as model predictive control. To alleviate the computational burden of solving the piecewise linear optimization problem online, multiparametric programming is utilized to obtain the full, offline, explicit solution of the optimal control problem. In this work, a strategy is presented for the integration of deep learning models, specifically neural networks with rectified linear units, and explicit model predictive control. The proposed strategy is demonstrated on the advanced control of a benchmark chemical process involving multiple reactors, a flash separator, and a recycle stream. The positive results showcase the relevance and strength of the proposed methodology.