Search

Bottom-up-fabricated crossbars promise superior circuit density and 3-D integrability compared with the traditional CMOS-based implementations. However, their inherent stochasticity presents difficulties in building complex circuits from components that demand precise patterning and high registration accuracies. With fewer terminals than active devices, passive components offer higher device densities and registration tolerances, making them amenable to bottom-up synthesized nanocrossbars. Motivated by this preference for passivity, we explore, in this article, neuromorphic classifiers based on passive neurons and passive synapses. We demonstrate via SPICE simulations how a shallow network of the diode-resistor-based passive rectifier neurons and resistive voltage summers, despite its inherent inability to buffer, amplify, and negate signals, can recognize MNIST digits with 95.4% accuracy. We introduce weight-to-conductance mappings that enable negative weights to be implemented in hardware without excessive memory overheads. The influences of soft and hard defects on the classification performance are evaluated, and the methods to boost fault-tolerance are proposed. The first-order evaluation of the area, speed, and power consumption of the passive multilayer perceptron classifiers is undertaken, and the results are compared with a benchmark study in neuromorphic hardware.

We analyze the learning problem of one-hidden-layer nonoverlapping convolutional neural networks with the rectified linear unit (ReLU) activation function from the perspective of model estimation. The training outputs are assumed to be generated by the neural network with the unknown ground-truth parameters plus some additive noise, and the objective is to estimate the model parameters by minimizing a nonconvex squared loss function of the training data. Assuming that the training set contains a finite number of samples generated from the Gaussian distribution, we prove that the accelerated gradient descent (GD) algorithm with a proper initialization converges to the ground-truth parameters (up to the noise level) with a linear rate even though the learning problem is nonconvex. Moreover, the convergence rate is proved to be faster than the vanilla GD. The initialization can be achieved by the existing tensor initialization method. In contrast to the existing works that assume an infinite number of samples, we theoretically establish the sample complexity of the required number of training samples. Although the neural network considered here is not deep, this is the first work to show that accelerated GD algorithms can find the global optimizer of the nonconvex learning problem of neural networks. This is also the first work that characterizes the sample complexity of gradient-based methods in learning convolutional neural networks with the nonsmooth ReLU activation function. This work also provides the tightest bound so far of the estimation error with respect to the output noise.

This article focuses on a problem important to automatic machine learning: the automatic processing of a nonpreprocessed time series. The convolutional neural network (CNN) is one of the most popular neural network (NN) algorithms for pattern recognition. Seasonal time series with trends are the most common data sets used in forecasting. Both the convolutional layer and the pooling layer of a CNN can be used to extract important features and patterns that reflect the seasonality, trends, and time lag correlation coefficients in the data. The ability to identify such features and patterns makes CNN a good candidate algorithm for analyzing seasonal time-series data with trends. This article reports our experimental findings using a fully connected NN (FNN), a nonpooling CNN (NPCNN), and a CNN to study both simulated and real time-series data with seasonality and trends. We found that convolutional layers tend to improve the performance, while pooling layers tend to introduce too many negative effects. Therefore, we recommend using an NPCNN when processing seasonal time-series data with trends. Moreover, we suggest using the Adam optimizer and selecting either a rectified linear unit (ReLU) function or a linear activation function. Using an NN to analyze seasonal time series with trends has become popular in the NN community. This article provides an approach for building a network that fits time-series data with seasonality and trends automatically.

Previous literature on unsupervised learning focused on designing structural priors with the aim of learning meaningful features. However, this was done without considering the description length of the learned representations which is a direct and unbiased measure of the model complexity. In this paper, first we introduce the $\varphi$ metric that evaluates unsupervised models based on their reconstruction accuracy and the degree of compression of their internal representations. We then present and define two activation functions (Identity, ReLU) as base of reference and three sparse activation functions (top-k absolutes, Extrema-Pool indices, Extrema) as candidate structures that minimize the previously defined $\varphi$. We lastly present Sparsely Activated Networks (SANs) that consist of kernels with shared weights that, during encoding, are convolved with the input and then passed through a sparse activation function. During decoding, the same weights are convolved with the sparse activation map and subsequently the partial reconstructions from each weight are summed to reconstruct the input. We compare SANs using the five previously defined activation functions on a variety of datasets (Physionet, UCI-epilepsy, MNIST, FMNIST) and show that models that are selected using $\varphi$ have small description representation length and consist of interpretable kernels., Comment: 10 pages, 5 figures, 4 algorithms, 4 tables, submission to IEEE Transactions on Neural Networks and Learning Systems

Training neural networks is recently a hot topic in machine learning due to its great success in many applications. Since the neural networks’ training usually involves a highly nonconvex optimization problem, it is difficult to design optimization algorithms with perfect convergence guarantees to derive a neural network estimator of high quality. In this article, we borrow the well-known random sketching strategy from kernel methods to transform the training of shallow rectified linear unit (ReLU) nets into a linear least-squares problem. Using the localized approximation property of shallow ReLU nets and a recently developed dimensionality-leveraging scheme, we succeed in equipping shallow ReLU nets with a specific random sketching scheme. The efficiency of the suggested random sketching strategy is guaranteed by theoretical analysis and also verified via a series of numerical experiments. Theoretically, we show that the proposed random sketching is almost optimal in terms of both approximation capability and learning performance. This implies that random sketching does not degenerate the performance of shallow ReLU nets. Numerically, we show that random sketching can significantly reduce the computational burden of numerous backpropagation (BP) algorithms while maintaining their learning performance.

This brief presents a new guaranteed $H_{\infty } $ performance state estimation criterion for delayed static neural networks. To facilitate the use of the slope information about activation function, the estimation error of activation function is separated into two parts for the first time. Then, a novel Lyapunov-Krasovskii functional (LKF) is constructed, which has fully captured the slope information of the activation. Based on the new LKF, a less conservative design criterion of estimator is derived to ensure the asymptotic stability of estimation error system with $H_{\infty } $ performance. The desired estimator gain matrices and the performance index are obtained by solving a convex optimization problem. The simulation results show that the proposed method has much better performance than the most recent results.

We present a theoretical analysis of singular points of artificial deep neural networks, resulting in providing deep neural network models having no critical points introduced by a hierarchical structure. It is considered that such deep neural network models have good nature for gradient-based optimization. First, we show that there exist a large number of critical points introduced by a hierarchical structure in deep neural networks as straight lines, depending on the number of hidden layers and the number of hidden neurons. Second, we derive a sufficient condition for deep neural networks having no critical points introduced by a hierarchical structure, which can be applied to general deep neural networks. It is also shown that the existence of critical points introduced by a hierarchical structure is determined by the rank and the regularity of weight matrices for a specific class of deep neural networks. Finally, two kinds of implementation methods of the sufficient conditions to have no critical points are provided. One is a learning algorithm that can avoid critical points introduced by the hierarchical structure during learning (called avoidant learning algorithm). The other is a neural network that does not have some critical points introduced by the hierarchical structure as an inherent property (called avoidant neural network).

Three-phase grid-connected converters are widely used in renewable and electric power system applications. Traditionally, grid-connected converters are controlled with standard decoupled d-q vector control mechanisms. However, recent studies indicate that such mechanisms show limitations in their applicability to dynamic systems. This paper investigates how to mitigate such restrictions using a neural network to control a grid-connected rectifier/inverter. The neural network implements a dynamic programming algorithm and is trained by using back-propagation through time. To enhance performance and stability under disturbance, additional strategies are adopted, including the use of integrals of error signals to the network inputs and the introduction of grid disturbance voltage to the outputs of a well-trained network. The performance of the neural-network controller is studied under typical vector control conditions and compared against conventional vector control methods, which demonstrates that the neural vector control strategy proposed in this paper is effective. Even in dynamic and power converter switching environments, the neural vector controller shows strong ability to trace rapidly changing reference commands, tolerate system disturbances, and satisfy control requirements for a faulted power system.