Search

Explainable Artificial Intelligence (XAI) is targeted at understanding how models perform feature selection and derive their classification decisions. This paper explores post-hoc explanations for deep neural networks in the audio domain. Notably, we present a novel Open Source audio dataset consisting of 30,000 audio samples of English spoken digits which we use for classification tasks on spoken digits and speakers' biological sex. We use the popular XAI technique Layer-wise Relevance Propagation LRP to identify relevant features for two neural network architectures that process either waveform or spectrogram representations of the data. Based on the relevance scores obtained from LRP, hypotheses about the neural networks' feature selection are derived and subsequently tested through systematic manipulations of the input data. Further, we take a step beyond visual explanations and introduce audible heatmaps. We demonstrate the superior interpretability of audible explanations over visual ones in a human user study. • We present a novel audio dataset consisting of 30,000 audio samples of spoken digits. • We use LRP to explain predictions of two different models in the audio domain. • We confirm hypotheses about the neural networks' use of features from explanations. • We present audible explanations and demonstrate their superior interpretability. [ABSTRACT FROM AUTHOR]

For a class of nontriangular nonlinear systems in presence of unknown disturbances, we propose a predictor-based neural dynamic surface control (PNDSC) strategy in this paper. This nontriangular system is transformed via the mean value theorem, and a predictor is then constructed. To avoid an algebraic loop problem, partial state vectors are employed as input signals of neural networks (NNs) for approximating unknown dynamics, and compensation items are designed to compensate for approximation errors from NNs. Different from the traditional NDSC, the PNDSC in this paper utilizes prediction errors to update learning parameters for improving NNs’ learning behaviors with overlarge adaptive gains. On the basis of improved NNs’ approximation behaviors, a predictor-based NNs disturbance observer (PNNDO) is constructed for compensation for external disturbances and approximation errors from NNs. Furthermore, with predictors, a normalization method of weights is developed to reduce the number of online learning parameters. On the basis of the aforementioned result, measurement noises are taken into account in our predictor-based neural control strategy. We employ predictor states, rather than measurement information paralyzed by noises, in design of our control strategy. This reduces high-frequency oscillations in control input. A Lyapunov-based stability analysis shows that all signals are ultimately bounded in the closed-loop system. Finally, the effectiveness of the proposed control strategy is verified by a numerical example and a permanent magnet brushless DC motor system. [ABSTRACT FROM AUTHOR]

An automatic inductor design process is helpful to reduce the design cycle of radio frequency (RF) integrated circuit (IC). This paper proposed an efficient synthesis-analysis machine (SAM) for on-chip inductor synthesis and modeling, as well as an automatic dataset generation (ADG) topology for the generation of artificial neural networks (ANNs) training dataset. The SAM consists of a synthesis ANN, two analysis ANNs and a proposed self-inspection machine (SIM). For a given design request, the SAM synthesizes a layout first, followed by analyzing the performance of the layout automatically, and finally improves the layout confidence through self-inspection. Compared to the modeling functions of analysis ANN, the synthesis ANN behaves as an inverse model, of which the inputs are desired inductor performances while the outputs are geometrical parameters of the layout. However, multi-value problems might occur in obtaining geometries of inductors, suggesting with evidence from mathematic relationships between geometrical and electrical parameters of an inductor. A multi-valued training dataset will mislead the synthesis ANN, resulting in unqualified layouts. To deal with this issue, a solution space contraction technique (SSCM) is also proposed. Furthermore, the SIM suspends most of the failures by comparing the output of analysis ANN back to the input of synthesis ANN. An electromagnetic simulation tool is used for experiments, and the effectiveness of the proposed SAM is proven by 6,500 inductor samples. [ABSTRACT FROM AUTHOR]

The memristor-based neuromorphic computing system (NCS) with emerging storage and computing integration architecture has drawn extensive attention. Because of the unique nonvolatility and programmability, the memristor is an ideal nano-device to realize neural synapses in VLSI circuit implementation of neural networks. However, in the hardware implementation, the performance of the memristive neural network is always affected by quantization error, writing error, and conductance drift, which seriously hinders its applications in practice. In this paper, a novel weight optimization scheme combining quantization and Bayesian inference is proposed to alleviate this problem. Specifically, the weight deviation in the memristive neural network is transformed into the weight uncertainty in the Bayesian neural network, which can make the network insensitive to unexpected weight changes. A quantization regularization term is designed and utilized during the training process of the Bayesian neural network, reducing the quantization error and improving the robustness of the network. Furthermore, a partial training method is raised to extend the applicability of the proposed scheme in large-scale neural networks. Finally, the experiments on a Multilayer Perceptron and LeNet demonstrate that the proposed weight optimization scheme can significantly enhance the robustness of memristive neural networks. [ABSTRACT FROM AUTHOR]

Purpose: This paper aims to compare stochastic gradient method used for neural network training with global optimizer without use of gradient information, in particular differential evolution. Design/methodology/approach: This contribute shows the application of heuristic optimization algorithms to the training phase of artificial neural network whose aim is to predict renewable power production as function of environmental variables such as solar irradiance and temperature. The training problem is cast as the minimization of a cost function whose degrees of freedom are the parameters of the neural network. A differential evolution algorithm is substituted to the more usual gradient-based minimization procedure, and the comparison of their performances is presented. Findings: The two procedures based on stochastic gradient and differential evolution reach the same results being the gradient based moderately quicker in convergence but with a lower value of reliability, as a significant number of runs do not reach convergence. Research limitations/implications: The approach has been applied to two forecasting problems and, even if results are encouraging, the need for extend the approach to other problems is needed. Practical implications: The new approach could open the training of neural network to more stable and general methods, exploiting the potentialities of parallel computing. Originality/value: To the best of the authors' knowledge, the research presented is fully original for the part regarding the neural network training with differential evolution. [ABSTRACT FROM AUTHOR]

Due to the low complexity in arithmetic unit design, stochastic computing (SC) has attracted considerable interest to implement Artificial Neural Networks (ANNs) for resources-limited applications, because ANNs must usually perform a large number of arithmetic operations. To attain a high computation accuracy in an SC-based ANN, extended stochastic logic is utilized together with standard SC units and thus, a stochastic divider is required to perform the conversion between these logic representations. However, the conventional divider incurs in a large computation latency, so limits an SC implementation for ANNs used in applications needing high performance. Therefore, there is a need to design fast stochastic dividers for SC-based ANNs. Recent works (e.g., a binary searching and triple modular redundancy (BS-TMR) based stochastic divider) are targeting a reduction in computation latency, while keeping the same accuracy compared with the traditional design. However, this divider still requires $N$ iterations to deal with $2^{N}$ -bit stochastic sequences, and thus the latency increases in proportion to the sequence length. In this paper, a decimal searching and TMR (DS-TMR) based stochastic divider is initially proposed to further reduce the computation latency; it only requires two iterations to calculate the quotient, so regardless of the sequence length. Moreover, a trade-off design between accuracy and hardware is also presented. An SC-based Multi-Layer Perceptron (MLP) is then considered to show the effectiveness of the proposed dividers over current designs. Results show that when utilizing the proposed dividers, the MLP achieves the lowest computation latency while keeping the same classification accuracy; although incurring in an area increase, the overhead due to the proposed dividers is low over the entire MLP. When using as combined metric for both hardware design and computation complexity the product of the implementation area, latency, power and number of clock cycles, the proposed designs are also shown to be superior to the SC-based MLPs (at the same level of accuracy) employing other dividers found in the technical literature as well as the commonly used 32-bit floating point implementation. [ABSTRACT FROM AUTHOR]

This paper researches the finite-time master–slave synchronization for implicit hybrid neural networks under event-triggered guaranteed cost control and random deception attacks. By developing Lyapunov–Krasovskii functional and exploiting singular value decomposition technique, the finite-time boundedness of synchronization error closed-loop system is achieved under deception attacks and event-triggered scheme, then the finite-time master–slave synchronization of implicit Markovian jump neural networks is realized. The desired gains of state feedback synchronization guaranteed cost controller are designed by solving a set of linear matrix inequalities. The availability of the given approach is confirmed by an artificial neural network and a numerical example. [ABSTRACT FROM AUTHOR]

This paper presents a novel approach to advancing artificial intelligence (AI) through the development of the Complex Recurrent Spectral Network (ℂ -RSN), an innovative variant of the Recurrent Spectral Network (RSN) model. The ℂ -RSN model introduces localized non-linearity, complex fixed eigenvalues, and a distinct separation of memory and input processing functionalities. These features enable the ℂ -RSN to evolve towards a dynamic, oscillating final state that bear some degree of similarity with biological cognition. The model's ability to classify data through a time-dependent function, and the localization of information processing, is demonstrated by using the MNIST dataset. Remarkably, distinct items supplied as a sequential input yield patterns in time which bear the indirect imprint of the insertion order (and of the separation in time between contiguous insertions). • The C-RSN overcomes some limitations in existing neural network models. • C-RSN introduces localized non-linearity, complex eigenvalues and a separated memory. • The network evolves towards a dynamic, oscillating final state. • The model's efficacy is demonstrated through empirical evaluation. • C-RSN is able to process sequential inputs keeping track of the insertion order. [ABSTRACT FROM AUTHOR]

This paper introduces a 3-layer feed-forward neuronet model, trained by novel beetle antennae search weights-and-structure-determination (BASWASD) algorithm. On the one hand, the beetle antennae search (BAS) algorithm is a memetic meta-heuristic optimization algorithm capable of solving combinatorial optimization problems. On the other hand, neuronets trained by a weights-and-structure-determination (WASD) algorithm are known to resolve the shortcomings of traditional back-propagation neuronets, including slow speed of training and local minimum. Combining the BAS and WASD algorithms, a novel BASWASD algorithm is created for training neuronets, and a multi-input BASWASD neuronet (MI-BASWASDN) model is introduced. Using a power sigmoid activation function and while managing the model fitting and validation, the BASWASD algorithm finds the optimal weights and structure of the MI-BASWASDN. Four financial datasets, taken from the European Central Bank publications, validate and demonstrate the MI-BASWASDN model's outstanding learning and predicting performance. Also included is a comparison of the MI-BASWASDN model to three other well-performing neural network models, as well as a MATLAB kit that is publicly available on GitHub to promote and support this research. [ABSTRACT FROM AUTHOR]

In this paper, a neural network approach is suggested to the product design analysis. Namely, fractional-order neural network models are proposed as more flexible mechanism to study product form design. Since control and stability methods are fundamental in the construction and practical significance of a neural network model, appropriate controllers are designed and practical stability criteria are proposed for the fractional-order model under consideration. The stability and control analysis are based on the Lyapunov function method. Examples are elaborated to demonstrate the established results. The proposed modeling approach and the stability results are also applicable to numerous industrial design tasks. • Fractional-order neural network modeling approach is introduced to the product form design. • Appropriate controllers are utilized. • The practical stability notion is adopted to the introduced model. • Practical stability criteria are established using the Lyapunov function technique. • Examples and discussion are also offered to verify and justify the proposed results. [ABSTRACT FROM AUTHOR]

Quantile regression is widely applied in various fields such as economy, energy, meteorological prediction research in recent years since it does not require distribution assumptions, has relatively loose conditions, and can effectively estimate the uncertainty of time series forecasting. In this paper, a monotone quantile regression neural network (MQRNN) framework is constructed for time series quantile forecasting. The proposed approach takes the monotonicity of quantile into consideration and handles the quantile crossing problem by adding the quantile information into the input structure and using the gradient based point-wise loss function. Aiming at the complex characteristics of time series, such as time-varying and asymmetric heavy-tailed features, a new quantile function is utilized to describe the complete conditional distribution information of data. Under this model framework, non-crossing multiple quantiles can be predicted simultaneously. The proposed approach is implemented based on artificial neural networks, and the constructed model is applied to actual data in different fields. The experimental results demonstrate that the proposed method combined with long short-term memory (LSTM) can provide accurate and reliable multi-quantile prediction, and alleviate the problem of quantile crossing. [ABSTRACT FROM AUTHOR]

In this paper, a new global exponential stability criterion is obtained for a general multidimensional delay difference equation using induction arguments. In the cases that the difference equation is periodic, we prove the existence of a periodic solution by constructing a type of Poincaré map. The results are used to obtain stability criteria for a general discrete-time neural network model with a delay in the leakage terms. As particular cases, we obtain new stability criteria for neural network models recently studied in the literature, in particular for low-order and high-order Hopfield and Bidirectional Associative Memory (BAM). • Global exponential stability criterion for a general delay difference system. • New method to prove global exponential stability of a delay difference system. • Existence of a periodic solution of a general delay difference system. • Exponential stability criteria for neural networks with delay in leakage terms. • Existence of periodic solution for neural networks with delay in the leakage terms. [ABSTRACT FROM AUTHOR]

This paper discusses the chaos prediction of nonlinear systems using various neural networks based on the modified substructure data-driven modeling architecture. In the modeling step, we construct two-coefficient loss functions according to the linear multi-step method to improve the prediction accuracy of neural networks. Then, the predicted response data of the system is given by the forward Euler method and neural networks. Under such architecture, chaos forecasting is carried out on a five-degree-of-freedom duffing oscillator system via the feedforward neural network (FNN), long short-term memory (LSTM) network and LSTM encoder-decoder (LSTM ED). The numerical simulation results show that the model can predict chaotic time series even if a small amount of information and samples are known, and the prediction window is twice that of the observation window. Among these models, LSTM ED exhibits the highest accuracy in both short-term and long-term chaos prediction. Furthermore, the prediction results mainly involve three evaluation indicators: absolute error, mean absolute error, normalized root mean square error. Through error analysis and noise processing, LSTM ED shows superior stability, robustness and extrapolation ability. Its prediction error is about half of FNN and the maximum increase in accuracy is 71.3 %. • Two-coefficient loss functions are constructed in the modeling step. • The forward Euler method is combined with neural networks in the prediction step. • Chaotic time series prediction based on various neural networks is investigated. • The accuracy, robustness and extrapolation ability of models are compared. [ABSTRACT FROM AUTHOR]

In this paper, we propose a general steganographic framework for neural networks to achieve covert communication. Firstly, we design a baseline steganographic method to embed secret data into convolutional layers of a given neural network (cover network) during the process of network training. With a network containing secret data (stego network), matrix multiplication is used to encode the parameters of convolutional layers for data extraction. Based on the baseline steganographic method, we respectively extended multi-source and multi-channel steganographic schemes for different scenarios, in which multiple agents are allowed to embed or extract secret data at the sender or receiver side. The multiple senders or receivers at the same side are ignorant of each other's existence, ensuring the independence and security of communication between each pair of sender and receiver. Experimental results verified the universality of the baseline method and the effectiveness of the extended schemes, including capacity, security, and robustness. [ABSTRACT FROM AUTHOR]

The softmax function is widely used in deep neural networks (DNNs), its hardware performance plays an important role in the training and inference of DNN accelerators. However, due to the complexity of the traditional softmax, the existing hardware architectures are resource-consuming or have low precision. In order to address the challenges, we study a base-2 softmax function in terms of its suitability for neural network training and efficient hardware implementation. Compared to the classical base- $e$ softmax function, the base-2 softmax function is a new softmax function that uses 2 as the exponential base instead of $e$. From the aspects of mathematical derivation and software simulation, we first demonstrate the feasibility and good accuracy of the base-2 softmax function in the application of neural network training. Then, we use the symmetric-mapping lookup table (SM-LUT) method to design a low-complexity architecture but with high precision to implement it. Under TSMC 28nm CMOS technology, an example design of our architecture has the area of $5676 ~\mu m^{2}$ and the power consumption of 13.12 mW for circuit synthesis at the frequency of 3 GHz. Compared with the latest works, our architecture achieves the best performance and efficiency. [ABSTRACT FROM AUTHOR]

Multi-bit-width convolutional neural network (CNN) maintains the balance between network accuracy and hardware efficiency, thus enlightening a promising method for accurate yet energy-efficient edge computing. In this work, we develop state-of-the-art multi-bit-width accelerator for NAS Optimized deep learning neural networks. To efficiently process the multi-bit-width network inferencing, multi-level optimizations have been proposed. Firstly, differential Neural Architecture Search (NAS) method is adopted for the high accuracy multi-bit-width network generation. Secondly, hybrid Booth based multi-bit-width multiply-add-accumulation (MAC) unit is developed for data processing. Thirdly, vector systolic array is proposed for effectively accelerating the matrix multiplications. With vector-style systolic dataflow, both the processing time and logic resources consumption can be reduced when compared with the classical systolic array. Finally, The proposed multi-bit-width CNN acceleration scheme has been practically deployed on FPGA platform of Xilinx ZCU102. Average performance on accelerating the full NAS optimized VGG16 network is 784.2 GOPS, and peek performance of the convolutional layer can reach as high as 871.26 GOPS for INT8, 1676.96 GOPS for INT4, and 2863.29 GOPS for INT2 respectively, which is among the best results in previous CNN accelerator benchmarks. [ABSTRACT FROM AUTHOR]

Automatic modulation recognition (AMR) detects the modulation scheme of the received signals for further signal processing without needing prior information, and provides the essential function when such information is missing. Recent breakthroughs in deep learning (DL) have laid the foundation for developing high-performance DL-AMR approaches for communications systems. Comparing with traditional modulation detection methods, DL-AMR approaches have achieved promising performance including high recognition accuracy and low false alarms due to the strong feature extraction and classification abilities of deep neural networks. Despite the promising potential, DL-AMR approaches also bring concerns to complexity and explainability, which affect the practical deployment in wireless communications systems. This paper aims to present a review of the current DL-AMR research, with a focus on appropriate DL models and benchmark datasets. We further provide comprehensive experiments to compare the state of the art models for single-input-single-output (SISO) systems from both accuracy and complexity perspectives, and propose to apply DL-AMR in the new multiple-input-multiple-output (MIMO) scenario with precoding. Finally, existing challenges and possible future research directions are discussed. [ABSTRACT FROM AUTHOR]

• A unified modeling and coupling framework for turbulence modeling over high Reynolds number airfoils is proposed. • A feature selection method based on the feature importance is proposed. It can effectively reduce the number of features and improve the stability and convergence of coupled computation. • Based on the new features, a data-driven turbulence model is constructed, which has good accuracy and generalization. • Mutual coupling between data-driven turbulence model and CFD solver is realized. In this paper, a turbulence model based on deep neural network is developed for turbulent flow around airfoil at high Reynolds numbers. According to the data got from the Spalart-Allmaras (SA) turbulence model, we build a neural network model that maps flow features to eddy viscosity. The model is then used to replace the SA turbulence model to mutually couple with the CFD solver. We build this suitable data-driven turbulence model mainly from the inputs, outputs features and loss function of the model. A feature selection method based on feature importance is also implemented. The results show that this feature selection method can effectively remove redundant features. The model based on the new input features has better accuracy and stability in mutual coupling with the CFD solver. The force coefficient obtained from solution can match the sample data well. The developed model also shows strong generalization at different inflow condition (angle of attack, Mach number, Reynolds number and airfoil). [ABSTRACT FROM AUTHOR]

The two main thrusts of computational science are increasingly accurate predictions and faster calculations; to this end, the zeitgeist in molecular dynamics (MD) simulations is pursuing machine learned and data driven interatomic models, e.g. neural network potentials, and novel hardware architectures, e.g. GPUs. Current implementations of neural network potentials are orders of magnitude slower than traditional interatomic models and while looming exascale computing offers the ability to run large, accurate simulations with these models, achieving portable performance for MD with new and varied exascale hardware requires rethinking traditional algorithms, using novel data structures, and library solutions. We re-implement a neural network interatomic model in CabanaMD, an MD proxy application, built on libraries developed for performance portability. Our implementation shows significantly improved thread scaling in this complex kernel as compared to a current LAMMPS implementation, across both strong and weak scaling. Our single-source solution enables simulations up to 20 million atoms on a single CPU node and 4 million atoms with improved performance on a single GPU. We also explore parallelism and data layout choices (using flexible data structures called AoSoAs) and their effect on performance, seeing up to ∼50% and ∼5% improvements in performance on a GPU by choosing the right level of parallelism and data layout respectively. Program title: CabanaMD-NNP CPC Library link to program files: https://doi.org/10.17632/x948kyy7jh.1 Developer's repository link: https://github.com/ECP-CoPA/CabanaMD , https://github.com/CompPhysVienna/n2p2 Licensing provisions: BSD3-Clause, GPL-3.0 Programming Language: C++ Nature of problem: Developing a performance portable implementation of a neural network potential for exascale architectures. Solution method: CabanaMD-NNP uses algorithms and data-structures from the Kokkos [1] and Cabana [2] libraries to re-implement the computations in Behler-Parrinello neural network potentials [3, 4] for performance portability across hardware. All molecular dynamics data is stored in performance portable data structures, with atomic properties in array-of-structs-of-arrays (Cabana::AoSoAs), and auxiliary values including potential parameters in arrays (Kokkos::Views). All computation is also done in a performance portable way: neural network propagation uses Kokkos parallel kernels (Kokkos::parallel_for), while calculations performed for each atom and neighbor, evaluation of descriptors (symmetry functions) and forces, use Cabana extensions of Kokkos constructs (Cabana::neighbor_parallel_for). These choices provide our implementation with significant speedups both on CPUs and GPUs for large systems, additionally allowing flexibility for parallelism and data layout for further optimizations. Additional comments including restrictions and unusual features: The previously developed n2p2 package [4] contains an interface to LAMMPS [5], which we compare to throughout the paper. We primarily extend the n2p2 library directly (https://github.com/CompPhysVienna/n2p2) and also add an interface to that extension within CabanaMD (https://github.com/ECP-CoPA/CabanaMD), to obtain the main results with an identical LAMMPS input file. [1] H.C. Edwards, C.R. Trott, D. Sunderland, Kokkos: Enabling manycore performance portability through polymorphic memory access patterns, Journal of Parallel and Distributed Computing. 74 (2014) 3202–3216. https://doi.org/10.1016/j.jpdc.2014.07.003. [2] S. Slattery, C. Junghans, D. Lebrun-Grandie, R. Halver, G. Chen, S. Reeve, A. Scheinberg, C. Smith, R. Bird, ECP-CoPA/Cabana: Cabana Version 0.2.0, Zenodo, 2019. [3] J. Behler, M. Parrinello, Generalized Neural-Network Representation of High-Dimensional Potential-Energy Surfaces, Phys. Rev. Lett. 98 (2007) 146401. [4] A. Singraber, J. Behler, C. Dellago, Library-Based LAMMPS Implementation of High-Dimensional Neural Network Potentials, J. Chem. Theory Comput. 15 (2019) 1827–1840. https://doi.org/10.1021/acs.jctc.8b00770. [5] S. Plimpton, Fast Parallel Algorithms for Short-Range Molecular Dynamics, Journal of Computational Physics. 117 (1995) 1–19. https://doi.org/10.1006/jcph.1995.1039. [ABSTRACT FROM AUTHOR]