Search

Spiking Neural Network (SNN) has been recognized as the third generation of neural networks. Conventionally, a SNN can be converted from a pre-trained Artificial Neural Network (ANN) with less computation and memory than training from scratch. But, these converted SNNs are vulnerable to adversarial attacks. Numerical experiments demonstrate that the SNN trained by optimizing the loss function will be more adversarial robust, but the theoretical analysis for the mechanism of robustness is lacking. In this paper, we provide a theoretical explanation by analyzing the expected risk function. Starting by modeling the stochastic process introduced by the Poisson encoder, we prove that there is a positive semidefinite regularizer. Perhaps surprisingly, this regularizer can make the gradients of the output with respect to input closer to zero, thus resulting in inherent robustness against adversarial attacks. Extensive experiments on the CIFAR10 and CIFAR100 datasets support our point of view. For example, we find that the sum of squares of the gradients of the converted SNNs is 13 ∼ 160 times that of the trained SNNs. And, the smaller the sum of the squares of the gradients, the smaller the degradation of accuracy under adversarial attack. • The adversarial robustness of SNNs based on rate encoder is explained mathematically. • A regularizer distinguishes the directly trained SNN from the converted SNN. • The coding length can also change the gradient of the input to affect the robustness. • We compare more types of the trained SNN with the converted SNN. [ABSTRACT FROM AUTHOR]

Deep learning has accomplished huge success in computer vision applications such as self-driving vehicles, facial recognition, and controlling robots. A growing need for deploying systems on resource-limited or resource-constrained environments such as smart cameras, autonomous vehicles, robots, smartphones, and smart wearable devices drives one of the current mainstream developments of convolutional neural networks: reducing model complexity but maintaining fine accuracy. In this study, the proposed efficient light convolutional neural network (ELNet) comprises three convolutional modules which perform ELNet using fewer computations, which is able to be implemented in resource-constrained hardware equipment. The classification task using CIFAR-10 and CIFAR-100 datasets was used to verify the model performance. According to the experimental results, ELNet reached 92.3% and 69%, respectively, in CIFAR-10 and CIFAR-100 datasets; moreover, ELNet effectively lowered the computational complexity and parameters required in comparison with other CNN architectures. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the effectiveness of applying deep neural network hidden Markov models, or DNN-HMMs, for acoustic modeling in the context of educational applications. Specifically, we focus on spoken responses from non-native and child speech that tend to show great acoustic variability. We perform comprehensive experiments to compare the performance between traditional Gaussian mixture model (GMM)-HMMs and DNN-HMMs in three large language assessment datasets that contain various spoken tasks, classified broadly as constrained and open-ended tasks. Our experimental results suggest useful conclusions that can help guide the design of real-life educational applications. DNN-HMMs outperform conventional GMM-HMMs by a large margin for all spoken tasks commonly used in spoken assessment applications. In our experiments, DNN-HMMs trained using 25 h of data can outperform GMM-HMMs trained with 6.7–9 times data. Specifically regarding overall performance, when all available training data were used (175, 227, 169 h respectively), we achieved a relative word error rate decrease of 20.4% for adult English and 29.3% for child English, and a relative character error rate decrease of 14.3% for adult Chinese, when switching from GMMs to DNNs. In comparing between types of tasks, we notice that the more challenging open-ended tasks benefit significantly more than constrained item types by the use of DNN-HMMs. For open-ended tasks, having large amounts of training data is the key, as DNN-HMMs can take full advantage of the added training data and further push performance. In contrast, the performance of constrained spoken tasks saturates at around 25 h of training data. At the same time, constrained spoken tasks require only a few hours of data (1 or 5 h) to build well-performing acoustic models. This is an encouraging observation, that indicates the potential to build reliable spoken assessment applications based on constrained tasks, when few domain specific training data are available. [ABSTRACT FROM AUTHOR]

Most traffic accidents resulted from distraction, inattention to surrounding cars, and driving fatigue. In order to protect drivers, a real-time lane-detection and front-vehicle distance measurement system that uses a mounted camera inside a vehicle has been designed for safe driving. For lane detection, the lane-boundary information is derived from the fan-scanning-detection method. The system calculates the departure degree according to the angular relationship of the boundaries and sends a suitable warning signal to drivers. For front-vehicle distance measurement, we use the front vehicle's shadow underneath it to identify the position of the front vehicle. The real distance is estimated by the use of the functional neurofuzzy network. The experimental results show that the system works successfully in real-time environment. [ABSTRACT FROM PUBLISHER]

A sequence of musical chords can facilitate musicians in music arrangement and accompaniment. To implement an intelligent system for chord recognition, in this article we propose a novel approach using artificial neural networks (ANN) trained bythe particle swarm optimization (PSO) technique and back-propagation (BP) learning algorithm. All of the training and testing data are generated from musical instrument digital interface (MIDI) symbolic data. Furthermore, in order to improve the recognition efficiency, an additional feature of cadencesis included. In other words, cadence is not only the structural punctuation of a melodic phrase but is considered as the important feature for chord recognition. Experimental results of our proposed approach show that adding a cadence feature significantly improves recognition rate, and the ANN-PSO method outperforms ANN-BP in chord recognition. In addition, because preliminary experimental recognition rates are generally not stable enough, we chose the optimal ANNs to propose a two-phase ANN model to integrate the results among many models. [ABSTRACT FROM AUTHOR]

Abstract: This study presents a functional neural fuzzy network (FNFN) for classification applications. The proposed FNFN model adopts a functional neural network (FLNN) to the consequent part of the fuzzy rules. Orthogonal polynomials and linearly independent functions are used for a functional expansion of the FLNN. Thus, the consequent part of the proposed FNFN model is a nonlinear combination of input variables. The FNFN model can construct its structure and adapt its free parameters with online learning algorithms, which consist of structure learning algorithm and parameter learning algorithm. The structure learning algorithm is based on the entropy measure to determine the number of fuzzy rules. The parameter learning algorithm, based on the gradient descent method, can adjust the shapes of the membership functions and the corresponding weights of the FLNN. Finally, the FNFN model is applied to various simulations. The simulation results for the Iris, Wisconsin breast cancer, and wine classifications show that FNFN model has superior performance than other models for classification applications. [ABSTRACT FROM AUTHOR]

Abstract: Kernel-based nonlinear characteristic extraction and classification algorithms are popular new research directions in machine learning. In this paper, we propose an improved photometric stereo scheme based on improved kernel-independent component analysis method to reconstruct 3D human faces. Next, we fetch the information of 3D faces for facial face recognition. For reconstruction, we obtain the correct normal vector’s sequence to form the surface, and use a method for enforcing integrability to reconstruct 3D objects. We test our algorithm on a number of real images captured from the Yale Face Database B, and use three kinds of methods to fetch characteristic values. Those methods are called contour-based, circle-based, and feature-based methods. Then, a three-layer, feed-forward neural network trained by a back-propagation algorithm is used to realize a classifier. All the experimental results were compared to those of the existing human face reconstruction and recognition approaches tested on the same images. The experimental results demonstrate that the proposed improved kernel independent component analysis (IKICA) method is efficient in reconstruction and face recognition applications. [ABSTRACT FROM AUTHOR]

Abstract: The implementation of adaptive neural fuzzy networks (NFNs) using field programmable gate arrays (FPGA) is proposed in this study. Hardware implementation of NFNs with learning ability is very difficult. The backpropagation (BP) method in the learning algorithm is widely used in NFNs, making it difficult to implement NFNs in hardware because calculating the backpropagation error of all parameters in a system is very complex. However, we use the simultaneous perturbation method as a learning scheme for the NFN hardware implementation. In order to reduce the chip area, we utilize the traditional non-linear activation function to implement the Gaussian function. We can confirm the reasonableness of NFN performance through some examples. [Copyright &y& Elsevier]

This article introduces a recurrent fuzzy neural network based on improved particle swarm optimisation (IPSO) for non-linear system control. An IPSO method which consists of the modified evolutionary direction operator (MEDO) and the Particle Swarm Optimisation (PSO) is proposed in this article. A MEDO combining the evolutionary direction operator and the migration operation is also proposed. The MEDO will improve the global search solution. Experimental results have shown that the proposed IPSO method controls the magnetic levitation system and the planetary train type inverted pendulum system better than the traditional PSO and the genetic algorithm methods. [ABSTRACT FROM AUTHOR]

The article presents a study on the adaptive neural fuzzy network (ANFN) controller based on a modified differential evolution (MODE) in solving control problems. It outlines the advantages and disadvantages of the ANFN, which motivated to the development of the MODE learning algorithm. It highlights the development of the hardware-in-the loop control technique for the ANFN controller by using MODE learning algorithm (ANFN-MODE), which has been used in planetary-train-type inverted pendulum system, and the magnetic visual simulation. The study has determined the positive result of the proposed ANFN-MODE controller.

This paper introduces a recurrent neural-fuzzy network (RNFN) based on improved particle swarm optimization (IPSO) for pattern recognition applications. The proposed IPSO method consists of the modified evolutionary direction operator (MEDO) and the traditional PSO. A novel MEDO combining the evolutionary direction operator (EDO) and the migration operation is also proposed. Hence, the proposed IPSO method can improve the ability of searching global solution. Experimental results have shown that the proposed IPSO method has a better performance than the traditional PSO in the human body classification and the skin color detection. [Copyright &y& Elsevier]

In this study, we proposed a new technique to compensate the backlight images. Two processing stages, called the backlight level detection and the backlight image compensation, are proposed. In the backlight level detection stage, we first transferred the color space to gray space by feature weighting, then obtain two backlight factors. We apply these two backlight factors to the proposed functional-link-based neuro-fuzzy network (FNFN) with immune particle swarm optimization (IPSO) for detecting compensation degree. In the backlight image compensation stage, we also proposed the adaptive cubic curve method to compensate and enhance the brightness of backlight images according to the compensation degree of each image. The backlight degree is indicated by histograms of the luminance distribution in the backlight level detection stage. The experiment results showed that the backlight images can be compensated effectively. [Copyright &y& Elsevier]

This study presents an evolutionary neural fuzzy network, designed using the functional-link-based neural fuzzy network (FLNFN) and a new evolutionary learning algorithm. This new evolutionary learning algorithm is based on a hybrid of cooperative particle swarm optimization and cultural algorithm. It is thus called cultural cooperative particle swarm optimization (CCPSO). The proposed CCPSO method, which uses cooperative behavior among multiple swarms, can increase the global search capacity using the belief space. Cooperative behavior involves a collection of multiple swarms that interact by exchanging information to solve a problem. The belief space is the information repository in which the individuals can store their experiences such that other individuals can learn from them indirectly. The proposed FLNFN model uses functional link neural networks as the consequent part of the fuzzy rules. This study uses orthogonal polynomials and linearly independent functions in a functional expansion of the functional link neural networks. The FLNFN model can generate the consequent part of a nonlinear combination of input variables. Finally, the proposed FLNFN with CCPSO (FLNFN-CCPSO) is adopted in several predictive applications. Experimental results have demonstrated that the proposed CCPSO method performs well in predicting the time series problems. [ABSTRACT FROM PUBLISHER]

Abstract: The brain–computer interface (BCI) is a system that transforms the brain activity of different mental tasks into a control signal. The system provides an augmentative communication method for patients with severe motor disabilities. In this paper, a neural classifier based on improved particle swarm optimization (IPSO) is proposed to classify an electroencephalogram (EEG) of mental tasks for left-hand movement imagination, right-hand movement imagination, and word generation. First, the EEG patterns utilize principle component analysis (PCA) in order to reduce the feature dimensions. Then a three-layer neural network trained using particle swarm optimization is used to realize a classifier. The proposed IPSO method consists of the modified evolutionary direction operator (MEDO) and the traditional particle swarm optimization algorithm (PSO). The proposed MEDO combines the evolutionary direction operator (EDO) and the migration. The MEDO can strengthen the searching global solution. The IPSO algorithm can prevent premature convergence and outperform the other existing methods. Experimental results have shown that our method performs well for the classification of mental tasks from EEG data. [Copyright &y& Elsevier]

Abstract: This paper introduces implementation of a wavelet neural network (WNN) with learning ability on field programmable gate array (FPGA). A learning algorithm using gradient descent method is not easy to implement in an electronic circuit and has local minimum. A more suitable method is the particle swarm optimization (PSO) that is a population-based optimization algorithm. The PSO is similar to the GA, but it has no evolution operators such as crossover and mutation. In the approximation of a nonlinear activation function, we use a Taylor series and a look-up table (LUT) to achieve a more accurate approximation. The results of the two experiments demonstrate the successful hardware implementation of the wavelet neural networks with the PSO algorithm using FPGA. From the results of the experiment, it can be seen that the performance of the PSO is better than that of the simultaneous perturbation algorithm at sufficient particle sizes. [Copyright &y& Elsevier]

Abstract: In this paper, a recurrent compensatory neuro-fuzzy system (RCNFS) for identification and prediction is proposed. The compensatory-based fuzzy method uses the adaptive fuzzy operations of neuro-fuzzy systems to make fuzzy logic systems more adaptive and effective. A recurrent network is embedded in the RCNFS by adding feedback connections in the second layer, where the feedback units act as memory elements. In this paper, the RCNFS model is proved to be a universal approximator. Also, an online learning algorithm is proposed which can automatically construct the RCNFS. There are no rules initially in the RCNFS. They are created and adapted as online learning proceeds through simultaneous structure and parameter learning. Structure learning is based on the degree measure and parameter learning is based on the ordered derivative algorithm. Finally, the RCNFS is used in several simulations. The simulation results of the dynamic system model have shown that (1) the RCNFS model converges quickly; (2) the RCNFS model requires a small number of tuning parameters; (3) the RCNFS model can solve temporal problems and approximate a dynamic system. [Copyright &y& Elsevier]

Abstract: A recurrent wavelet-based neuro fuzzy network (RWNFN) is proposed in this paper. The proposed RWNFN integrates wavelet transforms with fuzzy rules. Temporal relations are embedded in the network by adding feedback connections from memory units in the third layer of a feedforward wavelet-based neuro fuzzy network (WNFN). The RWNFN augments the basic ability of the WNFN to overcome temporal problems. Moreover, an online learning algorithm is proposed to automatically construct the RWNFN. Computer simulations were conducted to illustrate the performance and applicability of the proposed system. [Copyright &y& Elsevier]

This paper presents a wavelet-based recurrent fuzzy neural network (WRFNN) for prediction and identification of nonlinear dynamic systems. The proposed WRFNN model combines the traditional Thkagi-Sugeno-Kang (TSK) fuzzy model and the wavelet neural networks (WNN). This paper adopts the nonorthogonal and compactly supported functions as wavelet neural network bases. Temporal relations embedded in the network are caused by adding some feedback connections representing the memory units into the second layer of the feedforward wavelet-based fuzzy neural networks (WFNN). An online learning algorithm, which consists of structure learning and parameter learning, is also presented. The structure learning depends on the degree measure to obtain the number of fuzzy rules and wavelet functions. Meanwhile, the parameter learning is based on the gradient descent method for adjusting the shape of the membership function and the connection weights of WNN. Finally, computer simulations have demonstrated that the proposed WRFNN model requires fewer adjustable parameters and obtains a smaller rms error than other methods. [ABSTRACT FROM AUTHOR]

In this paper, we propose an Adaptive Neuro-Fuzzy Network (ANFN) to deal with forecasting problems. The ANFN model is inherently a modified Takagi-Sugeno-Kang-type fuzzy-rule-based model possessing a neural network's learning ability. We propose a hybrid learning algorithm which combines the Genetic Algorithm (GA) and the Least-Squares Estimate (LSE) method to construct the ANFN model. The GA is used to tune membership functions at the precondition part of fuzzy rules, while the LSE method is used to tune parameters at the consequent part of fuzzy rules. Simulations demonstrate that the proposed ANFN model has a good predictive capability. [ABSTRACT FROM AUTHOR]

One of the main tasks of chemometrics is to classify chemical objects to one of several distinct predefined categories. There are many classification methods in data mining, one of which is the boosting technique that can improve predicate performance of a given classifier and it is one of the most powerful methods in classification methodology. In this paper, we apply boosting neural network (NN) and boosting tree in classification for chemical data. Experimental results show that boosting can significantly improve the prediction performance of any single classification method. Two techniques to interpret the model are also introduced in order to help us better understand the experimental results. [Copyright &y& Elsevier]

Skeleton-based hand gesture recognition is an active research topic in computer graphics and computer vision and has a wide range of applications in VR/AR and robotics. Although the spatial–temporal graph convolutional network has been successfully used in skeleton-based hand gesture recognition, these works often use a fixed spatial graph according to the hand skeleton tree or use a fixed graph on the temporal dimension, which may not be optimal for hand gesture recognition. In this paper, we propose a two-stream graph attention convolutional network with spatial–temporal attention for hand gesture recognition. We adopt pose stream and motion stream as the two input streams for our network. In pose stream, we use the joint in each frame as the input; In motion stream, we use the joint offsets between neighboring frames as the input. We propose a new temporal graph attention module to model the temporal dependency and also use a spatial graph attention module to construct dynamic skeleton graph. For each stream, we adopt graph convolutional network with spatial–temporal attention to extract the features. Then, we concatenate the feature of the pose stream and motion stream for gesture recognition. We achieve the competitive performance on the main hand gesture recognition benchmark datasets, which demonstrates the effectiveness of our method. [ABSTRACT FROM AUTHOR]

• We propose a unified two-stage framework for automatic model compression. • We propose Dynamic BN to predict the performance of each compression policy. • We propose a solving algorithm to search for the compression ratio allocation. Deep Neural Networks have become ubiquitous in various domains. Meanwhile, the problems of massive storage and computation costs have hindered the deployment of these models to real-world applications. This paper proposes a novel and unified two-stage framework for automatic model compression. To determine the compression ratio of each layer, we improve the optimization from two aspects. First, to predict the performance of each compression policy, we propose Dynamic BN, which improves the correlation significantly with little computation overhead. Second, to search for the compression ratio allocation, we propose an efficient and hyperparameter-free solving algorithm based on the proposed Hessian matrix approximation and Knapsack problem reformulation. Moreover, comprehensive experiments and analyses are conducted on the CIFAR-100&ImageNet datasets and various network architectures to demonstrate its performance advantages over existing model compression methods under the quantization-only, pruning-only, and pruning-quantization settings. [ABSTRACT FROM AUTHOR]

Visualization from trained deep neural networks has drawn massive public attention in recent. One of the visualization approaches is to train images maximizing the activation of specific neurons. However, directly maximizing the activation would lead to unrecognizable images, which cannot provide any meaningful information. In this paper, we introduce a simple but effective technique to constrain the optimization route of the visualization. By adding two totally inverse transformations, image blurring and deblurring, to the optimization procedure, recognizable images can be created. Our algorithm is good at extracting the details in the images, which are usually filtered by previous methods in the visualizations. Extensive experiments on AlexNet, VGGNet and GoogLeNet illustrate that we can better understand the neural networks utilizing the knowledge obtained by the visualization. [ABSTRACT FROM AUTHOR]

Channel Pruning is one of the most widespread techniques used to compress deep neural networks while maintaining their performances. Currently, a typical pruning algorithm leverages neural architecture search to directly find networks with a configurable width, the key step of which is to identify representative subnet for various pruning ratios by training a supernet. However, current methods mainly follow a serial training strategy to optimize supernet, which is very time-consuming. In this work, we introduce PSE-Net, a novel parallel-subnets estimator for efficient channel pruning. Specifically, we propose a parallel-subnets training algorithm that simulate the forward–backward pass of multiple subnets by droping extraneous features on batch dimension, thus various subnets could be trained in one round. Our proposed algorithm facilitates the efficiency of supernet training and equips the network with the ability to interpolate the accuracy of unsampled subnets, enabling PSE-Net to effectively evaluate and rank the subnets. Over the trained supernet, we develop a prior-distributed-based sampling algorithm to boost the performance of classical evolutionary search. Such algorithm utilizes the prior information of supernet training phase to assist in the search of optimal subnets while tackling the challenge of discovering samples that satisfy resource constraints due to the long-tail distribution of network configuration. Extensive experiments demonstrate PSE-Net outperforms previous state-of-the-art channel pruning methods on the ImageNet dataset while retaining superior supernet training efficiency. For example, under 300M FLOPs constraint, our pruned MobileNetV2 achieves 75.2% Top-1 accuracy on ImageNet dataset, exceeding the original MobileNetV2 by 2.6 units while only cost 30%/16% times than BCNet/AutoAlim. • We propose PSE-Net, a novel parallel-subnets estimator for efficient channel pruning. • The prior-distributed-based sampling algorithm to facilitate subnets search. • 30%/16% total time for estimator training compared with BCNet/AutoSlim. • Exceeds the SOTAs on ImageNet and downstream tasks for a variety of FLOPs budgets. [ABSTRACT FROM AUTHOR]

The aim of this paper is to distinguish the vehicle detection and count the class number in each classification from the inputs. We proposed the use of Fuzzy Guided Scale Choice (FGSC)-based SSD deep neural network architecture for vehicle detection and class counting with parameter optimization. The 'FGSC' blocks are integrated into the convolutional layers of the model, which emphasize essential features while ignoring less important ones that are not significant for the operation. We created the passing detection lines and class counting windows and connected them with the proposed FGSC-SSD deep neural network model. The 'FGSC' blocks in the convolution layer emphasize essential features and find out unnecessary features by using the scale choice method at the training stage and eliminate that significant speedup of the model. In addition, FGSC blocks avoided many unusable parameters in the saturation interval and improved the performance efficiency. In addition, the Fuzzy Sigmoid Function (FSF) increases the activation interval through fuzzy logic. While performing operations, the FGSC-SSD model reduces the computational complexity of convolutional layers and their parameters. As a result, the model tested Frames Per Second (FPS) on edge artificial intelligence (AI) and reached a real-time processing speed of 38.4 and an accuracy rate of more than 94%. Therefore, this work might be considered an improvement to the traffic monitoring approach by using edge AI applications. [ABSTRACT FROM AUTHOR]