Search

Developments in deep learning with ANNs (Artificial Neural Networks) are paving the way for revolutionizing all application areas, especially related to non-linear regression and classification problems of predictive modelling and forecasting. Although their explainability is more complicated and challenging, deep neural networks are preferred over conventional machine learning methods for high accuracy in non-linear and complex problems. However, machine learning and data science practitioners often use ANN like a black-box. The present article concisely overviews the mathematics and computations involved in simple feed-forward neural networks (FNNs) or multilayer perceptrons (MLPs). The purpose is to spot light on what deep neural networks’ learning (or training) is and how it works. The article includes simplified derivations of the expressions for the main workhorse of neural networks (the backpropagation) and an example to explain how it works with graphical insights. An algorithm for a basic ANN application is presented in both component-form and matrix-form, together with a detailed note on the relevant data structures, to elaborate the scheme comprehensively. Python implementation of the basic algorithm is presented, and its performance results are compared with those produced using the TensorFlow library functions that implement the neural networks. The article discusses various techniques to improve the generalization capability of neural networks and how to address various training challenges. Finally, some well-established optimization approaches based on the Gradient Descent method are also discussed. The article may serve as a comprehensive premiere for a sound understanding of deep learning for undergraduate and graduate students before indulging in the relevant industry practices so that they can step into sustainable progress in the field.

Red, blue, white, pink, or black spots with irregular borders and small lesions on the skin are known as skin cancer that is categorized into two types: benign and malignant. Skin cancer can lead to death in advanced stages, however, early detection can increase the chances of survival of skin cancer patients. There exist several approaches developed by researchers to identify skin cancer at an early stage, however, they may fail to detect the tiniest tumours. Therefore, we propose a robust method for the diagnosis of skin cancer, namely SCDet, based on a convolutional neural network (CNN) having 32 layers for the detection of skin lesions. The images, having a size of 227 × 227, are fed to the image input layer, and then pair of convolution layers is utilized to withdraw the hidden patterns of the skin lesions for training. After that, batch normalization and ReLU layers are used. The performance of our proposed SCDet is computed using the evaluation matrices: precision 99.2%; recall 100%; sensitivity 100%; specificity 99.20%; and accuracy 99.6%. Moreover, the proposed technique is compared with the pre-trained models, i.e., VGG16, AlexNet, and SqueezeNet and it is observed that SCDet provides higher accuracy than these pre-trained models and identifies the tiniest skin tumours with maximum precision. Furthermore, our proposed model is faster than the pre-trained model as the depth of its architecture is not too high as compared to pre-trained models such as ResNet50. Additionally, our proposed model consumes fewer resources during training; therefore, it is better in terms of computational cost than the pre-trained models for the detection of skin lesions.

In the biometric field, vein identification is a vital process that is constrained by the invisibility of veins as well as other unique features. Moreover, users generally do not wish to have their personal information uploaded to the cloud, so edge computing has become popular for the sake of protecting user privacy. In this paper, we propose a low-complexity and lightweight convolutional neural network (CNN) and we design intellectual property (IP) for shortening the inference time in finger vein recognition. This neural network system can operate independently in client mode. After fetching the user’s finger vein image via a near-infrared (NIR) camera mounted on an embedded system, vein features can be efficiently extracted by vein curving algorithms and user identification can be completed quickly. Better image quality and higher recognition accuracy can be obtained by combining several preprocessing techniques and the modified CNN. Experimental data were collected by the finger vein image capture equipment developed in our laboratory based on the specifications of similar products currently on the market. Extensive experiments demonstrated the practicality and robustness of the proposed finger vein identification system.

Dealing with the insufficient detection accuracy and speed of aircraft targets in remote sensing images under complex background, this paper proposes a new detection method, YOLOv5-Aircraft, based on the YOLOv5 network. The YOLOv5-Aircraft model is improved in 3 ways: (1) At the beginning and end of original batch normalization module, centering and scaling calibration are added to enhance the effective features and form a more stable feature distribution, which strengthens the feature extraction ability of network model. (2) The cross-entropy loss function in the confidence of the original loss function is improved to the loss function based on smoothed Kullback-Leibler divergence. (3) For reducing information loss, the CSandGlass module is designed on the backbone feature extraction network of YOLOv5 to replace the residual module. Meanwhile, low-resolution feature layers are eliminated to reduce semantic loss. Experiment results demonstrate that the YOLOv5-Aircraft model can enhance the accuracy and speed of aircraft target detection in remote sensing images while achieving easier convergence.

International audience; Generative models get huge attention by researchers in different topics of artificial intelligence applications, especially generative adversarial networks (GANs) which have demonstrated good performance in data generation. In this paper, we would like to explore the potential of this class of models in producing human faces images. For that, we will use Deep Convolutional Generative Adversarial Network (DCGAN). Since that, the evaluation of GANs is still difficult even with the existing metrics like Inception Scores (IS), Mode Score (MS), Kernel Inception Distance (KID), Fréchet Inception Distance (FID), Multi-Scale Structural Similarity (MS-SSIM), etc. Thus, the best possible evaluation remains that carried out by human evaluators. This is why we propose a new hybrid measure combining qualitative and quantitative evaluation, we called this measure: Measuring the Quality of the Features of an Image (MEQFI). The images produced with the DC-GAN method were trained on three well known datasets from the literature and the results were evaluated with MEQFI.

Computed tomography (CT) is used in medical applications to produce digital medical imaging of the human body and is acquired by the reconstruction process, where X-rays are the key component of CT imaging. The present coronavirus outbreak has spawned new medical device and technology research fields. COVID-19 most severely affects people with poor immunity; children and pregnant women are more susceptible. A CT scan will be required to assess the infection’s severity. As a result, to reduce the radiation levels significantly there is a need to minimize the CT scan noise. The quality of CT images may degrade in the form of noisy images due to low radiation levels. Hence, this study proposes a novel denoising methodology for COVID-19 CT images with a low dose, where a convolution neural network (CNN) and batch normalization were utilized for denoising. From different output metrics such as peak signal-to-noise ratio (PSNR) and image quality index (IQI), the accuracy of the resulting CT images was checked and evaluated, where IQI obtained the best results in terms of 99% accuracy. The findings were also compared with the outcomes of related recent research in the domain. After a detailed review of the findings, it was noted that the proposed algorithm in the present study performed better in comparision to the existing literature.

Earthquakes threaten people, homes, and infrastructure. Early warning systems provide prior warning of oncoming significant shaking to decrease seismic risk by providing location, magnitude, and depth information of the event. Their usefulness depends on how soon a strong shake begins after the warning. In this article, the authors implement a deep learning model for predicting earthquakes. This model is based on a graph convolutional neural network with batch normalization and attention mechanism techniques that can successfully predict the depth and magnitude of an earthquake event at any number of seismic stations in any number of locations. After preprocessing the waveform data, CNN extracts the feature map. Attention mechanism is used to focus on important features. The batch normalization technique takes place in batches for stable and faster training of the model by adding an extra layer. GNN with extracted features and event location information predicts the event information accurately. We test the proposed model on two datasets from Japan and Alaska, which have different seismic dynamics. The proposed model achieves 2.8 and 4.0 RMSE values in Alaska and Japan for magnitude prediction, and 2.87 and 2.66 RMSE values for depth prediction. Low RMSE values show that the proposed model significantly outperforms the three baseline models on both datasets to provide an accurate estimation of the depth and magnitude of small, medium, and large-magnitude events.

Alzheimer’s disease is the most common cause of dementia and is a generic term for memory and other cognitive abilities that are severe enough to interfere with daily life. In this paper, we propose an improved prediction method for Alzheimer’s disease using a quantization method that transforms the MRI data set using a VGG-C Transform model and a convolutional neural network (CNN) consisting of batch normalization. MRI image data of Alzheimer’s disease are not fully disclosed to general research because it is data from real patients. So, we had to find a solution that could maximize the core functionality in a limited image. In other words, since it is necessary to adjust the interval, which is an important feature of MRI color information, rather than expressing the brain shape, the brain texture dataset was modified in the quantized pixel intensity method. We also use the VGG family, where the VGG-C Transform model with bundle normalization added to the VGG-C model performed the best with a test accuracy of about 0.9800. However, since MRI images are 208 × 176 pixels, conversion to 224 × 224 pixels may result in distortion and loss of pixel information. To address this, the proposed VGG model-based architecture can be trained while maintaining the original MRI size. As a result, we were able to obtain a prediction accuracy of 98% and the AUC score increased by up to 1.19%, compared to the normal MRI image data set. It is expected that our study will be helpful in predicting Alzheimer’s disease using the MRI dataset.

In wireless ccommunications receiver plays a main role to recognize modulation techniques which were used at the transmitter. While transferring information from transmitter to receiver, the receiver must retrieve original information. In order to achieve this goal we introduced a neural network architecture that recognizes the types of modulation techniques. The applications of deep learning can be categorized into classification and detection. The CNN architecture is used to perform feature extraction based on the layers to build a model which classifies the input data. A model that classifies the radio communication signals using deep learning method. The robust c (RCNN) is used to train the modulated signals; the transformations are used to help the neural network which estimate the signal to noise ratio of each signal ranges from -20dB to 18dB with loss and accuracy of 89.57% at SNR 0dB.

Earthquake is a major hazard to humans, buildings, and infrastructure. Early warning systems should detect an earthquake and issue a warning with earthquake information such as location, magnitude, and depth. Earthquake detection from raw waveform data using deep learning models such as graph neural networks (GNN) is becoming an important research area. The multilayered structure of the GNN with a number of epochs takes more training time. It is also hard to train the model with saturating nonlinearities. The batch normalization technique is applied to each mini-batch to reduce epochs in training and obtain a steady distribution of activation values. It improves model training and prediction accuracy. This study proposes a deep learning model batch normalization graph convolutional neural network (BNGCNN) for early earthquake detection. It consists of two main components: CNN and GNN. Input to the CNN model is multi-station and three-component waveform data with magnitude ≥3.0 were collected from January 2000 to January 2015 for Southern California. The extracted features of CNN are appended with location information and input to GNN model for earthquake detection. After hyperparameter tuning of the BNGCNN, when testing and evaluating the model on the Southern California dataset, our method shows promising results to the baseline model GNN by obtaining a low error rate to predict the magnitude, depth, and location of an earthquake.

It is considered t the Takagi-Sugeno-Kang fuzzy neural network and its modern variations. The use of regularization, random exclusion of rules from the rule base allows solving the problem of excessive similarity of rules in the rule base. The use of batch normalization to increase the generalizing properties of the network allows to increase the accuracy of the model, while maintaining the possibility of interpreting the results, which is characteristic of fuzzy neural networks. It is proposed to use an ensemble of fuzzy neural networks to increase the generalizing capabilities of the network. Studies of the Takagi-Sugeno-Kang fuzzy neural network for the task of diagnosing the coronavirus disease show that the proposed model works well and allows to improve the result. В статье рассматривается нечеткая нейронная сеть ТСК и ее современные вариации. Использование регуляризации, случайного исключения правил из базы правил позволяет решить проблему большого сходства правил в базе правил. Использование пакетной нормализации для увеличения обобщающих свойств сети позволяет повысить точность модели, сохраняя при этом возможность интерпретации результатов, свойственных нечетким нейронным сетям. Предложено использование ансамбля нечетких нейронных сетей для повышения обобщающих возможностей сети. Исследования с большими наборами данных как для задачи диагностики болезни сердца, так и для задачи диагностики коронавирусной болезни показывают, что предложенная модель хорошо работает и позволяет улучшить результат. В статті розглядається нечітка нейронна мережа ТСК та сучасні її варіації. Використання регуляризації, випадкового виключення правил з бази правил дозволяє вирішити проблему завеликої схожості правил в базі правил. Використання пакетної нормалізації для збільшення узагальнюючих властивостей мережі дозволяє підвищити точність моделі, зберігаючи при цьому можливість інтерпретації результатів, яка властива нечітким нейронним мережам. Запропоновано використання ансамблю нечітких нейронних мереж для підвищення узагальнюючих можливостей мережі. Дослідження з великими наборами даних як для задачі діагностики хвороби серця так і для задачі діагностики коронавірусної хвороби показують, що запропонована модель добре працює і дозволяє покращити результат.

A sudden blow or jolt to the human brain called traumatic brain injury (TBI) is one of the most common injuries recorded in the health insurance claim. Generally, computed tomography (CT) or magnetic resonance imaging (MRI) is required to identify the trauma’s severity. Unfortunately, CT and MRI equipment are bulky, expensive, and not always available, limiting their use in TBI detection. Therefore, as an alternative, this study presents a novel classification architecture that can classify non-severe TBI patients from healthy subjects by using resting-state electroencephalogram (EEG) as the input. The proposed architecture employs a convolutional neural network (CNN), and error-correcting output codes support vector machine (ECOC-SVM) to perform automated feature extraction and multi-class classification. In this architecture, complex feature selection and extraction steps are avoided. The proposed architecture attained a high-performance classification accuracy of 99.76%, potentially being used as a classification approach to preventing healthcare insurance fraud. The proposed method is compared to existing studies in the literature. The outcome from the comparisons indicates that the proposed method has outperformed the benchmarked methods by presenting the highest classification accuracy and precision.

Collaborative training of deep neural networks using edge devices has attracted substantial research interest recently. The two main architecture approaches for the training process are centrally orchestrated Federated Learning and fully decentralized peer-to-peer learning. In decentralized systems, edge devices, known as agents, collaborate in a peer-to-peer architecture, avoiding the need for a central system to orchestrate the process. Decentralized peer-to-peer (P2P) learning techniques are well researched under the assumption of independent and identically distributed (IID) data across the agents. IID data is seldom observed in real-world distributed systems, and the training performance varies significantly with non-IID data. This paper proposes a decentralized learning variant of the P2P gossip averaging method with Batch Normalization (BN) adaptation for P2P architectures. It is well-known that BN layers accelerate the convergence of the non-distributed deep learning models. Recent research confirms that Federated Learning methods benefit from using the BN method with some aggregation alterations. Our work demonstrated BN effectiveness in P2P architectures by mitigating the non-IID data characteristics across decentralized agents. We also introduce a variant of the early stopping technique that, combined with BN layers, acts as a fine-tuning technique for agent models. We validated our approach by conducting numerous simulations of different model-topology- communication combinations and comparing them to other decentralized baseline approaches. The evaluations were conducted on the next word prediction task using user comments from the Reddit and StackOverflow datasets representing comments from two different domains. Simulations showed that our approach, on average, achieves a mean relative top accuracy increase of 16.9% in ring (19.9% for Reddit, 13.9% for StackOverflow) and 29.8% in sparse (32.9% for Reddit, 26.6% for StackOverflow) communication topologies compared to the best baseline approach. Our code is available at https://github.com/fipu-lab/p2p_bn.

With the improvement of industrial requirements for the quality of cold rolled strips, flatness has become one of the most important indicators for measuring the quality of cold rolled strips. In this paper, the strip production data of a 1250 mm tandem cold mill in a steel plant is modeled by an improved deep neural network (the improved DNN) to improve the accuracy of strip shape prediction. Firstly, the type of activation function is analyzed, and the monotonicity of the activation function is deemed independent of the convexity of the loss function in the deep network. Regardless of whether the activation function is monotonic, the loss function is not strictly convex. Secondly, the non-convex optimization of the loss functionextended from the deep linear network to the deep nonlinear network, is discussed, and the critical point of the deep nonlinear network is identified as the global minimum point. Finally, an improved Swish activation function based on batch normalization is proposed, and its performance is evaluated on the MNIST dataset. The experimental results show that the loss of an improved Swish function is lower than that of other activation functions. The prediction accuracy of a deep neural network (DNN) with an improved Swish function is 0.38% more than that of a deep neural network (DNN) with a regular Swish function. For the DNN with the improved Swish function, the mean square error of the prediction for the flatness of cold rolled strip is reduced to 65% of the regular DNN. The accuracy of the improved DNN is up to and higher than the industrial requirements. The shape prediction of the improved DNN will assist and guide the industrial production process, reducing the scrap yield and industrial cost.

Past couple of years, the world is going through one of the biggest pandemic named COVID-19. In the mid of year 2019, it is a very difficult process to predict the COVID-19 just by viewing the images. Later on AI based technology has done a significant role in the prediction of COVID-19 through biomedical images such as CT scan, X ray etc. This study also implemented the deep learning model for the prediction of COVID-19 through X-ray images. The implemented model is termed as XR-CAPS which consist of two models such as U-Net model and the capsule network. The U Net model is used for performing the segmentation of the images and the capsule networks are applied for performing the feature extraction. The XR-CAPS model is applied on the X-ray images for the prediction of COVID-19 and the evaluation of the model is done by three parameters that are accuracy, sensitivity and specificity. The model is compared with other existing models like ResNet50, DenseNet121 and DenseCapsNet, this has achieved an accuracy of 93.2%, sensitivity of 94% and specificity of 97.1% which is better than other states of the art algorithms.

Alzheimer’s disease (AD) is a global health issue that predominantly affects older people. It affects one’s daily activities by modifying neural networks in the brain. AD is categorized by the death of neurons, the creation of amyloid plaques, and the development of neurofibrillary tangles. In clinical settings, an early diagnosis of AD is critical to limit the problems associated with it and can be accomplished using neuroimaging modalities, such as magnetic resonance imaging (MRI) and positron emission tomography (PET). Deep learning (DL) techniques are widely used in computer vision and related disciplines for various tasks such as classification, segmentation, detection, etc. CNN is a sort of DL architecture, which is normally useful to categorize and extract data in the spatial and frequency domains for image-based applications. Batch normalization and dropout are commonly deployed elements of modern CNN architectures. Due to the internal covariance shift between batch normalization and dropout, the models perform sub-optimally under diverse scenarios. This study looks at the influence of disharmony between batch normalization and dropout techniques on the early diagnosis of AD. We looked at three different scenarios: (1) no dropout but batch normalization, (2) a single dropout layer in the network right before the softmax layer, and (3) a convolutional layer between a dropout layer and a batch normalization layer. We investigated three binaries: mild cognitive impairment (MCI) vs. normal control (NC), AD vs. NC, AD vs. MCI, one multiclass AD vs. NC vs. MCI classification problem using PET modality, as well as one binary AD vs. NC classification problem using MRI modality. In comparison to using a large value of dropout, our findings suggest that using little or none at all leads to better-performing designs.

A power device is a semiconductor device for power control used for power conversion such as converting direct current to alternating current and alternating current to direct current. It is widely used such as refrigerators, air conditioners which is implemented electronic components that are closely related to our daily lives. Therefore, high reliability and safety are required, and power cycle tests are conducted for the purpose of evaluating them. In the conventional test, there is a problem that it is difficult to perform analysis because sparks are generated during the test and the device is severely damaged after the test. To solve this problem, a new technology has been developed that adds ultrasonic that enable internal observation during the test. However, there are remains a problem that the method for analyzing the ultrasonic image obtained in the new technology has not been established. Also, few abnormal images are obtained in the test. In this paper, we propose a method for detection of abnormal devices based on CNN. Especially, we implement a Cycle-GAN to extend the abnormal data and classify the known image based on improved VGG16. As an experimental result, classification accuracy of Precision = 97.06%, Recall = 93.58%, F - measure = 95.17% were obtained., 21th International Conference on Control, Automation and Systems, ICCAS 2021, October 12-15, 2021, Jeju, Korea and online

Problem određivanja dobi na temelju slika lica riješen je uporabom duboke konvolucijske mreže s 5 konvolucijskih slojeva, 5 slojeva sažimanja, 5 normalizacijskih slojeva i 3 potpuno povezana sloja. Problemu se pristupilo višeklasnom klasifikacijom gdje je model trebao prediktirati dob osobe na slici u 8 dobnih skupina. IMDB-WIKI je korišteni skup podataka, a on se podijelio na skup uzoraka za učenje, validacijski i testni skup. Postupak učenja koristi mini grupe veličine 128 slika koje se propagiraju kroz mrežu unaprijed sve do izlaznog sloja mreže. Na izlazu iz mreže računa se funkcija gubitka, a korištena je unakrsna entropija kojom se provjerava koliko model griješi. Optimizacijski algoritam nastojao je minimizirati funkciju gubitka, a prilikom učenja je koristio gradijent kao informaciju kojom je provodio ažuriranje težina u modelu. Gradijenti su se računali unatražnim prolazom od izlaznog do ulaznog sloja modela. Korišteni algoritam je proširenje stohastičkog gradijentnog spusta, a zove se Adam. Slike nad kojima se model učio bile su veličine 128 x 128 na ulazu, a na izlazu, zbog raznih linearnih transformacija poput konvolucije, su smanjene na 4 x 4. Da ne bi došlo do prenaučenosti modela u njega su eksplicitno ugrađene dvije regularizacijske tehnike: regularizacija L2 normom vektora parametara modela i rano zaustavljanje koje je na temelju generalizacijske pogreške odlučivalo o nastavku ili prekidu treninga. Također, model osjeti implicitne efekte regularizacije koje su vezane uz korištenje normalizacije nad grupom slika nakon svakog konvolucijskog sloja. Ovakvim pristupom problemu postignuta je točnost modela od 35%, a uspoređujući s točnosti ispitanih ljudi može se reći da model dobro klasificira osobe u dobne skupine kojima pripadaju. Age Estimaton of Persons in Facial Images problem is solved with deep convolutional neural network which has 5 convolution layers, 5 max pooling layers, 5 normalization layers and 3 fully connected layers. Approach to the problem was multiclass classification, where model should estimate age of a person on the image in 8 different age groups. IMDB-WIKI dataset is used as a main dataset which is divided in the three parts: training set, validation set and test set. Learning process uses mini batches of size 128 images that propagate through the network from the input to the output layer. Output of the network gave calculation of the loss function which tells us how wrong the model is. Cross entropy was used as a loss function. Purpose of the optimization algorithm is to minimize loss function and as the information for update weights in the model, algorithm used gradient. Backpropagation gave calculations for all gradients in the model. Used algorithm is extension of the stohastic gradient descent called Adam. Images that was used as the inputs by the model was dimensions 128 x 128. Because of different linear transformations between hidden layers in the network, like convolution, images was reduced on 4 x 4. To avoid overfitting, two regularization techniques is explicitly built-in in the model: early stopping and regularization with L2 norm vector parameters. Early stopping is technique that tells us when to stop training based on generalization error. Also, model has some implicit efects of the regularization as batch normalization of the images after every convolution layer. Whole approach to the problem led to the accuracy of the model of 35%. Considering conducted human estimation we can say that model is pretty good classifier for this problem.

An enormous number of CNN classification algorithms have been proposed in the literature. Nevertheless, in these algorithms, appropriate filter size selection, data preparation, limitations in datasets, and noise have not been taken into consideration. As a consequence, most of the algorithms have failed to make a noticeable improvement in classification accuracy. To address the shortcomings of these algorithms, our paper presents the following contributions: Firstly, after taking the domain knowledge into consideration, the size of the effective receptive field (ERF) is calculated. Calculating the size of the ERF helps us to select a typical filter size which leads to enhancing the classification accuracy of our CNN. Secondly, unnecessary data leads to misleading results and this, in turn, negatively affects classification accuracy. To guarantee the dataset is free from any redundant or irrelevant variables to the target variable, data preparation is applied before implementing the data classification mission. Thirdly, to decrease the errors of training and validation, and avoid the limitation of datasets, data augmentation has been proposed. Fourthly, to simulate the real-world natural influences that can affect image quality, we propose to add an additive white Gaussian noise with σ = 0.5 to the MNIST dataset. As a result, our CNN algorithm achieves state-of-the-art results in handwritten digit recognition, with a recognition accuracy of 99.98%, and 99.40% with 50% noise.

A conditional generative adversarial network (cGAN) is a generative adversarial network (GAN) that generates data with a desired condition from a latent vector. Among the different types of cGAN, the auxiliary classifier GAN (ACGAN) is the most frequently used. In this study, we describe the problems of an AC-GAN and propose replacing it with a conditional activation GAN (CAGAN) to reduce the number of hyperparameters and improve the training speed. The loss function of a CAGAN is defined as the sum of the loss of each GAN created for each condition. The proposed CAGAN is an integration of multiple GANs, where each GAN shares all hidden layers, and their integration can be considered as a single GAN. Therefore, the structure of the integrated GANs does not significantly increase the number of computations. Additionally, to prevent the conditions given in the discriminator of a cGAN from being ignored with batch normalization, we propose mixed batch training, in which every batch for the discriminator keeps the ratio of the real and generated data consistent.

Recurrent neural network (RNN) has the ability to learn long-term dependencies, which makes it suitable for acoustic modeling in speech recognition. In this paper, we revise RNN model used in acoustic modeling, namely, mGRUIP with Context module (mGRUIP-Ctx), and propose an advanced model which named Projected minimal Gated Recurrent Unit (PmGRU). The paper demonstrates two major contributions: firstly, in the case that adding context information to context module in mGRUIP-Ctx will bring about large amount of parameter, we propose to insert a smaller output projection layer after the mGRUIP-Ctx cell's output to form the PmGRU, which is inspired by the idea of low-rank decomposition of matrix. The output projection layer has been proved to be able to save most of the effective information with the reduction of model parameters. Secondly, in the case that too much context information of previous layer introduced by context module will cause declining of model performance, we adjust the ratio of context information of the previous layer to the current layer by moving the position of batch normalization layer, and the final RNN model Normalization Projected minimal Gated Recurrent Unit (Norm-PmGRU) is generated. In the five automatic speech recognition (ASR) tasks, the Norm-PmGRU has been proved more effectively in the experiments compared with mGRUIP-Ctx, TDNN-OPGRU, TDNN-LSTMP and other RNN baseline acoustics models.

Les réseaux de neurones sont une famille de modèles de l'apprentissage automatique qui sont capable d'apprendre des tâches complexes directement des données. Bien que produisant déjà des résultats impressionnants dans beaucoup de domaines tels que la reconnaissance de la parole, la vision par ordinateur ou encore la traduction automatique, il y a encore de nombreux défis dans l'entraînement et dans le déploiement des réseaux de neurones. En particulier, entraîner des réseaux de neurones nécessite typiquement d'énormes ressources computationnelles, et les modèles entraînés sont souvent trop gros ou trop gourmands en ressources pour être déployés sur des appareils dont les ressources sont limitées, tels que les téléphones intelligents ou les puces de faible puissance. Les articles présentés dans cette thèse étudient des solutions à ces différents problèmes. Les deux premiers articles se concentrent sur l'amélioration de l'entraînement des réseaux de neurones récurrents (RNNs), un type de réseaux de neurones particulier conçu pour traiter des données séquentielles. Les RNNs sont notoirement difficiles à entraîner, donc nous proposons d'améliorer leur paramétrisation en y intégrant la normalisation par lots (BN), qui était jusqu'à lors uniquement appliquée aux réseaux non-récurrents. Dans le premier article, nous appliquons BN aux connections des entrées vers les couches cachées du RNN, ce qui réduit le décalage covariable entre les différentes couches; et dans le second article, nous montrons comment appliquer BN aux connections des entrées vers les couches cachées et aussi des couches cachée vers les couches cachée des réseau récurrents à mémoire court et long terme (LSTM), une architecture populaire de RNN, ce qui réduit également le décalage covariable entre les pas de temps. Nos expériences montrent que les paramétrisations proposées permettent d'entraîner plus rapidement et plus efficacement les RNNs, et ce sur différents bancs de tests. Dans le troisième article, nous proposons un nouvel optimiseur pour accélérer l'entraînement des réseaux de neurones. Les optimiseurs diagonaux traditionnels, tels que RMSProp, opèrent dans l'espace des paramètres, ce qui n'est pas optimal lorsque plusieurs paramètres sont mis à jour en même temps. A la place, nous proposons d'appliquer de tels optimiseurs dans une base dans laquelle l'approximation diagonale est susceptible d'être plus efficace. Nous tirons parti de l'approximation K-FAC pour construire efficacement cette base propre Kronecker-factorisée (KFE). Nos expériences montrent une amélioration en vitesse d'entraînement par rapport à K-FAC, et ce pour différentes architectures de réseaux de neurones profonds. Le dernier article se concentre sur la taille des réseaux de neurones, i.e. l'action d'enlever des paramètres du réseau, afin de réduire son empreinte mémoire et son coût computationnel. Les méthodes de taille typique se base sur une approximation de Taylor de premier ou de second ordre de la fonction de coût, afin d'identifier quels paramètres peuvent être supprimés. Nous proposons d'étudier l'impact des hypothèses qui se cachent derrière ces approximations. Aussi, nous comparons systématiquement les méthodes basées sur des approximations de premier et de second ordre avec la taille par magnitude (MP), et montrons comment elles fonctionnent à la fois avant, mais aussi après une phase de réapprentissage. Nos expériences montrent que mieux préserver la fonction de coût ne transfère pas forcément à des réseaux qui performent mieux après la phase de réapprentissage, ce qui suggère que considérer uniquement l'impact de la taille sur la fonction de coût ne semble pas être un objectif suffisant pour développer des bon critères de taille., Neural networks are a family of Machine Learning models able to learn complex tasks directly from the data. Although already producing impressive results in many areas such as speech recognition, computer vision or machine translation, there are still a lot of challenges in both training and deployment of neural networks. In particular, training neural networks typically requires huge amounts of computational resources, and trained models are often too big or too computationally expensive to be deployed on resource-limited devices, such as smartphones or low-power chips. The articles presented in this thesis investigate solutions to these different issues. The first couple of articles focus on improving the training of Recurrent Neural Networks (RNNs), networks specially designed to process sequential data. RNNs are notoriously hard to train, so we propose to improve their parameterisation by upgrading them with Batch Normalisation (BN), a very effective parameterisation which was hitherto used only in feed-forward networks. In the first article, we apply BN to the input-to-hidden connections of the RNNs, thereby reducing internal covariate shift between layers. In the second article, we show how to apply it to both input-to-hidden and hidden-to-hidden connections of the Long Short-Term Memory (LSTM), a popular RNN architecture, thus also reducing internal covariate shift between time steps. Our experiments show that these proposed parameterisations allow for faster and better training of RNNs on several benchmarks. In the third article, we propose a new optimiser to accelerate the training of neural networks. Traditional diagonal optimisers, such as RMSProp, operate in parameters coordinates, which is not optimal when several parameters are updated at the same time. Instead, we propose to apply such optimisers in a basis in which the diagonal approximation is likely to be more effective. We leverage the same approximation used in Kronecker-factored Approximate Curvature (K-FAC) to efficiently build this Kronecker-factored Eigenbasis (KFE). Our experiments show improvements over K-FAC in training speed for several deep network architectures. The last article focuses on network pruning, the action of removing parameters from the network, in order to reduce its memory footprint and computational cost. Typical pruning methods rely on first or second order Taylor approximations of the loss landscape to identify which parameters can be discarded. We propose to study the impact of the assumptions behind such approximations. Moreover, we systematically compare methods based on first and second order approximations with Magnitude Pruning (MP), showing how they perform both before and after a fine-tuning phase. Our experiments show that better preserving the original network function does not necessarily transfer to better performing networks after fine-tuning, suggesting that only considering the impact of pruning on the loss might not be a sufficient objective to design good pruning criteria.

Efficient machine learning implementations optimized for inference in hardware have wide-ranging benefits, depending on the application, from lower inference latency to higher data throughput and reduced energy consumption. Two popular techniques for reducing computation in neural networks are pruning, removing insignificant synapses, and quantization, reducing the precision of the calculations. In this work, we explore the interplay between pruning and quantization during the training of neural networks for ultra low latency applications targeting high energy physics use cases. Techniques developed for this study have potential applications across many other domains. We study various configurations of pruning during quantization-aware training, which we term quantization-aware pruning, and the effect of techniques like regularization, batch normalization, and different pruning schemes on performance, computational complexity, and information content metrics. We find that quantization-aware pruning yields more computationally efficient models than either pruning or quantization alone for our task. Further, quantization-aware pruning typically performs similar to or better in terms of computational efficiency compared to other neural architecture search techniques like Bayesian optimization. Surprisingly, while networks with different training configurations can have similar performance for the benchmark application, the information content in the network can vary significantly, affecting its generalizability., 22 pages, 7 Figures, 1 Table

At present, part of people's body is in the state of sub-health, and more people pay attention to physical exercise. Dance is a relatively simple and popular activity, it has been widely concerned. The traditional action recognition method is easily affected by the action speed, illumination, occlusion and complex background, which leads to the poor robustness of the recognition results. In order to solve the above problems, an improved residual dense neural network method is used to study the automatic recognition of dance action images. Firstly, based on the residual model, the features of dance action are extracted by using the convolution layer and pooling layer. Then, the exponential linear element (ELU) activation function, batch normalization (BN) and Dropout technology are used to improve and optimize the model to mitigate the gradient disappearance, prevent over-fitting, accelerate convergence and enhance the model generalization ability. Finally, the dense connection network (DenseNet) is introduced to make the extracted dance action features more rich and effective. Comparison experiments are carried out on two public databases and one self-built database. The results show that the recognition rate of the proposed method on three databases are 99.98, 97.95, and 0.97.96%, respectively. It can be seen that this new method can effectively improve the performance of dance action recognition.

A system for automatic pain detection whereby pain-related features are extracted from facial images using a four-layer Convolutional Deep Belief Network (CDBN) is proposed in this study. The CDBN is trained by greedy layer-wise procedure whereby each added layer is trained as a Convolutional Restricted Boltzmann Machine (CRBM) by contrastive divergence. Since conventional CRBM is trained in a purely unsupervised manner, there is no guarantee that learned features are appropriate for the supervised task at hand. A discriminative objective based on between-class and within-class distances is proposed to adapt CRBM to learn task-related features. When discriminative and generative objectives are appropriately combined, a competitive classification performance can be achieved. Moreover, we introduced batch normalization (BN) units in the structure of the CRBM model to smooth optimization landscape and speed up the learning process. BN units come right before sigmoid units. Extracted features are then used to train a linear SVM to classify each frame into pain or no-pain classes. Extensive experiments on UNBC-McMaster Shoulder Pain database demonstrate the effectiveness of the proposed method for automatic pain detection.

The COVID-19 pandemic has caused a global alarm. With the advances in artificial intelligence, the COVID-19 testing capabilities have been greatly expanded, and hospital resources are significantly alleviated. Over the past years, computer vision researches have focused on convolutional neural networks (CNNs), which can significantly improve image analysis ability. However, CNN architectures are usually manually designed with rich expertise that is scarce in practice. Evolutionary algorithms (EAs) can automatically search for the proper CNN architectures and voluntarily optimize the related hyperparameters. The networks searched by EAs can be used to effectively process COVID-19 computed tomography images without expert knowledge and manual setup. In this paper, we propose a novel EA-based algorithm with a dynamic searching space to design the optimal CNN architectures for diagnosing COVID-19 before the pathogenic test. The experiments are performed on the COVID-CT data set against a series of state-of-the-art CNN models. The experiments demonstrate that the architecture searched by the proposed EA-based algorithm achieves the best performance yet without any preprocessing operations. Furthermore, we found through experimentation that the intensive use of batch normalization may deteriorate the performance. This contrasts with the common sense approach of manually designing CNN architectures and will help the related experts in handcrafting CNN models to achieve the best performance without any preprocessing operations

Offline Arabic Handwriting Recognition (OAHR) has recently become instrumental in the areas of pattern recognition and image processing due to its application in several fields, such as office automation and document processing. However, OAHR continues to face several challenges, including the high variability of the Arabic script and its intrinsic characteristics such as cursiveness, ligatures, and diacritics, the unlimited variation in human handwriting, and the lack of large public databases. In this paper, we have introduced a novel context-aware model based on deep neural networks to address the challenges of recognizing offline handwritten Arabic text, including isolated digits, characters, and words. Specifically, we have proposed a supervised Convolutional Neural Network (CNN) model that contextually extracts optimal features and employs batch normalization and dropout regularization parameters to prevent overfitting and further enhance its generalization performance when compared to conventional deep learning models. We employed numerous deep stacked-convolutional layers to design the proposed Deep CNN (DCNN) architecture. The proposed model was extensively evaluated, and it was observed to achieve excellent classification accuracy when compared to the existing state-of-the-art OAHR approaches on a diverse set of six benchmark databases, including MADBase (Digits), CMATERDB (Digits), HACDB (Characters), SUST-ALT (Digits), SUST-ALT (Characters), and SUST-ALT (Names). Further comparative experiments were conducted on the respective databases using the pre-trained VGGNet-19 and Mobile-Net models, additionally, generalization capabilities experiments on another language database (i.e., MNIST English Digits) were conducted, which showed the superiority of the proposed DCNN model.

Offline Arabic Handwriting Recognition (OAHR) has recently become instrumental in the areas of pattern recognition and image processing due to its application in several fields, such as office automation and document processing. However, OAHR continues to face several challenges, including high variability of the Arabic script and its intrinsic characteristics such as cursiveness, ligatures, and diacritics, the unlimited variation in human handwriting, and the lack of large public databases. In this paper, we introduce a novel context-aware model based on deep neural networks to address the challenges of recognizing offline handwritten Arabic text, including isolated digits, characters, and words. Specifically, we propose a supervised Convolutional Neural Network (CNN) model that contextually extracts optimal features and employs batch normalization and dropout regularization parameters. This aims to prevent overfitting and further enhance generalization performance when compared to conventional deep learning models. We employ a number of deep stacked-convolutional layers to design the proposed Deep CNN (DCNN) architecture. The model is extensively evaluated and shown to demonstrate excellent classification accuracy when compared to conventional OAHR approaches on a diverse set of six benchmark databases, including MADBase (Digits), CMATERDB (Digits), HACDB (Characters), SUST-ALT (Digits), SUST-ALT (Characters), and SUST-ALT (Names). A further experimental study is conducted on the benchmark Arabic databases by exploiting transfer learning (TL)-based feature extraction which demonstrates the superiority of our proposed model in relation to state-of-the-art VGGNet-19 and MobileNet pre-trained models. Finally, experiments are conducted to assess comparative generalization capabilities of the models using another language database , specifically the benchmark MNIST English isolated Digits database, which further confirm the superiority of our proposed DCNN model.

Normalization as a layer within neural networks has over the years demonstrated its effectiveness in neural network optimization across a wide range of different tasks, with one of the most successful approaches being that of batch normalization. The consensus is that better estimates of the BatchNorm normalization statistics (μ and σ2) in each mini-batch result in better optimization. In this work, we challenge this belief and experiment with a new variant of BatchNorm known as GhostNorm that, despite independently normalizing batches within the mini-batches, i.e., μ and σ2 are independently computed and applied to groups of samples in each mini-batch, outperforms BatchNorm consistently. Next, we introduce sequential normalization (SeqNorm), the sequential application of the above type of normalization across two dimensions of the input, and find that models trained with SeqNorm consistently outperform models trained with BatchNorm or GhostNorm on multiple image classification data sets. Our contributions are as follows: (i) we uncover a source of regularization that is unique to GhostNorm, and not simply an extension from BatchNorm, and illustrate its effects on the loss landscape, (ii) we introduce sequential normalization (SeqNorm) a new normalization layer that improves the regularization effects of GhostNorm, (iii) we compare both GhostNorm and SeqNorm against BatchNorm alone as well as with other regularization techniques, (iv) for both GhostNorm and SeqNorm models, we train models whose performance is consistently better than our baselines, including ones with BatchNorm, on the standard image classification data sets of CIFAR–10, CIFAR-100, and ImageNet ((+0.2%, +0.7%, +0.4%), and (+0.3%, +1.7%, +1.1%) for GhostNorm and SeqNorm, respectively).

Building a deep learning model based on small dataset is difficult, even impossible. Toavoiding overfitting, we must constrain model, which we train. Techniques as data augmentation,regularization or data normalization could be crucial. We have created a benchmark with a simpleCNN image classifier in order to find the best techniques. As a result, we compare different types ofdata augmentation and weights regularization and data normalization on a small dataset.

In this paper, we propose a high performance Two-Stream spectral-spatial Residual Network (TSRN) for hyperspectral image classification. The first spectral residual network (sRN) stream is used to extract spectral characteristics, and the second spatial residual network (saRN) stream is concurrently used to extract spatial features. The sRN uses 1D convolutional layers to fit the spectral data structure, while the saRN uses 2D convolutional layers to match the hyperspectral spatial data structure. Furthermore, each convolutional layer is preceded by a Batch Normalization (BN) layer that works as a regularizer to speed up the training process and to improve the accuracy. We conducted experiments on three well-known hyperspectral datasets, and we compare our results with five contemporary methods across various sizes of training samples. The experimental results show that the proposed architecture can be trained with small size datasets and outperforms the state-of-the-art methods in terms of the Overall Accuracy, Average Accuracy, Kappa Value, and training time.

Jedan od glavnih izazova treniranja dubokih neuronskih mreža s milijunima parametara je izbjegavanje prenaučenosti koja rezultira lošom generalizacijom na novim podatcima. Problemu prenaučenosti može se pristupiti dodavanjem novih podataka skupu za treniranje ili regularizacijom. Cilj ovog diplomskog rada je dati pregled različitih metoda regularizacije koje se koriste u dubinskom strojnom učenju te eksperimentalno ispitati njihov utjecaj na generalizacijsku pogrešku modela. Eksperimentima je pokazano da se odabirom prikladnih metoda regularizacije za dani problem performanse modela na novim podatcima mogu znatno poboljšati., One of the major challenges of training deep neural networks with millions of parameters is avoiding overfitting which results in bad generalization on new data. There are two main approaches to problem of overfitting, collecting more training data and regularization. The main objective of this master’s thesis is to give an overview of different regularization methods and experimentally examine their effect on generalization error. Experiments showed that model performance on new data can be greately improved by choosing appropriate regularization techniques for a given problem.

With the advance of many image manipulation tools, carrying out image forgery and concealing the forgery is becoming easier. In this paper, the convolution neural network (CNN) innovation for image forgery detection and localization is discussed. A novel image forgery detection model using AlexNet framework is introduced. We proposed a modified model to optimize the AlexNet model by using batch normalization instead of local Response normalization, a maxout activation function instead of a rectified linear unit, and a softmax activation function in the last layer to act as a classifier. As a consequence, the AlexNet proposed model can carry out feature extraction and as well as detection of forgeries without the need for further manipulations. Throughout a number of experiments, we examine and differentiate the impacts of several important AlexNet design choices. The proposed networks model is applied on CASIA v2.0, CASIA v1.0, DVMM, and NIST Nimble Challenge 2017 datasets. We also apply k-fold cross-validation on datasets to divide them into training and test data samples. The experimental results achieved prove that the proposed model can accomplish a great performance for detecting different sorts of forgeries. Quantitative performance analysis of the proposed model can detect image forgeries with 98.176% accuracy.

In this paper, we explore the effect of using different convolutional layers, batch normalization and the global average pooling layer upon a convolutional neural network (CNN) based gaze tracking system. A novel method is proposed to label the participant&rsquo, s face images as gaze points retrieved from eye tracker while watching videos for building a training dataset that is closer to human visual behavior. The participants can swing their head freely, therefore, the most real and natural images can be obtained without too many restrictions. The labeled data are classified according to the coordinate of gaze and area of interest on the screen. Therefore, varied network architectures are applied to estimate and compare the effects including the number of convolutional layers, batch normalization (BN) and the global average pooling (GAP) layer instead of the fully connected layer. Three schemes, including the single eye image, double eyes image and facial image, with data augmentation are used to feed into neural network to train and evaluate the efficiency. The input image of the eye or face for an eye tracking system is mostly a small-sized image with relatively few features. The results show that BN and GAP are helpful in overcoming the problem to train models and in reducing the amount of network parameters. It is shown that the accuracy is significantly improved when using GAP and BN at the mean time. Overall, the face scheme has a highest accuracy of 0.883 when BN and GAP are used at the mean time. Additionally, comparing to the fully connected layer set to 512 cases, the number of parameters is reduced by less than 50% and the accuracy is improved by about 2%. A detection accuracy comparison of our model with the existing George and Routray methods shows that our proposed method achieves better prediction accuracy of more than 6%.

Diabetes is a serious threat to health development, because diabetes is a disease that caused most other diseases (complications). Diabetic retinopathy is the most important manifestation of diabetic microangiopathy and is also one of the most common complications in people with diabetes. At present, the diagnosis of diabetic retinal complications mainly depends on the pictures for diagnosis. The fundus images are the main ways to diagnose retinal diseases at present, but the diagnosis process is complicated. Based on this, this paper uses the electronic medical record information of 301 hospitalized patients with diabetes from 2009 to 2013, mainly using the diabetes diagnostic data, diabetes glycosylation data and diabetes biochemical test data, the depth of learning methods and medical diabetes combined with the application of convolution Neural Network Method (CNN) to build a diagnostic model, and thus draw the diagnosis. The main contribution of this study is twofold: 1) In this paper, we apply the CNN method to one-dimensional unrelated data sets and solve the problem of how to do one-dimensional irrelevant data convolution. 2) In this paper, the CNN model is combined with the BN layer to prevent the dispersion of the gradient, speed up the training speed and improve the accuracy of the model. In addition, this model incorporates an adaptive learning rate algorithm and optimizes the model. The experiments show that this method can achieve a training accuracy of 99.85% and a testing accuracy of 97.56%, which is more than 2% higher than that of using logistic regression. The model methods involved in this study can not only be used for the diagnosis of diabetic retinopathy, but also for the diagnosis of other diseases, such as chronic kidney disease, cardiovascular, and cerebrovascular diseases.

Recent research works have highlight the invariance or the symmetry that exists in the weight space of a typical neural network and the negative effect of the symmetry on the training neural network due to the Euclidean gradient being not scaling-invariant. Although the problem of the symmetry can be solved by either defining a suitable Riemannian gradient, which is a scale-invariant or placing appropriate constraints on the weights, it will introduce very high-computation cost. In this paper, we first discuss various invariances or symmetries in the weight space, and then we propose to solve the problem via the scaling-invariance of the neural network itself, instead of the scaling-invariant updates methods. The motivation behind our method is that the optimized parameter point in the weight space may be moved from the ill-conditioning region to the flat region via a series of node-wise rescaling without changing the function represented by the neural network. Second, we proposed the scaling-based weight normalization. The proposed method can compatible with the commonly used optimization algorithms and collaborates well with batch normalization. Although our algorithm is very simple, it can accelerate the convergence speed. The additional computation cost introduced by our method is lower. Lastly, the experiments show that our proposed method can improve the performance of various networks architectures over large-scale datasets consistently. Our method outperforms the state-of-the-art methods on CIFAR-100: we obtain test errors as 17.18%.

Median filtering (MF), a non-linear smoothing operation, has often been utilized as a means of image denoising to protect image edges and hide the traces of image tampering. Although the existing MF forensics methods have achieved excellent performance without post-processing, there is still a challenge of detecting MF in small image size and JPEG compression scenario. To meet this challenge, a robust MF forensic method using convolutional-neural-network (CNN)-based multiple residuals learning is proposed in this paper. Firstly, to reveal the traces left by MF, we use multiple high-pass filters to initialize the weights of the pre-processing layer, and obtain discriminative residuals to characterize MF artifacts in various aspects. Then the output of the pre-processing layer is employed as the input of CNN, which is elaborately designed to extract rich hierarchical features for further classification. Furthermore, Batch Normalization (BN) is introduced as a regularization method to help accelerate convergence of the entire network. The extensive experimental results on the composite database demonstrate that the proposed method is superior to the state-of-the-art methods when detecting MF in both JPEG compressed and small size images.

This paper addressed the vessel segmentation and disease diagnostic in coronary angiography image and proposed an Encoder-Decoder architecture of deep learning with End-to-End model, where Encoder is based on ResNet, and the deep features are exacted automatically, and the Decoder produces the segmentation result by balanced cross-entropy cost function. Furthermore, batch normalization is employed to decrease the gradient vanishing in the training process, so as to reduce the difficulty of training the deep neural network. The experiment results show that the algorithm effectively exacts the feature and edge information, therefore the complex background disturbance is suppressed convincingly, and the vessel segmentation precision is improved effectively, the segmentation precision for three typical vessels are 0.8365, 0.8924 and 0.6297 respectively; and the F-measure are 0.8514, 0.8786 and 0.7298, respectively. In addition, the experiment results show that our proposed can be generalized to the angiography image within limits.

By analyzing the recovery and reconstruction process of various power quality single disturbances and composite disturbance signals, we proposed a set of acquisition methods suitable for power quality disturbance (PQD) signals. The proposed acquisition method is applied to the compression sensing (CS) technology for data compression, the demand for the acquisition device memory is reduced, and the transmission rate is increased. An end-to-end intelligent classification framework is designed, which can directly classify the collected data without any time-consuming data pre-processing operations. The model is designed with noise adaptation module, which can cope with the error of compressed sensing recovery and has also showed good classification performance in noise data. Simultaneously, the model applies a lot of easy-to-implement techniques, which makes the trained model have better generalization ability and classification effect. The proposed method is verified by both simulation and measured data. The method showed superior performance compared to the existing disturbance identification methods based on the classification results.

In order to improve the problem of low accuracy in human behavior recognition task, a neural network based on batch normalization convolution neural network(CNN) and long short-term memory(LSTM) neural network is proposed. The CNN part introduces the idea of batch normalization, and the training data of the input network are normalized in mini-batch. After full connection, they are sent to long short-term memory neural network. The algorithm adopts the space-time dual stream network model structure. The RGB image of video data is taken as spatial stream network input, and the optical flow field image is taken as time flow network input. Then the recognition results obtained by the time-space dual-stream network are combined in a certain proportion to obtain the final behavior recognition result. The experimental results show that the space-time dual stream neural network algorithm designed in this paper has a high recognition accuracy in human behavior recognition tasks.

The novel Coronavirus named COVID-19 that World Health Organization (WHO) announced as a pandemic rapidly spread worldwide. Fast diagnosis of the virus infection is critical to prevent further spread of the virus, help identify the infected population, and cure the patients. Due to the increasing rate of infection and the limitations of the diagnosis kit, auxiliary detection tools are needed. Recent studies show that a deep learning model that comes up with the salient information of CT images can aid in the COVID-19 diagnosis. This study proposes a novel deep learning structure that the pooling layer of this model is a combination of pooling and the Squeeze Excitation Block (SE-block) layer. The proposed model uses Batch Normalization and Mish Function to optimize convergence time and performance of COVID-19 diagnosis. A dataset of two public hospitals was used to evaluate the proposed model. Moreover, it was compared to some different popular deep neural networks (DNN). The results expressed an accuracy of 99.03 with a recognition time of test mode of 0.069 ms in graphics processing unit (GPU). Furthermore, the best network results in classification metrics parameters and real-time applications belong to the proposed model.

Alcoholism is attributed to regular or excessive drinking of alcohol and leads to the disturbance of the neuronal system in the human brain. This results in certain malfunctioning of neurons that can be detected by an electroencephalogram (EEG) using several electrodes on a human skull at appropriate positions. It is of great interest to be able to classify an EEG activity as that of a normal person or an alcoholic person using data from the minimum possible electrodes (or channels). Due to the complex nature of EEG signals, accurate classification of alcoholism using only a small dataset is a challenging task. Artificial neural networks, specifically convolutional neural networks (CNNs), provide efficient and accurate results in various pattern-based classification problems. In this work, we apply CNN on raw EEG data and demonstrate how we achieved 98% average accuracy by optimizing a baseline CNN model and outperforming its results in a range of performance evaluation metrics on the University of California at Irvine Machine Learning (UCI-ML) EEG dataset. This article explains the stepwise improvement of the baseline model using the dropout, batch normalization, and kernel regularization techniques and provides a comparison of the two models that can be beneficial for aspiring practitioners who aim to develop similar classification models in CNN. A performance comparison is also provided with other approaches using the same dataset.

Previous studies on Artificial Neural Network (ANN)-based automatic berthing showed considerable increases in performance by training ANNs with a set of berthing datasets. However, the berthing performance deteriorated when an extrapolated initial position was given. To overcome the extrapolation problem and improve the training performance, recent developments in Deep Learning (DL) are adopted in this paper. Recent activation functions, weight initialization methods, input data-scaling methods, a higher number of hidden layers, and Batch Normalization (BN) are considered, and their effectiveness has been analyzed based on loss functions, berthing performance histories, and berthing trajectories. Finally, it is shown that the use of recent activation and weight initialization method results in faster training convergence and a higher number of hidden layers. This leads to a better berthing performance over the training dataset. It is found that application of the BN can overcome the extrapolated initial position problem.

Spiking Neural Networks (SNNs) have recently emerged as an alternative to deep learning owing to sparse, asynchronous and binary event (or spike) driven processing, that can yield huge energy efficiency benefits on neuromorphic hardware. However, training high-accuracy and low-latency SNNs from scratch suffers from non-differentiable nature of a spiking neuron. To address this training issue in SNNs, we revisit batch normalization and propose a temporal Batch Normalization Through Time (BNTT) technique. Most prior SNN works till now have disregarded batch normalization deeming it ineffective for training temporal SNNs. Different from previous works, our proposed BNTT decouples the parameters in a BNTT layer along the time axis to capture the temporal dynamics of spikes. The temporally evolving learnable parameters in BNTT allow a neuron to control its spike rate through different time-steps, enabling low-latency and low-energy training from scratch. We conduct experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and event-driven DVS-CIFAR10 datasets. BNTT allows us to train deep SNN architectures from scratch, for the first time, on complex datasets with just few 25-30 time-steps. We also propose an early exit algorithm using the distribution of parameters in BNTT to reduce the latency at inference, that further improves the energy-efficiency., Comment: Accepted to Frontiers in Neuroscience (2021)

This work presents a novel deep learning architecture called BNU-Net for the purpose of cardiac segmentation based on short-axis MRI images. Its name is derived from the Batch Normalized (BN) U-Net architecture for medical image segmentation. New generations of deep neural networks (NN) are called convolutional NN (CNN). CNNs like U-Net have been widely used for image classification tasks. CNNs are supervised training models which are trained to learn hierarchies of features automatically and robustly perform classification. Our architecture consists of an encoding path for feature extraction and a decoding path that enables precise localization. We compare this approach with a parallel approach named U-Net. Both BNU-Net and U-Net are cardiac segmentation approaches: while BNU-Net employs batch normalization to the results of each convolutional layer and applies an exponential linear unit (ELU) approach that operates as activation function, U-Net does not apply batch normalization and is based on Rectified Linear Units (ReLU). The presented work (i) facilitates various image preprocessing techniques, which includes affine transformations and elastic deformations, and (ii) segments the preprocessed images using the new deep learning architecture. We evaluate our approach on a dataset containing 805 MRI images from 45 patients. The experimental results reveal that our approach accomplishes comparable or better performance than other state-of-the-art approaches in terms of the Dice coefficient and the average perpendicular distance.

Research in human activity recognition (HAR) benefits many applications such as intelligent surveillance systems to track humans’ abnormal activities. It could also be applied to robots to understand human activity, which improves smart home efficiency and usability. This chapter aims to accurately recognize different sports types in the Sports Video in the Wild (SVW) dataset employing transfer learning. The dataset consists of noisy and similar classes shot in daily environments, not in controlled laboratory environments. Heretofore, different methods have been used and developed for this purpose. Transfer learning is the process of using pre-trained neural networks. The experimental results on different splits of the dataset, size, and pre-trained models show that accuracy of 80.7% is achievable. In another experiment, we have used the famous UCF101 dataset which is collected from YouTube and trained a convolutional neural network (CNN) with Batch Normalization (BN). The achieved accuracy for the test dataset is around 91.2%. One application of the proposed system is to integrate it with a smart home platform to identify sports activities of individuals and track their progress.

U završnom radu naveden je napredak algoritama za detekciju lica u zadnja tri desetljeća. Opisuju se postupci predobrade digitalnih slika koji se poduzimaju prije same detekcije karakterističnih točaka na licima. Navedena je arhitektura slojeva konvolucijske neuronske mreže koja se koristi za predviđanje koordinata karakterističnih točaka. Također, opisana je baza podataka slika koje se koriste u svrhe treniranja i testiranja mreže. Primjeri modela mreže implementirani su bibliotekom Keras u programskom jeziku Python u alatu Colaboratory. Opisana je funkcija gubitka i vrsta optimizacije koje modeli koriste. Objašnjene su promjene napravljene na originalnom modelu te su uspoređeni rezultati točnosti i brzine pojedinih modela. This bachelor's thesis describes the progress of face detection algorithms over the past three decades. Image preprocessing procedures performed prior to the detection of facial feature points are described. Layered architecture of the convolutional neural network for facial feature point detection is explained. Moreover, the image database used to train and test the neural network is described. Examples of neural network models are implemented using the Python library Keras and run on the Colaboratory platform. A brief description of loss functions and optimizers used by the models is also given. The changes applied to the original model are explained and their accuracy and speed results mutually compared.

Batch-normalization (BN) layers are thought to be an integrally important layer type in today's state-of-the-art deep convolutional neural networks for computer vision tasks such as classification and detection. However, BN layers introduce complexity and computational overheads that are highly undesirable for training and/or inference on low-power custom hardware implementations of real-time embedded vision systems such as UAVs, robots and Internet of Things (IoT) devices. They are also problematic when batch sizes need to be very small during training, and innovations such as residual connections introduced more recently than BN layers could potentially have lessened their impact. In this paper we aim to quantify the benefits BN layers offer in image classification networks, in comparison with alternative choices. In particular, we study networks that use shifted-ReLU layers instead of BN layers. We found, following experiments with wide residual networks applied to the ImageNet, CIFAR 10 and CIFAR 100 image classification datasets, that BN layers do not consistently offer a significant advantage. We found that the accuracy margin offered by BN layers depends on the data set, the network size, and the bit-depth of weights. We conclude that in situations where BN layers are undesirable due to speed, memory or complexity costs, that using shifted-ReLU layers instead should be considered; we found they can offer advantages in all these areas, and often do not impose a significant accuracy cost., Comment: 8 pages, published IEEE conference paper

It is challenging to build and train a Convolutional Neural Network model that can achieve a high accuracy rate for the first time. There are many variables to consider such as initial parameters, learning rate, and batch size. Unsuccessfully training a model is one of the most inevitable problems. In some cases, the model struggles to find a lower Loss Function value which results in a poor performance. Batch Normalization is considered as a remedy to overcome this problem. In this paper, two models reinvented from VGG16 are created with and without using Batch Normalization to evaluate their model performance. It is clear that the model using Batch Normalization provides a better result in terms of Loss Function value and model accuracy, which also achieves a very high accuracy rate. It also reaches the saturation point of the highest model accuracy faster than the model without Batch Normalization. This paper also finds that the accuracy of 3D Convolutional Neural Network model reinvented from VGG16 with Batch Normalization is at 91.2% which can beat many benchmarking results on UCF101 such as IDT [6], Two-Stream [11], and Dynamic Image Networks IDT [4]. The technique introduced in this paper shows a fast, reliable and accurate estimation of human activity type and could be used in smart environments.