Search

The number of papers on machine learning and deep neural networks applied to intrusion detection systems (IDS) is ever-increasing. Differently from existing work on the topic, this paper explores the effect of training-time randomness of deep neural networks, which is overlooked by the related literature. Training-time randomness is regulated by the seed of the pseudorandom number generator, and affects the performance of IDS models. The seed selection is studied in conjunction with other critical learning parameters: to the best of our knowledge, there are no similar studies in IDS. The experiments are done with a recent and widely consolidated intrusion detection benchmark, which is used to train and test a neural network under different combinations of seeds and parameters both in supervised and semi-supervised learning modes. The results are inferred by a mixture of explorative analysis, design of experiments, and analysis of variance. According to the results, the choice of the seed yields either excellent or scarce detection metrics; more importantly, the seed selection might be as relevant as the other major learning parameters assessed. [ABSTRACT FROM AUTHOR]

One of the predominant health issues affecting Saudi Arabia and leading to many complications is Type 2 diabetes (T2D). Early detection and significant preventative measures lead to curbing and controlling the health issue. There are fewer datasets in the literature for the detection of T2D in the Saudi population. Past studies using Saudi data have favoured machine learning algorithms to classify T2D. Although the application of this data in machine learning is evident, no studies exist in the literature that compare this data, especially those related to deep learning algorithms. This study's objective is to use specific Saudi data to develop multiple deep learning models that could be used to detect T2D. The research uses a Deep Neural Network (DNN), an Autoencoder (AE), and a Convolutional Neural Network (CNN) to create predictive models and compare their performance with a traditional machine learning classifier used on the same dataset that outperformed other machine learning algorithms such as a Decision Forest (DF). Various metrics were used to evaluate the effectiveness of the models, such as accuracy, precision, recall, F1 score and area under the ROC curve (AUC) where the ROC acts as a receiver operating characteristic curve. There are two cases in this paper: (i) uses all features of the dataset and (ii) uses six of the ten features, such as DF. In case (i), the results were shown that AE outperformed other models with the highest accuracy for imbalanced and balanced data 81.12% and 79.16%, respectively. The results for case (ii) showed that AE scored the highest 81.01% accuracy with imbalanced data compared to DF and DF achieved the highest accuracy of 82.1% with balanced data. As a result, both cases explored in this study revealed that AE has a constant superior performance if imbalanced data is used. In contrast, DF demonstrated the highest accuracy when a balanced dataset was used with a feature set reduction. They help to identify the undiagnosed T2D, and they are essential for professionals in Saudi Arabia in the health sector to promote health connections, identify risks and contain or improve their diabetes management. [ABSTRACT FROM AUTHOR]

Background: The effectiveness of in vitro fertilization depends on the assessment and selection of oocytes and embryos with the highest developmental potential. One of the tasks in the ICSI (intracytoplasmic sperm injection) procedure is the classification of oocytes according to the stages of their meiotic maturity. Oocytes classification traditionally is done manually during their observation under the light microscope. The paper is part of the bigger task, the development of the system for optimal oocyte and embryos selection. In the hereby work, we present the method for the automatic classification of oocytes based on their images, that employs DNN algorithms. Results: For the purpose of oocyte class determination, two structures based on deep neural networks were applied. DeepLabV3Plus was responsible for the analysis of oocyte images in order to extract specific regions of oocyte images. Then extracted components were transferred to the network, inspired by the SqueezeNet architecture, for the purpose of oocyte type classification. The structure of this network was refined by a genetic algorithm in order to improve generalization abilities as well as reduce the network's FLOPs thus minimizing inference time. As a result, Acc ¯ at the level of 0.964 was obtained at the level of the validation set and 0.957 at the level of the test set. Generated neural networks as well as code that allows running the processing pipe were made publicly available. Conclusions: In this paper, the complete pipeline was proposed that is able to automatically classify human oocytes into three classes MI, MII, and PI based on the oocytes' microscopic image. [ABSTRACT FROM AUTHOR]

The issue of climate change holds immense significance, affecting various aspects of life, including the environment, the interaction between soil conditions and the atmosphere, and agriculture. Over the past few decades, a range of spatio-temporal and Deep Neural Network (DNN) techniques had been proposed within the field of Machine Learning (ML) for climate forecasting, using spatial and temporal data. The forecasting model in this paper is highly complex, particularly due to the presence of nonlinear data in the residual modeling of General Space-Time Autoregressive Integrated Moving Average (GSTARIMA), which represented nonstationary data with time and location dependencies. This model effectively captured trends and seasonal data with time and location dependencies. On the other hand, DNNs proved reliable for modeling nonlinear data that posed challenges for spatio-temporal approaches. This research presented a comprehensive overview of the integrated approach between the GSTARIMA model and DNNs, following the six-stage Data Analytics Lifecycle methodology. The focus was primarily on previous works conducted between 2013 and 2022. The review showed that the GSTARIMA–DNN integration model was a promising tool for forecasting climate in a specific region in the future. Although spatio-temporal and DNN approaches have been widely employed for predicting the climate and its impact on human life due to their computational efficiency and ability to handle complex problems, the proposed method is expected to be universally accepted for integrating these models, which encompass location and time dependencies. Furthermore, it was found that the GSTARIMA–DNN method, incorporating multivariate variables, locations, and multiple hidden layers, was suitable for short-term climate forecasting. Finally, this paper presented several future directions and recommendations for further research. [ABSTRACT FROM AUTHOR]

A high cognitive load can overload a person, potentially resulting in catastrophic accidents. It is therefore important to ensure the level of cognitive load associated with safety-critical tasks (such as driving a vehicle) remains manageable for drivers, enabling them to respond appropriately to changes in the driving environment. Although electroencephalography (EEG) has attracted significant interest in cognitive load research, few studies have used EEG to investigate cognitive load in the context of driving. This paper presents a feasibility study on the simulation of various levels of cognitive load through designing and implementing four driving tasks. We employ machine learning-based classification techniques using EEG recordings to differentiate driving conditions. An EEG dataset containing these four driving tasks from a group of 20 participants was collected to investigate whether EEG can be used as an indicator of changes in cognitive load. The collected dataset was used to train four Deep Neural Networks and four Support Vector Machine classification models. The results showed that the best model achieved a classification accuracy of 90.37%, utilising statistical features from multiple frequency bands in 24 EEG channels. Furthermore, the Gamma and Beta bands achieved higher classification accuracy than the Alpha and Theta bands during the analysis. The outcomes of this study have the potential to enhance the Human–Machine Interface of vehicles, contributing to improved safety. [ABSTRACT FROM AUTHOR]

Nucleic-acid G-quadruplexes (G4s) play vital roles in many cellular processes. Due to their importance, researchers have developed experimental assays to measure nucleic-acid G4s in high throughput. The generated high-throughput datasets gave rise to unique opportunities to develop machine-learning-based methods, and in particular deep neural networks, to predict G4s in any given nucleic-acid sequence and any species. In this paper, we review the success stories of deep-neural-network applications for G4 prediction. We first cover the experimental technologies that generated the most comprehensive nucleic-acid G4 high-throughput datasets in recent years. We then review classic rule-based methods for G4 prediction. We proceed by reviewing the major machine-learning and deep-neural-network applications to nucleic-acid G4 datasets and report a novel comparison between them. Next, we present the interpretability techniques used on the trained neural networks to learn key molecular principles underlying nucleic-acid G4 folding. As a new result, we calculate the overlap between measured DNA and RNA G4s and compare the performance of DNA- and RNA-G4 predictors on RNA- and DNA-G4 datasets, respectively, to demonstrate the potential of transfer learning from DNA G4s to RNA G4s. Last, we conclude with open questions in the field of nucleic-acid G4 prediction and computational modeling. [ABSTRACT FROM AUTHOR]

A major issue in the application of deep learning is the definition of a proper architecture for the learning machine at hand, in such a way that the model is neither excessively large (which results in overfitting the training data) nor too small (which limits the learning and modeling capabilities of the automatic learner). Facing this issue boosted the development of algorithms for automatically growing and pruning the architectures as part of the learning process. The paper introduces a novel approach to growing the architecture of deep neural networks, called downward-growing neural network (DGNN). The approach can be applied to arbitrary feed-forward deep neural networks. Groups of neurons that negatively affect the performance of the network are selected and grown with the aim of improving the learning and generalization capabilities of the resulting machine. The growing process is realized via replacement of these groups of neurons with sub-networks that are trained relying on ad hoc target propagation techniques. In so doing, the growth process takes place simultaneously in both the depth and width of the DGNN architecture. We assess empirically the effectiveness of the DGNN on several UCI datasets, where the DGNN significantly improves the average accuracy over a range of established deep neural network approaches and over two popular growing algorithms, namely, the AdaNet and the cascade correlation neural network. [ABSTRACT FROM AUTHOR]

Data-driven intelligent surrogate models gain popularity recently. Particularly in Monte-Carlo-style stochastic analysis, the influencing factors are considered as inputs, the quantities of interest are considered as outputs, and cheaper-to-evaluate surrogates models are built from a small amount of sample data and are used for the full Monte-Carlo analysis. This paper presents a framework with three innovations: (1) we build surrogate models for a particular problem that covers any possible material properties or boundary conditions commonly encountered in practice, so the models are ready to use, and do not require new data or training anymore. (2) The inputs and outputs to the problem are both spatially variable. Even after discretization, the input and output sizes are in the order of tens of thousands, which is challenging for traditional machine-learning algorithms. We take the footing failure mechanism as an example. Two types of neural networks are examined, fully connected networks and deep neural networks with complicated non-sequential structures (a modified U-Net). (3) This study is also the first attempt to use U-Nets as surrogate models for geotechnical problems. Results show that fully connected networks can fit well simple problems with a small input and output size, but fail for complex problems. Deep neural networks that account for the data structure give better results. [ABSTRACT FROM AUTHOR]

Deep neural networks have achieved great success in solving many machine learning and computer vision problems. In this paper, we propose a deep neural network called the Tucker network derived from the Tucker format and analyze its expressive power. The results demonstrate that the Tucker network has exponentially higher expressive power than the shallow network. In other words, a shallow network with an exponential width is required to realize the same score function as that computed by the Tucker network. Moreover, we discuss the expressive power between the hierarchical Tucker tensor network (HT network) and the proposed Tucker network. To generalize the Tucker network into a deep version, we combine the hierarchical Tucker format and Tucker format to propose a deep Tucker tensor decomposition. Its corresponding deep Tucker network is presented. Experiments are conducted on three datasets: MNIST, CIFAR-10 and CIFAR-100. The results experimentally validate the theoretical results and show that the Tucker network and deep Tucker network have better performance than the shallow network and HT network. [ABSTRACT FROM AUTHOR]

The explosive growth of geospatial data is increasing requirements for automatic and efficient data learning abilities. Many deep learning methods have been widely applied for geospatial data understanding tasks, such as road networks and geospatial object detection. However, the demands for more accurate learning of high-level features require the use of deeper neural networks. To further improve the learning efficiency of deep neural networks, in this paper, we propose an improved convolutional neural network named EDense. First, we use its dense connectivity to integrate a CNN with an extreme learning machine. Then, we expand the kernels in the convolutional layers to increase the width of the network model. Furthermore, we propose one-feature EDense (OF-EDense), which is a simplified version of EDense, to fit conditions in which the number of parameters is strictly limited. Finally, the experimental results fully demonstrate the strong learning ability and high learning efficiency of EDense. [ABSTRACT FROM AUTHOR]

Aim/Purpose This paper aims to analyze the availability and pricing of perishable farm produce before and during the lockdown restrictions imposed due to Covid-19. This paper also proposes machine learning and deep learning models to help the farmers decide on an appropriate market to sell their farm produce and get a fair price for their product. Background Developing countries like India have regulated agricultural markets governed by country-specific protective laws like the Essential Commodities Act and the Agricultural Produce Market Committee (APMC) Act. These regulations restrict the sale of agricultural produce to a predefined set of local markets. Covid-19 pandemic led to a lockdown during the first half of 2020 which resulted in supply disruption and demand-supply mismatch of agricultural commodities at these local markets. These demand-supply dynamics led to disruptions in the pricing of the farm produce leading to a lower price realization for farmers. Hence it is essential to analyze the impact of this disruption on the pricing of farm produce at a granular level. Moreover, the farmers need a tool that guides them with the most suitable market/city/town to sell their farm produce to get a fair price. Methodology One hundred and fifty thousand samples from the agricultural dataset, released by the Government of India, were used to perform statistical analysis and identify the supply disruptions as well as price disruptions of perishable agricultural produce. In addition, more than seventeen thousand samples were used to implement and train machine learning and deep learning models that can predict and guide the farmers about the appropriate market to sell their farm produce. In essence, the paper uses descriptive analytics to analyze the impact of COVID-19 on agricultural produce pricing. The paper explores the usage of prescriptive analytics to recommend an appropriate market to sell agricultural produce. Contribution Five machine learning models based on Logistic Regression, K-Nearest Neighbors, Support Vector Machine, Random Forest, and Gradient Boosting, and three deep learning models based on Artificial Neural Networks were implemented. The performance of these models was compared using metrics like Precision, Recall, Accuracy, and F1-Score. Findings Among the five classification models, the Gradient Boosting classifier was the optimal classifier that achieved precision, recall, accuracy, and F1 score of 99%. Out of the three deep learning models, the Adam optimizer-based deep neural network achieved precision, recall, accuracy, and F1 score of 99%. Recommendations for Practitioners Gradient boosting technique and Adam-based deep learning model should be the preferred choice for analyzing agricultural pricing-related problems. Recommendations for Researchers Ensemble learning techniques like Random Forest and Gradient boosting perform better than non-Ensemble classification techniques. Hyperparameter tuning is an essential step in developing these models and it improves the performance of the model. Impact on Society Statistical analysis of the data revealed the true nature of demand and supply and price disruption. This analysis helps to assess the revenue impact borne by the farmers due to Covid-19. The machine learning and deep learning models help the farmers to get a better price for their crops. Though the dataset used in this paper is related to India, the outcome of this research work applies to many developing countries that have similar regulated markets. Hence farmers from developing countries across the world can benefit from the outcome of this research work. Future Research The machine learning and deep learning models were implemented and tested for markets in and around Bangalore. The model can be expanded to cover other markets within India. [ABSTRACT FROM AUTHOR]

Social media platforms such as Twitter are a vital source of information during major events, such as natural disasters. Studies attempting to automatically detect textual communications have mostly focused on machine learning and deep learning algorithms. Recent evidence shows improvement in disaster detection models with the use of contextual word embedding techniques (i.e., transformers) that take the context of a word into consideration, unlike the traditional context-free techniques; however, studies regarding this model are scant. To this end, this paper investigates a selection of ensemble learning models by merging transformers with deep neural network algorithms to assess their performance in detecting informative and non-informative disaster-related Twitter communications. A total of 7613 tweets were used to train and test the models. Results indicate that the ensemble models consistently yield good performance results, with F-score values ranging between 76% and 80%. Simpler transformer variants, such as ELECTRA and Talking-Heads Attention, yielded comparable and superior results compared to the computationally expensive BERT, with F-scores ranging from 80% to 84%, especially when merged with Bi-LSTM. Our findings show that the newer and simpler transformers can be used effectively, with less computational costs, in detecting disaster-related Twitter communications. [ABSTRACT FROM AUTHOR]

Fully connected deep neural networks (DNN) often include redundant weights leading to overfitting and high memory requirements. Additionally, in tabular data classification, DNNs are challenged by the often superior performance of traditional machine learning models. This paper proposes periodic perturbations (prune and regrow) of DNN weights, especially at the self-supervised pre-training stage of deep autoencoders. The proposed weight perturbation strategy outperforms dropout learning or weight regularization (L1 or L2) for four out of six tabular data sets in downstream classification tasks. Unlike dropout learning, the proposed weight perturbation routine additionally achieves 15% to 40% sparsity across six tabular data sets, resulting in compressed pretrained models. The proposed pretrained model compression improves the accuracy of downstream classification, unlike traditional weight pruning methods that trade off performance for model compression. Our experiments reveal that a pretrained deep autoencoder with weight perturbation can outperform traditional machine learning in tabular data classification, whereas baseline fully-connected DNNs yield the worst classification accuracy. However, traditional machine learning models are superior to any deep model when a tabular data set contains uncorrelated variables. Therefore, the performance of deep models with tabular data is contingent on the types and statistics of constituent variables. [ABSTRACT FROM AUTHOR]

In today's world, a Network Intrusion Detection System (NIDS) plays a vital role in order to secure the Wireless Sensor Network (WSN). However, the traditional NIDS model faced critical constraints with network traffic data due to growth in the complexity of modern attacks. These constraints have a direct impact on the overall performance of the WSN. In this paper, a new robust network intrusion classification framework based on the enhanced Visual Geometry Group (VGG-19) pre-trained model has been proposed to prolong the performance of WSN. Primarily, the pre-trained weights from the ImageNet dataset are utilized to train the parameters of the VGG-19. Afterward, a Hybrid Deep Neural Network based on Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) will be employed to extract the influential features from network traffic data to enlarge the intrusion detection accuracy. The proposed VGG-19 + Hybrid CNN-LSTM model exploits both binary classification and multi-classification to classify attacks as either normal or attacked. A network intrusion benchmark dataset is used to assess the performance of the suggested system. The results reveal that the proposed VGG-19 + Hybrid CNN-LSTM learning system surpasses other pre-trained models with a superior accuracy of 98.86% during the multi-classification test. [ABSTRACT FROM AUTHOR]

Latent fingerprint recognition plays an essential role for law enforcement agencies to detect criminals and security purposes. One of the key stages utilized in the latent fingerprint recognition model is to automatically learn consistent minutiae from fingerprint images. However, the existing state-of-the-art recognition approaches are not adequate since live-scan fingerprint images and enhancements are necessary for each step of the recognition process. Hence, an automated recognition system along with appropriate minutiae learning algorithm is required for matching the latent fingerprint exactly. In this paper, an efficient recognition system using dictionary learning and Local Context-Perception deep neural network (LCPnet) has been proposed to enhance the accuracy of latent fingerprint recognition. Primarily, the Total Variation decomposition model is utilized to remove the smooth background noise and dictionary learning contributes to the extraction of multiple patches. Afterward, the LCPnet is trained for 12 patch types to develop a salient minutiae descriptor where every descriptor is trained using LCPnet with a particular patch size at a location surrounding the minutiae. The proposed detection system has been tested through two latent public datasets. Here, three different types of templates (LCPnet minutiae, LCPnet texture, and LCPnet minutiae+texture) are analyzed for evaluating the proposed fingerprint detection system. The performance results manifest that the proposed system acquires a superior recognition accuracy of 99.44% and 99.58% under two different datasets. [ABSTRACT FROM AUTHOR]

In this paper, a tracking algorithm based on the residual neural network model and machine learning is proposed. Compared with the widely used VGG network, the residual neural network has deeper characteristic layers and special additional layer structure, which break the symmetry of the network and reduce the degradation of the neural network. The additional layer and convolution layer are used for feature fusion to represent the target. The multi-features of the object can be captured by using the developed algorithm, so that the accuracy of tracking can be improved in some complex scenarios. In addition, we defined a new measure to calculate the similarity of different image regions and find the optimal matched region. The search area is delimited according to the continuity of the target motion, which improves the real-time performance of tracking. The experimental results illustrate that the proposed algorithm achieved a higher accuracy while taking into account the real time performance, especially in dealing with some complex scenarios such as deformation, rotation changes and background clutters, in comparison with the Multi-Domain Network (MDNet) algorithm based on a convolutional neural network. [ABSTRACT FROM AUTHOR]

Fitness is important in people's lives. Good fitness habits can improve cardiopulmonary capacity, increase concentration, prevent obesity, and effectively reduce the risk of death. Home fitness does not require large equipment but uses dumbbells, yoga mats, and horizontal bars to complete fitness exercises and can effectively avoid contact with people, so it is deeply loved by people. People who work out at home use social media to obtain fitness knowledge, but learning ability is limited. Incomplete fitness is likely to lead to injury, and a cheap, timely, and accurate fitness detection system can reduce the risk of fitness injuries and can effectively improve people's fitness awareness. In the past, many studies have engaged in the detection of fitness movements, among which the detection of fitness movements based on wearable devices, body nodes, and image deep learning has achieved better performance. However, a wearable device cannot detect a variety of fitness movements, may hinder the exercise of the fitness user, and has a high cost. Both body-node-based and image-deep-learning-based methods have lower costs, but each has some drawbacks. Therefore, this paper used a method based on deep transfer learning to establish a fitness database. After that, a deep neural network was trained to detect the type and completeness of fitness movements. We used Yolov4 and Mediapipe to instantly detect fitness movements and stored the 1D fitness signal of movement to build a database. Finally, MLP was used to classify the 1D signal waveform of fitness. In the performance of the classification of fitness movement types, the mAP was 99.71%, accuracy was 98.56%, precision was 97.9%, recall was 98.56%, and the F1-score was 98.23%, which is quite a high performance. In the performance of fitness movement completeness classification, accuracy was 92.84%, precision was 92.85, recall was 92.84%, and the F1-score was 92.83%. The average FPS in detection was 17.5. Experimental results show that our method achieves higher accuracy compared to other methods. [ABSTRACT FROM AUTHOR]

This paper proposes a novel hardware architecture for a Feed-Forward Neural Network (FFNN) with the objective of minimizing the number of execution clock cycles needed for the network's computation. The proposed architecture depends mainly on using two physical layers that are multiplexed and reused during the computation of the FFNN to achieve an efficient parallel design. Two physical layers are designed to handle the computation of different sizes of Neural Networks (NN). The proposed FFNN architecture hardware resources are independent of the NN's number of layers, instead, they depend only on the number of neurons in the largest layer. This versatile architecture serves as an accelerator in Deep Neural Network (DNN) computations as it exploits parallelism by making the two physical layers work in parallel through the computations. The proposed implementation was implemented with 18-bit fixed point representation reaching 200 MHz clock speed on a Spartan7 FPGA. Furthermore, the proposed architecture achieves a lower neuron computation factor compared to previous works in the literature. [Display omitted] [ABSTRACT FROM AUTHOR]

With the significant increase in photovoltaic (PV) electricity generation, more attention has been given to PV power forecasting. Indeed, accurate forecasting allows power grid operators to better schedule and dispatch their assets, such as energy storage systems and reserve. In this paper, a hybrid deep learning model and a convolutional neural network with memory is proposed, to provide intraday (2 h) solar irradiance forecasts using sequentially-collected images from public webcams. The performance of the proposed model is compared to those of a standard time-series forecast models, a linear regression as well as state-of-the-art neural networks. All models are trained and tested using images collected from two webcams on EPFL's campus for just over a year. The results show that the proposed method outperforms all other models and matches the state-of-the-art methodology while providing simplicity of implementation and efficient computation. • A novel method for estimating global horizontal irradiance is proposed. • The method uses images public webcams instead of all-sky cameras. • The method combines images from different cameras to improve forecasts' accuracy. • The method can be extended to include other types of data (meteorological data). • This simple method matches state-of-the-art methods. [ABSTRACT FROM AUTHOR]

A comprehensive systematic study was carried out in order to identify various deep learning methods developed and used for predicting student academic performance. Predicting academic performance allows for the implementation of various preventive and supportive measures earlier in order to improve academic performance and reduce failure and dropout rates. Although machine learning schemes were once popular, deep learning algorithms are now being investigated to solve difficult predictions of student performance in larger datasets with more data attributes. Deep neural network prediction methods with clear modelling and parameter measurements formulated on publicly available and recognised datasets are the focus of the research. Widely used for academic performance prediction, backpropagation algorithms have been trained and tested with various datasets, especially those related to learning management systems (LMS) and massive open online courses (MOOC). The most widely used prediction method appears to be the standard artificial neural network approach. The long-short-term memory (LSTM) approach has been reported to achieve an accuracy of around 87 percent for temporal student performance data. The number of papers that study and improve this method shows that there is a clear rise in deep learning-based academic performance prediction over the last few years. [ABSTRACT FROM AUTHOR]

Forecasting has always been the backbone of planning studies in the power system. With the advent of the restructured power system and also the integration of renewable energy sources, it has become more challenging. Accurate load forecasting leads to a reduction in supply–demand mismatch, whereas solar insolation/wind speed forecasting improves generation side planning. With the rapid growth in computer-based intelligence, self-learning mechanisms together with continuously growing volume of the dataset and affordable cost-effective data storage options, deep learning performance keeps getting better and has made a huge impact to solve the complex problems associated with the power system network. In this paper, we propose a classical long short-term memory neural network model for a combined day-ahead load and solar insolation forecasting of residential consumers equipped with rooftop PV based on weather classification. The effectiveness of the proposed model based on mean absolute error, mean bias error and coefficient of correlation (R-Squared) is compared with that of the standard artificial neural network and convolutional neural network models. [ABSTRACT FROM AUTHOR]

This paper is focused on the application of the machine learning for predicting structural response of ring stiffened cylindrical shells subjected to far-field underwater explosion. Three deep neural network (DNN) models are combined to predict the progressive plastic deformation behaviour of stiffened cylindrical shells subjected to the shock wave produced by a TNT charge. In the DNN regression models, the input variables include the charge mass, stand-off distance, detonation depth, cylindrical shell thickness, stiffener web and flange. These three DNN models composed of multi hidden layers are trained based on the data collection originated from a series of simulations performed by the ABAQUS finite element solver. Grid search is introduced to the hyperparameter tuning for deep learning, improving the prediction performance of machine learning methods. The DNN models are compared to the models trained using another three methods: multiple linear regression, decision tree and k-nearest neighbour. An improved Adam optimisation algorithm is used to increase the accuracy of prediction. The challenges encountered during the predictions are discussed to provide a more comprehensive insight into the advantage of the proposed DNN models. The DNN regression models can well predict the permanent plastic deformation and strain of stiffened cylindrical shells. • Improved deep neural network is applied to predict the impact response of stiffened cylindrical shells. • Various regression algorithms in machine learning are compared. • Grid search method is used for the hyperparameter optimisation. • It is achieved the rapid and accurate assessment of plastic deformation and strain. [ABSTRACT FROM AUTHOR]

This paper introduces a novel deep learning approach for generating fine resolution structures that preserve all the information from the topology optimization (TO). The proposed approach utilizes neural networks (NNs) that map the desired engineering properties to seed for determining optimized structure. The main novelty of this framework is the utilization of parameters such as density and nodal deflections to predict optimized topologies for a wide range of design space. In this research, a three-stage NN framework is employed, wherein the first stage is a feedforward deep neural network. The second stage uses a convolutional neural network (CNN), and the final stage is the post-processing of results obtained from CNN. The first stage maps input design variables vector to output density distribution image. The image output of the first stage and design variables vector inputted to the first network serve as inputs to the second stage, which outputs optimized structure. Finally, post-processing of this optimized structure is done to ensure volume fraction constraint and optimal solution. These structures can be tessellated per the nodal displacement of an FEA solved larger structure. Because of this framework, the proposed approach makes TO computations faster and captures essential physics while using NNs to determine the final structure. The application of the proposed model to three-dimensional problems is also demonstrated. [ABSTRACT FROM AUTHOR]

Accurate time transfer by time of flight measurements via diffuse reflections on passive orbiting space debris targets requires a selection of suitable objects out of a large catalogue of debris items. In this paper, we report on our development of an automatic classification system of space objects based on photometric observations of sun illuminated satellite and debris items from the Mini–Mega TORTORA (MMT) system observation data base by a deep learning algorithm. A deep neural network model based on a convolutional long short-term memory network has been designed to identify four different object categories with a test accuracy of over 85%. The method is also suitable for an automated analysis of the temporal evolution of the orbit motion of specific space objects. • Neural networks exploit a single observable to classify many space objects. • Reliable space object classification is achieved where rule based systems fail. • Satellite or space debris orbit anomalies are automatically detected. • A body of thousands of different objects can be monitored continuously. [ABSTRACT FROM AUTHOR]

This paper addresses the torque and pitch control problems of wind turbines. The main contribution of this work is the development of an innovative reinforcement learning (RL)-based control method targeting wind turbine applications. Our RL-based control framework synergistically combines the advantages of deep neural networks (DNNs) and model predictive control (MPC) technologies. The proposed control strategy is data-driven, adapting to real-time changes in system dynamics and enhancing control performance and robustness. Additionally, the incorporation of an MPC structure within our design improves learning efficiency and reduces the high computational complexity typically found in deep RL algorithms. Specifically, a DNN is designed to approximate the wind turbine dynamics based on a continuously updated dataset composed of state and action measurements taken at specified sampling intervals. The real-time control policy is generated by integrating the online trained DNN into an MPC architecture. The proposed method iteratively updates the DNN and control policy in real-time to optimize performance. As a primary result of this work, the proposed method demonstrates superior robustness and control performance compared to commonly-employed MPC and other baseline wind turbine controllers in the presence of uncertainties and unexpected actuator faults. This effectiveness is showcased through simulations with a high-fidelity wind turbine simulator. • A novel reinforcement learning-based wind turbine control method is developed. • The proposed method is data-driven without relying on analytical turbine models. • This method shows better performance under uncertainties and actuator faults. [ABSTRACT FROM AUTHOR]

Sparse reparameterization in Deep Neural Networks (DNNs) aims to achieve a better tradeoff between the network parameter count and performance. Recently, the lottery ticket hypothesis suppose that excellent sub-networks ("winning tickets") exist in dense randomly-initialized networks. These sparse sub-networks trained from scratch are able to reach the performance of their dense counterparts. Compared with Iterative Magnitude Pruning that relies on pruning strategies, the Continuous Sparsification algorithm learns the "winning tickets" with gradient-based methods, achieving better performance. In this paper, we propose Layer-wise Continuous Sparsification (LCS) scheme for finding sparse sub-networks, in which the parameterized relaxation of step functions used to remove network parameters in each layer is integrated into the DNN loss as an optimization objective. LCS utilizes a family of sigmoid functions to asynchronously filter important per-layer weights throughout training, yielding sparser and better sub-networks. Experiments show that our method surpasses state-of-the-art methods for sparse reparameterization. Additionally, the proposed method can be utilized as a regularization technique to further improve the accuracy of dense networks 1 1 Our code is publicly available at https://github.com/RiyaoDong/LCS.. [ABSTRACT FROM AUTHOR]

[Display omitted] In recent years deep neural networks have been proposed as a lightweight data-driven model to capture high-dimensional, nonlinear physical processes to predict building thermal responses. However, the need of a large amount of data for the training process of deep neural networks clashes with the potential limited data availability in most existing or new buildings. Transfer learning aims to enhance the performance of a target learner exploiting knowledge from related and similar environments. This study conducted a suite of experiments that leveraged 250 data-driven models based on a synthetic dataset of a building archetype to study the influence of data availability, energy efficiency level, occupancy and climate for the transfer process of thermal dynamics. The performance of the transfer learning process was compared against a classical machine learning approach. The results suggest that building thermal dynamics can be effectively transferred under the same climatic conditions, increasing performance when dealing with different occupancy schedules, efficiency levels and low data availability. Furthermore, the paper compares the performance of both transfer learning and machine learning approaches in an online fashion, to support the implementation in real-world deployment. [ABSTRACT FROM AUTHOR]

• We re-investigate the pipeline of fine-grained visual categorization (FGVC) techniques from the view of human visual recognition system, and propose a novel Attention-Shift based Deep Neural Network (AS-DNN) for automatic parts locating and semantic correlation learning. • We propose an end-to-end trainable sub-network structure C sft to simulate the attention-shift process. C sft locates the discriminative regions automatically and encodes and decodes the semantic relations among diverse discriminative parts iteratively. • Comprehensive experiments show that AS-DNN achieves state-of-the-art performances in three widely used challenging datasets. Moreover, the visualization of located discriminative parts proves the robustness of AS-DNN in complex backgrounds and postures. Fine-grained visual categorization (FGVC) has attracted extensive attention in recent years. The general pipeline of current FGVC techniques is to 1) locate the discriminative regions; 2) extract features from each region independently; and 3) feed the integrated features to a classifier. In this paper, we re-investigate the pipeline from the view of human visual recognition mechanisms. The perceiving of discriminative regions is a temporal processing by the human visual system (HVS) via the attention-shift mechanism. However, the existing independent feature extracting and one-pass feeding strategy ignore the inherent semantic relationships among discriminative regions, and thus is improper to model the attention-shift process properly. Therefore, in this paper, we propose a novel end-to-end FGVC network structure named Attention-Shift based Deep Neural Network (AS-DNN) to locate the discriminative regions automatically and encode the semantic correlations iteratively. AS-DNN consists of two channels: 1) the global perception channel C glb and 2) the attention-shift channel C sft , simulating the global perception and the attention-shift mechanism, respectively. Experimental results show that AS-DNN achieves state-of-the-art performances by outperforming both the CNN-based weakly or strongly-supervised FGVC algorithms on several widely-used fine-grained datasets, and the visualization of attention regions exhibit that the proposed method can locate the discriminative regions robustly in complex backgrounds and postures. [ABSTRACT FROM AUTHOR]

The requirements of future aeroengines challenge turbomachinery designs to be quieter, greener, and more efficient; furthermore, they must be developed at considerably reduced cycles and costs. Traditional turbomachinery design systems involve the iterative design–evaluation–improvement process, leading to high computational costs and considerable reliance on expert experience. In this paper, a knowledge-based turbomachinery aerodynamic design system consisting of a design of experiment framework and a two-stage modeling is proposed. In the first phase of the two-stage modeling, a deep neural network (DNN)-enhanced multi-output Gaussian process model is constructed. This model determines the optimal loading distribution from reference designs that are optimized for the desired performance. In the second stage, a DNN model is developed to predict the turbine profile geometry for a given loading distribution. This formulation enables the automatic determination of the optimal turbine geometry without an iterative process. Furthermore, the knowledge in the design system grows with the increasing database. The results indicate that the L 2 relative errors for predicting the optimal pressure coefficient distribution and turbine profile geometry are less than 0.25% and 1.7%, respectively. A design case of a low-pressure turbine is used to validate the entire design system. The results reveal that the turbine geometry generated by the proposed design system outperforms that of the benchmark case in terms of the design and off-design aerodynamic performance. The foregoing observations are finally confirmed through experiments. • A knowledge-based turbine design method is developed based on machine learning. • A new knowledge-based self-expansion design procedure is proposed. • DNN is combined with the multi-output Gaussian process to improve accuracy. • The accuracy of the method is verified by CFD and flat plate test rig. [ABSTRACT FROM AUTHOR]

Illustration: A gradient-free topological design method integrating the material-field series expansion topological parameterization and the deep neural networks is proposed. This optimization method reduces the computational time by 1–4 orders of magnitude compared with the coarse-mesh-based gradient-free methods and can handle up to 200 design parameters in design dimension reduction model. Several designs with high performances are obtained with the present method. [Display omitted] • A novel topology optimization method combining self-adaptive neural network and material-field series expansion is proposed. • The self-adaptive neural network model handles up to 200 design parameters under topological dimensionality reduction. • This method reduces the computational time by 1–4 orders of magnitude compared with the coarse-mesh-based gradient-free methods. • The optimized cellular material effective bulk modulus by the proposed method achieves 98% of the theory bound. Powerful gradient-free topology optimization methods are needed for structural design concerning complex responses. In this paper, a novel gradient-free optimization method is proposed by integrating the material-field series expansion topological parameterization and the deep neural networks, providing two-fold advances: firstly, it generally reduces the massive topological design variables to fewer than 200, while keeps the capability to represent relative complex 3D topologies and clear boundaries; secondly, by constructing a sequential neural network surrogate model, it sufficiently explores the reduced design space and is capable of handling multi-peak and discontinuous optimization problems. The effectiveness of this method is illustrated via several design problems, among which the optimized material effective bulk modulus achieves 98% of the H-S bound and the highly-nonlinear peak weld stress in a phone dropping process is decreased by 16.59%. This method reduces the computational time by 1–4 orders of magnitude compared with the coarse-mesh-based gradient-free methods, and it is the first time to successfully conduct gradient-free 3D topology optimization with thousands of finite elements. The method's ease of implementation and compatibility with various simulation software, brings topology optimization into complex industrial applications and proves that gradient-free technology represents an effective optimization benchmark for improving structural performance. [ABSTRACT FROM AUTHOR]

• Saffron adulterants detected and classified using a deep learning algorithm. • RGB photos took under uncontrolled/unstructured conditions. • Five CNN models were proposed for classification. • The best-proposed model can process an image in 36.26 ms with 99.67% accuracy. Saffron is the most expensive spice in the world which is traded internationally. Hence, Saffron adulterants is always a challenging issue. It should be noted that traditional methods that are being used to solve this problem, are costly and time-consuming. In this study, for the first time, Saffron adulterants was detected using a deep neural network and RGB photos taken under uncontrolled/unstructured conditions. There were five classes of fake Saffron include mixed saffron stamen and dyed straw, mixed stigma and stamens, dyed orange blossom, Safflower and dyed stamens, and two classes of genuine Saffron consisting of freeze-drying and microwave-drying saffron used to design, develop and optimize a Convolutional Neural Network (CNN). 60% of the images were used to train and optimize the neural network model. For preventing over-training, the number of images used in the training phase increased by applying the augmentation method. The general structure of the model to detect saffron adulterants consisted of multiple convolutional (input layer), pooling layers and fully-connected layers, respectively. In this study, five structures (S1, S2, S3, S4 and S5) were proposed for CNN using Batch Normalization methods. Additionally, this paper compared these five structures with Popular Convolutional Networks consisted of VGG11, ResNet18, ResNet50, Inception V3 and DarkNet53. According to the results, the best configuration was S5 that achieved the accuracy of 99.8% and the cross-entropy loss of 0.019. Although the time process of this structure was 36.26 ms, the shortest time process was 19.38 ms related to the S1. This number was more than 300 ms for Popular CNN. Based on the results of this study, imaging without restrictions with CNN can detect Saffron adulterants. [ABSTRACT FROM AUTHOR]

• Data-driven coarse mesh turbulence model based on deep neural networks that can learn from high-resolution CFD data. • The proposed Dense-CNN/LSTM architecture can efficiently learn the spatial-temporal information from transient CFD results. • Good agreement observed between model predictions and testing CFD data on reactor loss-of-flow transient case study. • Evaluated model's generalization capability by exploring intrisic data similarity. Advanced nuclear reactors often exhibit complex thermal-fluid phenomena during transients. To accurately capture such phenomena, a coarse-mesh three-dimensional (3-D) modeling capability is desired for modern nuclear-system code. In the coarse-mesh 3-D modeling of advanced-reactor transients that involve flow and heat transfer, accurately predicting the turbulent viscosity is a challenging task that requires an accurate and computationally efficient model to capture the unresolved fine-scale turbulence. In this paper, we propose a data-driven coarse-mesh turbulence model based on local flow features for the transient analysis of thermal mixing and stratification in a sodium-cooled fast reactor. The model has a coarse-mesh setup to ensure computational efficiency, while it is trained by fine-mesh computational fluid dynamics (CFD) data to ensure accuracy. A novel neural network architecture, combining a densely connected convolutional network and a long-short-term-memory network, is developed that can efficiently learn from the spatial-temporal CFD transient simulation results. The neural network model was trained and optimized on a loss-of-flow transient and demonstrated high accuracy in predicting the turbulent viscosity field during the whole transient. The trained model's generalization capability was also investigated on two other transients with different inlet conditions. The study demonstrates the potential of applying the proposed data-driven approach to support the coarse-mesh multi-dimensional modeling of advanced reactors. [ABSTRACT FROM AUTHOR]

Due to the significant delay and cost associated with experimental tests, a model-based evaluation of concrete compressive strength is of high value, both for the purpose of strength prediction as well as the mixture optimization. In contrast to the prior recent studies employing a single regression model, in this paper, we present a combined multi-model framework where the regression methods based on artificial neural network, random forest regression and polynomial regression are jointly implemented for compressive strength prediction with a higher accuracy. The outcomes of the individual regression models are combined via a linear weighting strategy and optimized over the training data set as a quadratic convex optimization problem. It is worth mentioning that due to the convexity of the formulated problem, the globally optimum weighting strategy is obtained via standard numerical solvers. Afterwards, employing the obtained regression model, a multi-objective genetic algorithm-based method is proposed for mixture optimization under practical constraints, where a Pareto front of the cost-CS trade-off has been obtained employing the available data set. Numerical evaluations show that the proposed multi-model regression achieves a significantly higher prediction accuracy, i.e., approximately 18% reduction in the obtained prediction mean squared error, without weight optimization, and roughly 30% reduction in the obtained prediction mean squared error with an optimized combination following a convex quadratic optimization, compared to the best single model regression method employing a multi-layered artificial neural network. [Display omitted] • Proposing a data-aided regression method for concrete compressive strength prediction. • Employing a combined multi-model approach to improve prediction accuracy. • Mixture parameter optimization via NSGA-II, employing the obtained regression model. [ABSTRACT FROM AUTHOR]

The purpose of feeder-level energy disaggregation is to decouple the net load measured at the feeder-head into various components. This technology is vital for power system utilities since increased visibility of controllable loads enables the realization of demand-side management strategies. However, energy disaggregation at the feeder level is difficult to realize since the high penetration of embedded generation masks the actual demand and different loads are highly aggregated. In this paper, the solar energy at the grid supply point is separated from the net load at first via either an unsupervised upscaling method or the supervised gradient boosting regression tree (GBRT) method. To deal with the uncertainty of the load components, the probabilistic energy disaggregation models based on multi-quantile recurrent neural network model (multi-quantile long short-term memory (MQ-LSTM) model and multi-quantile gated recurrent unit (MQ-GRU) model) are proposed to disaggregate the demand load into thermostatically controlled loads (TCLs), non-thermostatically controlled loads (non-TCLs), and non-controllable loads. A variety of relevant information, including feeder measurements, meteorological measurements, calendar information, is adopted as the input features of the model. Instead of providing point prediction, the probabilistic model estimates the conditional quantiles and provides prediction intervals. A comprehensive case study is implemented to compare the proposed model with other state-of-the-art models (multi-quantile convolutional neural network (MQ-CNN), quantile gradient boosting regression tree (Q-GBRT), Quantile Light gradient boosting machine (Q-LGB)) from training time, reliability, sharpness, and overall performance aspects. The result shows that the MQ-LSTM can estimate reliable and sharp Prediction Intervals for target load components. And it shows the best performance among all algorithms with the shortest training time. Finally, a transfer learning algorithm is proposed to overcome the difficulty to obtain enough training data, and the model is pre-trained via synthetic data generated from a public database and then tested on the local dataset. The result confirms that the proposed energy disaggregation model is transferable and can be applied to other feeders easily. • Feeder level energy disaggregation system to separate demand into sub-components. • A multi-quantile LSTM model that estimates the conditional quantiles. • A solar energy separation model to detect the behind-the-meter PV generation. • A transfer learning model to apply a pre-trained model to other location. [ABSTRACT FROM AUTHOR]

• A deep neural network model (DNN) is developed to predict the compressive strength of rubber concrete. • The database construction procedure focuses on the choice of different input parameters in the input space. • The compressive strength specimen type is considered as input parameter. • The accuracy of the proposed DNN model reaches an outstanding value. • Alternative possibilities for the input space are discussed from a practical engineering point of view. This paper presents an innovative development process of a Deep Neural Network model to predict the compressive strength of rubber concrete. To this goal, a rubber concrete database is carefully constructed, incorporating a set of binder, aggregate, and other related concrete variables as input parameters, whereas the compressive strength is considered as output. The development of the DNN model includes extensive analysis of the number of hidden layers and the neurons in each layer, combining with a statistical investigation of the models' prediction outputs. The results show that the DNN model outperforms other neural network structures according to several well-known performance indices, such as coefficient of determination, root mean square error, and mean absolute error. The proposed DNN model also exhibits higher prediction accuracy than previously published results, using different machine learning algorithms in the literature. A sensitivity analysis using partial dependence plots is performed within the DNN algorithm in order to achieve an in-depth examination of the influence of each single input variable on the predicted compressive strength of rubber concrete. Finally, the possibility of using other input variables is given to pave the way for applications in regular, high strength, or light-weight foamed concrete containing rubber particles. [ABSTRACT FROM AUTHOR]

Non-orthogonal multiple access (NOMA) emerges as a promising candidate for 5G, which radically alters the way users share the spectrum. In the NOMA system, user clustering (UC) becomes another research issue as grouping the users on different subcarriers with different power levels has a significant impact on spectral utilization. In previous literature, plenty of works have been devoted to solving the UC problem. Recently, the artificial neural network (ANN) has gained considerable attention due to the availability of UC datasets, obtained from the Brute-Force search (BF-S) method. In this paper, deep neural network-based UC (DNN-UC) is employed to effectively characterize the nonlinearity between the cluster formation with channel diversity and transmission powers. Compared to the ANN-UC, the DNN-UC is more competent as UC is a non-convex NP-complete problem, which cannot be entirely captured by the ANN model. In this work, the DNN-UC is first trained with the training samples and then validated with the testing samples to examine its mean square error (MSE) and throughput performance in an asymmetrical fading NOMA channel. Unlike the ANN-UC, the DNN-UC model offers greater room for hyper-parameter optimizations to maximize its learning capability. With the optimized hyper-parameters, the DNN-UC can achieve near-optimal throughput performance, approximately 97% of the throughput of the BF-S method. [ABSTRACT FROM AUTHOR]