Search

• Review of papers on the diagnosis of brain diseases using the fusion of neuroimaging modalities is presented. • For each brain disease, the most appropriate neuroimaging modalities and best combinations are introduced. • Advantages and disadvantages of fusion levels and different modalities are introduced. • Details of various types of deep neural networks used for the task at hand are also examined. • Finally, future directions and challenges in the field are discussed. Brain diseases, including tumors and mental and neurological disorders, seriously threaten the health and well-being of millions of people worldwide. Structural and functional neuroimaging modalities are commonly used by physicians to aid the diagnosis of brain diseases. In clinical settings, specialist doctors typically fuse the magnetic resonance imaging (MRI) data with other neuroimaging modalities for brain disease detection. As these two approaches offer complementary information, fusing these neuroimaging modalities helps physicians accurately diagnose brain diseases. Typically, fusion is performed between a functional and a structural neuroimaging modality. Because the functional modality can complement the structural modality information, thus improving the performance for the diagnosis of brain diseases by specialists. However, analyzing the fusion of neuroimaging modalities is difficult for specialist doctors. Deep Learning (DL) is a branch of artificial intelligence that has shown superior performances compared to more conventional methods in tasks such as brain disease detection from neuroimaging modalities. This work presents a comprehensive review paper in the field of brain disease detection from the fusion of neuroimaging modalities using DL models like convolutional neural networks (CNNs), recurrent neural networks (RNNs), pretrained, generative adversarial networks (GANs), and Autoencoders (AEs). First, neuroimaging modalities and the need for fusion are discussed. Then, review papers published in the field of neuroimaging multimodalities using AI techniques are explored. Moreover, fusion levels based on DL methods, including input, layer, and decision, with related studies conducted on diagnosing brain diseases, are discussed. Other sections present the most important challenges for diagnosing brain diseases from the fusion of neuroimaging modalities. In the discussion section, the details of previous research on the fusion of neuroimaging modalities based on MRI and DL models are reported. In the following, the most important future directions include Datasets, DA, imbalanced data, DL models, explainable AI, and hardware resources are presented. Finally, the main findings of this study are presented in the conclusion section. [ABSTRACT FROM AUTHOR]

Over the past decades, deep learning techniques have attracted increased attention from various research and industrial domains aligned with the development of Industry Internet-of-Things(IIoT). Specifically, with the advantage of data-driven methods, industrial organizations are seeking novel proactive strategies supported by analytic models to guarantee the quality of their production by observing degradation or predicting failure ahead of the occurrence of the component or asset. Predictive strategies are expected to promise the influence of unnecessary maintenance interruptions and mitigate the consequence of that, hence, extending the remaining useful life of products. This paper conducts a survey of the utilization of deep learning technologies on engineering applications where they provide satisfactory solutions with respect to specific data types or input signals. 106 primary papers are reviewed on deep learning–driven approaches which mainly explore five of the most popular architectures in the application of predictive maintenance. The main content of this paper summarizes the common advantages of each architecture and, accordingly, points out their limitations, as well as describes the application scopes of fully connected deep neural networks, convolutional neural networks, stacked autoencoders, deep belief networks, and deep recurrent neural networks. Based on the technique discussion for each of them, we intend to provide a comprehensive understanding and guidance of the appropriate usage of deep learning architectures to devise an effective predictive maintenance strategy for the scientific and industrial developers whose expertise lies in the prior domain knowledge of multi-source isomerization data. Moreover, the main content demonstrated the summarization of the decisive factor by which the incremental stages of the approaches were determined, fundamentally including the dataset specification, feature extraction, and the integration of deep learning approaches. [ABSTRACT FROM AUTHOR]

In personal identity recognition systems, detecting a person's age, gender, and language using voice signal characteristics is a crucial issue, especially because of the importance of security considerations. Age, gender, and language classification problems are important in signal processing because they are used to analyze and understand human behavior, interactions, and preferences. This can be especially useful in the fields of human-computer interaction, psychology, and social science research. In this paper, a new system for detecting a speaker's age, gender, and language based on deep learning models is presented. Deep learning models have shown great efficacy in various fields of signal processing. For this paper, a range of deep models were tested, including convolutional neural networks (CNNs), recurrent neural network (RNN), and a fine-tuning ResNet34 architecture. Additionally, techniques such as transfer learning were applied to improve the efficiency of the proposed system. The input voice signals are preprocessed by applying the spectro-temporal transform to obtain additional features that can be fed to the ResNet34 model, which is designed specifically for the task of voice signal processing. The dataset used in this paper was sourced from the Mozilla common voice initiative, which is dedicated to advancing speech recognition and language identification technologies. The performance of the proposed algorithm was evaluated in the presence of Gaussian noise to determine its robustness. The experimental results demonstrated that the proposed algorithm significantly outperformed basic algorithms and other deep neural networks in terms of age and gender recognition from voice signals. [ABSTRACT FROM AUTHOR]

Transformers are Deep Neural Networks (DNN) that utilize a self-attention mechanism to capture contextual relationships within sequential data. Unlike traditional neural networks and variants of Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM), Transformer models excel at managing long dependencies among input sequence elements and facilitate parallel processing. Consequently, Transformer-based models have garnered significant attention from researchers in the field of artificial intelligence. This is due to their tremendous potential and impressive accomplishments, which extend beyond Natural Language Processing (NLP) tasks to encompass various domains, including Computer Vision (CV), audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published, spotlighting the Transformer's contributions in specific fields, architectural disparities, or performance assessments, there remains a notable absence of a comprehensive survey paper that encompasses its major applications across diverse domains. Therefore, this paper addresses this gap by conducting an extensive survey of proposed Transformer models spanning from 2017 to 2022. Our survey encompasses the identification of the top five application domains for Transformer-based models, namely: NLP, CV, multi-modality, audio and speech processing, and signal processing. We analyze the influence of highly impactful Transformer-based models within these domains and subsequently categorize them according to their respective tasks, employing a novel taxonomy. Our primary objective is to illuminate the existing potential and future prospects of Transformers for researchers who are passionate about this area, thereby contributing to a more comprehensive understanding of this groundbreaking technology. • The paper presents a comprehensive survey on transformers for deep learning tasks. • The paper conducts a thorough analysis on highly effective models in five domains. • The paper classifies the models based on respective tasks using a proposed taxonomy. • The characteristics of the surveyed models are deeply explored and analyzed. • Future directions and challenges for transformer-based models are deciphered. [ABSTRACT FROM AUTHOR]

The combination of convolutional and recurrent neural networks is a promising framework. This arrangement allows the extraction of high-quality spatio-temporal features together with their temporal dependencies. This fact is key for time series prediction problems such as forecasting, classification or anomaly detection, amongst others. In this paper, the TSFE DL library is introduced. It compiles 22 state-of-the-art methods for both time series feature extraction and prediction, employing convolutional and recurrent deep neural networks for its use in several data mining tasks. The library is built upon a set of Tensorflow+Keras and PyTorch modules under the AGPLv3 license. The performance validation of the architectures included in this proposal confirms the usefulness of this Python package. [ABSTRACT FROM AUTHOR]

Accurate relative pose estimation of a spacecraft during space landing operation is critical to ensure a safe and successful landing. This paper presents a 3D Light Detection and Ranging (LiDAR) based AI relative navigation architecture solution for autonomous space landing. The proposed architecture is based on a hybrid Deep Recurrent Convolutional Neural Network (DRCNN) combining a Convolutional Neural Network (CNN) with an Recurrent Neural Network (RNN) based on a Long–Short Term Memory (LSTM) network. The acquired 3D LiDAR data is converted into a multi-projected images and feed the DRCNN with depth and other multi-projected imagery. The CNN module of the architecture allows an efficient representation of features, and the RNN module, as an LSTM, provides robust navigation motion estimates. A variety of landing scenarios are considered, simulated and experimented to evaluate the efficiency of the proposed architecture. A LiDAR based imagery data (Range, Slope, and Elevation) is initially created using PANGU (Planet and Asteroid Natural Scene Generation Utility) software and an evaluation of the proposed solution using this data is conducted. Tests using an instrumented Aerial Robot in Gazebo software to simulate landing scenarios on a synthetic but representative lunar terrain (3D digital elevation model) is proposed. Finally, real experiments using a real flying drone equipped with a Velodyne VLP16 3D LiDAR sensor to generate real 3D scene point clouds while landing on a designed down scaled lunar moon landing surface are conducted. All the test results achieved show that the suggested architecture is capable of delivering good 6 Degree of Freedom (DoF) pose precision with a good and reasonable computation. • Effective feature representation due to the adoption of the CNN module. • The RNN (LSTM) module provides a robust and efficient modelling of the navigation kinematics. • The hybrid network is flexible to train and test on different data modalities. [ABSTRACT FROM AUTHOR]

Urban ride-hailing demand prediction is a crucial but challenging task for intelligent transportation system construction. Predictable ride-hailing demand can facilitate more reasonable vehicle scheduling and online car-hailing platform dispatch. Conventional deep learning methods with no external structured data can be accomplished via hybrid models of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) by meshing plentiful pixel-level labeled data, but data sparsity of high grid granularity in spatial perspective and limited learning capabilities of long-term dependencies in temporal perspective are still two striking bottlenecks. To address these problems, we propose a novel virtual graph modeling approach to focus on significant demand regions and a novel Deep Multi-View Spatio-temporal Virtual Graph Neural Network (DMVST-VGNN) to strengthen the learning capabilities of spatial dynamics and long-term temporal dependencies. Specifically, DMVST-VGNN integrates structures of 1D CNN, Multi-Graph Attention Neural Network and Transformer Network, which correspond to short-term temporal dynamics view, spatial dynamics view and long-term temporal dynamics view respectively. In this paper, multiple experiments are conducted on two large-scale New York City datasets in higher granularity prediction scenes. And the experimental results demonstrate the effectiveness of DMVST-VGNN framework in ride-hailing demand prediction, no matter in spatial scale or the temporal scale. [ABSTRACT FROM AUTHOR]

Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural networks have been introduced to deal with different types of problems. However, the main goal of any neural network is to transform the non-linearly separable input data into more linearly separable abstract features using a hierarchy of layers. These layers are combinations of linear and nonlinear functions. The most popular and common non-linearity layers are activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish and Mish. In this paper, a comprehensive overview and survey is presented for AFs in neural networks for deep learning. Different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based are covered. Several characteristics of AFs such as output range, monotonicity, and smoothness are also pointed out. A performance comparison is also performed among 18 state-of-the-art AFs with different networks on different types of data. The insights of AFs are presented to benefit the researchers for doing further research and practitioners to select among different choices. The code used for experimental comparison is released at: https://github.com/shivram1987/ActivationFunctions. [ABSTRACT FROM AUTHOR]

• Introduction of a generalizable Bayesian recurrent neural network framework for uncertainty quantification in traffic flow prediction. • Utilization of spectral normalization in hidden layers of neural network to enhance generalizability and improve uncertainty estimates. • Demonstration of how normalization alters the training process, controlling model complexity and reducing overfitting. • Spectral normalization improves generalization performance of deep learning models on out-of-distribution datasets, which is crucial for traffic management applications. • Ability of spectral normalization to localize latent feature space under data perturbations, leading to improved generalizability. • Significant outperformance of spectral normalization over layer normalization and models without normalization in both single and multistep prediction horizons. Deep-learning models for traffic data prediction can have superior performance in modeling complex functions using a multi-layer architecture. However, a major drawback of these approaches is that most of these approaches do not offer forecasts with uncertainty estimates, which are essential for traffic operations and control. Without uncertainty estimates, it is difficult to place any level of confidence in the model predictions, and operational strategies relying on overconfident predictions can lead to worsening traffic conditions. In this study, we propose a Bayesian recurrent neural network framework for uncertainty quantification in traffic prediction with higher generalizability by introducing spectral normalization to its hidden layers. In our paper, we have shown that normalization alters the training process of deep neural networks by controlling the model's complexity and reducing the risk of overfitting the training data. This, in turn, helps improve the generalization performance of the model on out-of-distribution datasets. Results demonstrate that spectral normalization improves uncertainty estimates and significantly outperforms both the layer normalization and model without normalization in both single and multistep prediction horizons. This improved performance can be attributed to the ability of spectral normalization to better localize the latent feature space of the data under perturbations. Our findings are especially relevant to traffic management applications, where predicting traffic conditions across multiple locations is the goal, but the availability of training data from multiple locations is limited. Spectral normalization, therefore, provides a more generalizable approach that can effectively capture the underlying patterns in traffic data without requiring location-specific models. [ABSTRACT FROM AUTHOR]

Today, a significant content of the system on chips (SOCs) is dedicated to static random access memory (SRAM) cells. Due to the stress during SRAM operation, MOSFET aging is becoming an increasingly serious problem. The manufacturing variability of the MOSFETs causes further differences in the aging of various devices. This paper presents a simulation methodology using a deep neural network (DNN) to predict the performance of SRAM cells, taking into account the impacts of fabrication variations, temperature, and aging. A dataset of four input features and six output criteria is utilized, which are derived from power consumption, power supply transient current, signal rise time, read static noise margin (RSNM), and butterfly and N-curve. The accuracy of the long short-term memory (LSTM) model in predicting the performance of SRAM cells outperformed the other models such as multilayer perceptron (MLP), one-dimensional convolutional neural network (1D-CNN), gated recurrent unit (GRU), and recurrent neural network (RNN) by 1.85 % to 1.11 %. For instance, by processing the supply current data, the LSTM model achieved 97.41 %, 97.96 %, and 97.41 % accuracy in predicting the RMS power consumption, the signal rise time, and the average power supply current of the SRAM cell, respectively. • DNN successfully predicts the aging and process variation impact on SRAM performance. • Supply current illustrates superior results in performance prediction. • LSTM DNN model outperforms MLP, RNN, 1D-CNN, and GRU, exhibiting sufficient accuracy. [ABSTRACT FROM AUTHOR]

Artificial neural networks have achieved great success in artificial intelligence. Back propagation is one of the most widely used parameter search techniques for artificial neural networks, and it has been widely used in many deep learning approaches such as convolutional neural networks (CNNs), generator adversarial networks (GANs) and recurrent neural networks (RNNs) to train their fully-connected layers. However, the fully-connected layers of the neural networks with back propagation technique depends on the gradient descent strategy, which has the disadvantage of easily falling into local optima. Meanwhile, many evolutionary computation (EC) techniques are applied to the parameters optimization of neural networks. They take advantage of populations to achieve global search. But their convergence speed is relatively slow since they use their original evolutionary search strategies. In this paper, we propose a self-adaptive gradient descent search algorithm (SaGDSA) to search the parameters for fully-connected neural networks. Different from the existing adaptive gradient descent algorithm, the proposed algorithm does not require the user to have any prior knowledge and manually design the learning rate of different stages to match the different search stages. In addition, four kinds of gradient descent strategies are used to optimize parameters of fully-connected neural networks. All used datasets are collected from the University of California Irvine (UCI) machine learning repository. The experimental results indicate that SaGDSA is better than the comparison algorithms on most of the datasets. By introducing the self-adaptive mechanism, SaGDSA not only has excellent global search capabilities from evolutionary computation but also has local search capabilities from gradient descent. [ABSTRACT FROM AUTHOR]

This paper proposes a novel sequence-to-sequence recurrent neural network model to predict re-entry trajectories of uncontrolled space objects. The epoch and altitude information of the objects from two-line element data were interpolated to equally spaced data and were used as inputs to the model. More than 200 datasets from five different objects with already known re-entry times were used as the training set for the model. Test datasets being not used for training, were chosen to validate the model's performance. To assess the performance, the model predictions were compared with the ground-truth data of the test objects and with predictions from the Inter-Agency Space Debris Coordination re-entry campaign, in which space agencies from various countries had participated. The results indicate that the proposed approach is highly suitable for re-entry prediction compared to classical dynamics-based approaches, despite the small quantity of training data. [ABSTRACT FROM AUTHOR]

• We explored two largest smartphone sensor-based datasets for classifying daily activities and transportation mode. • The impact and eficiency of different smartphone sensors have been analyzed for locomotion activities. • We proposed a post-processing classification approach (Mode technique) to refine classifier predictions. • We proposed window length optimization and smartphone orientation independent features. • We evaluated our proposed post-processing technique for both statistical machine learning and deep learning-based models. Recognition of daily human activities in various locomotion and transportation modes has numerous applications like coaching users for behavior modification and maintaining a healthy lifestyle. Besides, applications and user interfaces aware of user mobility through their smartphones can also aid in urban transportation planning, smart parking, and vehicular traffic monitoring. In this paper, we explored smartphone sensor-based two benchmark datasets (Sussex Huawei Locomotion (SHL) and Transportation Mode Detection (TMD)). Firstly, we demonstrated preprocesssing of sensor data, window length optimization based on Akaike Information Criteria (AIC), and introduced smartphone orientation independent features. We also provided an in-depth analysis of different smartphone sensors' importance for classifying daily activities and transportation modes. We justified the sensor relevance by showing the variation of performances with the number of sensors explored. For refining classifier predictions, we also proposed a post-processing approach named "Mode technique". This method primarily concentrates on the statistical analysis of transportation modes and improves the activity recognition rate in statistical classifiers: Decision Tree, K-Nearest Neighbors, Linear Discriminant Analysis, Logistic Regression, Support Vectors Machine with RBF kernel, Random Forest, and deep learning-based methods: Artificial Neural Network and Recurrent Neural Network by smoothing the outputs of these classifiers. Besides, we showed the use of magnitude and jerk-based features to overcome the overfitting problem due to smartphone orientation. We obtained 97.2% accuracy in the SHL dataset and 99.13% accuracy in the TMD dataset. These results demonstrate that our approach can profoundly recognize various activities in advanced locomotion and transportation modes compared to existing methods in two large-scale datasets. [ABSTRACT FROM AUTHOR]

Researchers and practitioners have attempted to predict the financial market by analyzing textual (e.g., news articles and social media) and numeric data (e.g., hourly stock prices, and moving averages). Among textual data, while many papers have been published that analyze social media, news content has gained limited attention in predicting the stock market. Acknowledging that news is critical in predicting the stock market, the focus of this systematic review is on papers investigating machine learning and text mining techniques to predict the stock market using news. Using Kitchenham's methodology, we present a systematic review of the literature on intelligent financial market prediction, examining data mining and machine learning approaches and the employed datasets. From five digital libraries, we identified 61 studies from 2015–2022 for synthesis and interpretation. We present notable gaps and barriers to predicting financial markets, then recommend future research scopes. Various input data, including numerical (stock prices and technical indicators) and textual data (news text and sentiment), have been employed for news-based stock market prediction. News data collection can be costly and time-consuming: most studies have used custom crawlers to gather news articles; however, there are financial news databases available that could significantly facilitate news collection. Furthermore, although most datasets have covered fewer than 100K records, deep learning and more sophisticated artificial neural networks can process enormous datasets faster, improving future model performance. There is a growing trend toward using artificial neural networks, particularly recurrent neural networks and deep learning models, from 2018 to 2021. Furthermore, regression and gradient-boosting models have been developed for stock market prediction during the last four years. Although word embedding approaches for feature representation have been employed recently with good accuracy, emerging language models may be a focus for future research. Advanced natural language processing methods like transformers have undeniably contributed to intelligent stock market prediction. However, stock market prediction has not yet taken full advantage of them. • Emerging articles (2015–2021) on news-based stock market prediction are reviewed. • There is a growing trend toward using news headlines for stock market prediction. • Recurrent neural networks are advantageous models for stock market prediction. • Stock market prediction not fully took advantage of Natural language processing yet. • Quarterly and annual reports of companies are authorized credible data sources. [ABSTRACT FROM AUTHOR]

• Fault diagnosis of rotating machines is important for nuclear power plants. • Four deep learning models are presented for comparative study. • Three cases, original sample size, sample reduction, noise addition, are considered. • The convolutional recurrent neural network achieves the best performance. Deep learning methods with powerful automatic feature extraction and end-to-end modeling capabilities can build fault diagnosis models based on raw data without relying on manual feature extraction procedures. In this paper, a comparative study of deep learning-based fault diagnosis methods for rotating machines in nuclear power plants is conducted. 4 deep learning models, namely, Deep Feed-forward Neural Network, Convolutional Neural Network, Gated Recurrent Unit Neural Network and Convolutional Recurrent Neural Network (CRNN), are selected. 2 publicly available experimental datasets of bearing faults are selected as modeling data. The model performance is compared under 3 cases: original sample size, sample reduction and noise addition. The results show that the CRNN model can achieve state-of-the-art accuracy and the best performance in all test cases. It has the advantages of good small sample learning capability and anti-noise robustness compared to other models in this paper. [ABSTRACT FROM AUTHOR]

Well production forecasting has a very important guiding significance for oilfield production and management. The traditional BP neural network is difficult to deal with the data with time continuity, and the recurrent neural network (RNN) has some shortcomings such as vanishing gradients and exploding gradients. In order to avoid these problems and further improve the accuracy of neural network models in well production prediction. In this study, a prediction model combining temporal convolutional network (TCN) and attention mechanism is proposed. TCN is able to obtain more sensory fields to solve longer time series with fewer network layers by causally expanding convolution. And the residual connectivity in the model structure enables the model to transfer information across layers to solve the problem of network degradation caused by overly deep networks. The attention mechanism can reinforce the influence of important features on the prediction results to improve the prediction accuracy of the model. To evaluate the effectiveness of the model, the TCN-attention model was used to forecast the oil production of four producing wells, and the prediction results were compared with those of LSTM, TCN, RNN, and ARIMA. The results showed that TCN-Attention outperformed the other models in the prediction of the four wells in all evaluation metrics. The method proposed in this paper has excellent data fitting ability, can provide accurate prediction results, and have a certain application value in oilfield production. • The highlights of the paper are: • The model achieves better performance in forecasting than conventional models. • Temporal Convolutional Network can better extract time-series features. • Attention mechanism can effectively improve the model prediction accuracy. [ABSTRACT FROM AUTHOR]

Alzheimer's Disease (AD) is an irreversible neurodegenerative disease that results in a progressive decline in cognitive abilities. Since AD starts several years before the onset of the symptoms, its early detection is challenging due to subtle changes in biomarkers mainly detectable in different neuroimaging modalities. Developing computer-aided diagnostic models based on deep learning can provide excellent opportunities for the analysis of different neuroimage modalities along with other non-image biomarkers. In this survey, we perform a comparative analysis of about 100 published papers since 2019 that employ basic deep architectures such as CNN, RNN, and generative models for AD diagnosis. Moreover, about 60 papers that have applied a trending topic or architecture for AD are investigated. Explainable models, normalizing flows, graph-based deep architectures, self-supervised learning, and attention mechanisms are considered. The main challenges in this body of literature have been categorized and explained from data-related, methodology-related, and clinical adoption aspects. We conclude our paper by addressing some future perspectives and providing recommendations to conduct further studies for AD diagnosis. [ABSTRACT FROM AUTHOR]

Deep neural networks have become increasingly important in recent years for creating molecules with desirable properties. In general, SMILES strings are used to train deep neural network based models. The trained model is then used to generate the desired molecules. Unfortunately, due to syntactical and semantic flaws in the representation, the SMILES string generates a substantial number of invalid molecules. SMILES representation fails to efficiently handle rings, branch and bonds between atoms. Lack of robustness in dealing with the cited aspects results in abundance of invalid strings. To overcome this limitation, this paper proposes a SMILES like representation, called GenSMILES. GenSMILES tackles syntactical and semantic issues by relying upon derivative rules to apply constraints. This causes a generative model produces more valid SMILES strings. By substituting a single notation for the pair representation of branches and rings in SMILES with a) and ^, respectively, GenSMILES corrects the syntactical issues. The mismatching of atom's bonds is the main cause of semantic errors. GenSMILES addresses such issues by employing derivation rules during string conversion from GenSMILES to SMILES. Every SMILES string can be represented with an equivalent GenSMILES. When used for designing drug molecule, GenSMILES increases molecule's validity when compared with SMILES and DeepSMILES on two popular architectures i.e., Recurrent Neural Network and Variational Autoencoder. The main benefit of GenSMILES is that it can be applied directly to generative algorithms without adapting the model environment. GenSMILES is beneficial not only to generative approaches of DL but also to the approaches that use SMILES string-like representation. On most of the datasets, GenSMILES is effective in improving validity above 90% and diversity score 15. GenSMILES results in more diversity in the properties of generated molecules and allows exploration of larger portion of undiscovered chemical space as compared to SMILES and DeepSMILES. GenSMILES guarantee that the generative model does not need to remember any long dependencies and is principal contribution of this work. [ABSTRACT FROM AUTHOR]

• Deep learning models learn plausible relationships between features and crop yield. • Neural networks learn features that perform similar to expert-designed features. • Feature influence on yield matches expert knowledge when model accuracy is high. Machine learning models for crop yield forecasting often rely on expert-designed features or predictors. The effectiveness and interpretability of these handcrafted features depends on the expertise of the people designing them. Neural networks have the ability to learn features directly from input data and train the feature learning and prediction steps simultaneously. In this paper, we evaluate the performance and interpretability of neural network models for crop yield forecasting using data from the MARS Crop Yield Forecasting System of the European Commission's Joint Research Centre. The selected neural networks can handle sequential or time series data and include long short-term memory (LSTM) recurrent neural network and 1-dimensional convolutional neural network (1DCNN). Performance was compared with a linear trend model and a Gradient-Boosted Decision Trees (GBDT) model, trained using hand-designed features. Feature importance scores of input variables were computed using feature attribution methods and were analyzed by crop yield modeling and agronomy experts. Results showed that LSTM models perform statistically better than GBDT models for soft wheat in Germany and similar to GBDT models for all other case studies. In addition, LSTM models captured the effect of yield trend, static features (e.g. elevation, soil water holding capacity) and biomass features on crop yield well, but struggled to capture the impact of extreme temperature and moisture conditions. Our work shows the potential of deep learning to automatically learn features and produce reliable crop yield forecasts, and highlights the importance and challenges of involving human stakeholders in assessing model interpretability. [ABSTRACT FROM AUTHOR]

The ongoing COVID-19 pandemic has created an unprecedented predicament for global supply chains (SCs). Shipments of essential and life-saving products, ranging from pharmaceuticals, agriculture, and healthcare, to manufacturing, have been significantly impacted or delayed, making the global SCs vulnerable. A better understanding of the shipment risks can substantially reduce that nervousness. Thenceforth, this paper proposes a few Deep Learning (DL) approaches to mitigate shipment risks by predicting "if a shipment can be exported from one source to another", despite the restrictions imposed by the COVID-19 pandemic. The proposed DL methodologies have four main stages: data capturing, de-noising or pre-processing, feature extraction, and classification. The feature extraction stage depends on two main variants of DL models. The first variant involves three recurrent neural networks (RNN) structures (i.e., long short-term memory (LSTM), Bidirectional long short-term memory (BiLSTM), and gated recurrent unit (GRU)), and the second variant is the temporal convolutional network (TCN). In terms of the classification stage, six different classifiers are applied to test the entire methodology. These classifiers are SoftMax, random trees (RT), random forest (RF), k-nearest neighbor (KNN), artificial neural network (ANN), and support vector machine (SVM). The performance of the proposed DL models is evaluated based on an online dataset (taken as a case study). The numerical results show that one of the proposed models (i.e., TCN) is about 100% accurate in predicting the risk of shipment to a particular destination under COVID-19 restrictions. Unarguably, the aftermath of this work will help the decision-makers to predict supply chain risks proactively to increase the resiliency of the SCs. • Proposes a wide range of DL approaches to predict supply chain risks. • Introducing a temporal convolutional network (TCN) by combining RNN and CNN models. • Identification of the most promising DL approach in terms of performance. • Performance demonstration of the selected classifiers on the advanced DL networks. [ABSTRACT FROM AUTHOR]

Machine learning (ML) and a nonlinear model predictive controller (NMPC) are used in this paper to minimize the emissions and fuel consumption of a compression ignition engine. In this work machine learning is applied in two methods. In the first application, ML is used to identify a model for implementation in model predictive control optimization problems. In the second application, ML is used as a replacement of the NMPC where the ML controller learns the optimal control action by imitating or mimicking the behavior of the model predictive controller. In this study, a deep recurrent neural network including long–short term memory (LSTM) layers are used to model the emissions and performance of an industrial 4.5 liter 4-cylinder Cummins diesel engine. This model is then used for model predictive controller implementation. Then, a deep learning scheme is deployed to clone the behavior of the developed controller. In the LSTM integration, a novel scheme is used by augmenting hidden and cell states of the network in an NMPC optimization problem. The developed LSTM-NMPC and the imitative NMPC are compared with the Cummins calibrated Engine Control Unit (ECU) model in an experimentally validated engine simulation platform. Results show a significant reduction in Nitrogen Oxides (NO x) emissions and a slight decrease in the injected fuel quantity while maintaining the same load. In addition, the imitative NMPC has a similar performance as the NMPC but with a two orders of magnitude reduction of the computation time. [Display omitted] [ABSTRACT FROM AUTHOR]

Ontology development is a challenging task that encompasses many time-consuming activities. One of these activities is the classification of the domain entities (concepts and instances) according to top-level concepts. This activity is usually performed manually by an ontology engineer. However, when the set of entities increases in size, associating each entity to the proper top-level ontological concept becomes challenging and requires a high level of expertise in both the target domain and ontology engineering. This paper proposes a deep learning approach that automatically classifies domain entities into top-level concepts using their informal definitions and the word embedding of the terms that represent them. From these inputs, we feed a deep neural network consisting of two modules: a feed-forward neural network and a bi-directional recurrent neural network with long short-term units. Our architecture combines both outputs of these modules into a dense layer and provides the probabilities of each candidate class. For validating our proposal, we have developed a dataset based on the OntoWordNet ontology, which provides a classification of WordNet synsets into concepts specified by DOLCE-lite-plus top-level ontology. Our experiments show that our proposal outperforms the baseline approaches by 6% regarding the F-score. In addition, our proposal is less affected by the polysemy in the terms that represent the domain entities than the compared approaches. Consequently, our proposal can consider more instances during its training than the baseline methods. • Word embedding and deep learning can be used to predict top-level ontology classes. • The polysemy of word vectors can be solved by using informal definitions of the word. • Combining different neural network models have better performance than a single one. [ABSTRACT FROM AUTHOR]

In general insurance companies, a correct estimation of liabilities plays a key role due to its impact on management and investing decisions. Since the Financial Crisis of 2007–2008 and the strengthening of regulation, the focus is not only on the total reserve but also on its variability, which is an indicator of the risk assumed by the company. Thus, measures that relate profitability with risk are crucial in order to understand the financial position of insurance firms. Taking advantage of the increasing computational power, this paper introduces a stochastic reserving model whose aim is to improve the performance of the traditional Mack's reserving model by applying an ensemble of Recurrent Neural Networks. The results demonstrate that blending traditional reserving models with deep and machine learning techniques leads to a more accurate assessment of general insurance liabilities. • The proposed Mack-Net model leads to a more accurate reserve prediction. • Mack-Net model generates more appropriate risk measures than Mack's model. • Mack-Net model does not increase the variance to generate an appropriate VaR. • Neural Networks estimate more accurate parameters than usual reserving models. [ABSTRACT FROM AUTHOR]

For dealing with uncertainty in Remaining Useful Life (RUL) predictions, numerous studies in literature use stochastic models to characterize the degradation process and predict the RUL distribution. However, in practice, it is difficult to derive stochastic models to capture degradation mechanisms of complex physical systems. Besides, the outstanding achievements in sensing technologies have facilitated the development of data-driven methods. Among them, deep learning methods become one of the most popular trends in recent studies; but they usually provide point predictions without quantifying the output uncertainties. In this paper, we present a new probabilistic deep leaning methodology for uncertainty quantification of multi-component systems' RUL. It is a combination of a probabilistic model and a deep recurrent neural network to predict the components' RUL distributions. Then, using the information about the system's architecture, the formulas to quantify system reliability or system-level-RUL uncertainty are derived. The performance of the proposed methodology is investigated through the benchmark data provided by NASA. The obtained results highlight the point prediction accuracy and the uncertainty management capacity of the proposed methodology. In addition, thanks to the explicit RUL distributions of components, the system reliability for different structures is obtained with high accuracy, especially for series structures. • New data-driven methodology for dealing with uncertainties in prognostics. • Handling uncertainty propagation from component's RUL to system reliability. • New approach based on combination between probabilistic model and neural network. • Discussion of data quantity impact on system prognostics uncertainty management. • Verification of proposed methodology performance through an engineering application. [ABSTRACT FROM AUTHOR]

• Informed deep learning-based framework is proposed for on-board battery health monitoring. • Impedance-related features model battery degradation efficiently. • Layer-wise relevance propagation reveals impedance features' contribution to output. • Knowledge infusion to a recurrent neural network improves estimation accuracy. • Monte Carlo dropout secures the model reliability by providing uncertainty measures. This paper proposes a novel, informed deep-learning-based prognostics framework for on-board state of health and remaining useful life estimations of lithium-ion batteries, which are critical components for strategizing energy and power used in electric vehicles. The framework comprises three phases. First, reliable and online accessible impedance-related features are collected from discharge curves. Second, these features are inputted into the proposed knowledge-infused recurrent neural network, a hybrid model that combines an empirical model with a deep neural network. Third, Monte Carlo dropout, a deep learning method for obtaining a probabilistic prediction of a neural network, is addressed to secure robustness in estimating the state of health and remaining useful life. Layer-wise relevance propagation, a deep learning technique for tracking the evolution of feature importance and offering scientific reasoning of the output, confirms that impedance-related features significantly contribute to the estimation accuracy compared to other features investigated in previous studies. Moreover, the hybrid model improves the estimation accuracy and robustness, whereas Monte Carlo dropout ensures robustness and reliability. Specifically, the estimation results for the public degradation data reveal that the proposed model can output significantly more accurate state of health and remaining useful life estimations than the baseline deep neural networks. The findings of this study provide insight into the explicable and uncertainty-based pipeline of deep neural networks with respect to battery health monitoring, which are highly recommendable features for decision-making and corrective planning of power and energy used in lithium-ion battery cells and packs. [ABSTRACT FROM AUTHOR]

To combat the recent coronavirus disease 2019 (COVID-19), academician and clinician are in search of new approaches to predict the COVID-19 outbreak dynamic trends that may slow down or stop the pandemic. Epidemiological models like Susceptible–Infected–Recovered (SIR) and its variants are helpful to understand the dynamics trend of pandemic that may be used in decision making to optimize possible controls from the infectious disease. But these epidemiological models based on mathematical assumptions may not predict the real pandemic situation. Recently the new machine learning approaches are being used to understand the dynamic trend of COVID-19 spread. In this paper, we designed the recurrent and convolutional neural network models: vanilla LSTM, stacked LSTM, ED_LSTM, BiLSTM, CNN, and hybrid CNN+LSTM model to capture the complex trend of COVID-19 outbreak and perform the forecasting of COVID-19 daily confirmed cases of 7, 14, 21 days for India and its four most affected states (Maharashtra, Kerala, Karnataka, and Tamil Nadu). The root mean square error (RMSE) and mean absolute percentage error (MAPE) evaluation metric are computed on the testing data to demonstrate the relative performance of these models. The results show that the stacked LSTM and hybrid CNN+LSTM models perform best relative to other models. • Various variants of LSTM and CNN deep learning architectures have been designed. • Predicted COVID-19 confirmed cases for India and its four most affected states. • RMSE and MAPE are computed to compare the performance of the designed models. • Experimental results show that predicted cases are very close to actual cases. [ABSTRACT FROM AUTHOR]

With the increase of natural gas in the world's energy consumption, the efficient and reliable management of natural gas pipeline networks is becoming even more important than before. In recent years, Demand Response (DR) is considered an effective approach for cleaner production and economic strategy, by introducing the participation of customers (CUs). This paper proposes a novel DR method for predictive management in multi-level natural gas markets with different stakeholders. This method is able to make a better trade-off among supplier's profits, gas demand volatility and CU satisfaction. This method includes three parts: dynamic pricing model, intelligent decision making and data-driven demand forecasting. A Markov decision process-based model is developed to illustrate the process of dynamical optimizing energy prices. Then, deep learning and reinforcement learning are integrated to efficiently solve the sequential decision-making problem, based on the physics constraints of natural gas pipeline networks. Besides, to realize the function of predictive optimization, an energy demand forecasting model is developed based on the deep recurrent neural network model. The proposed dynamic pricing method is able to optimize the pricing strategies in accordance to the demand patterns, and dynamically improve the system stability and energy efficiency. Finally, we apply the developed method to a natural gas network with relatively complex topology and different CUs. The results indicate that the proposed method can achieve the targets of peak shaving and valley filling under different pricing periods. Besides, the sensitivity analysis of the critical parameters in the dynamic pricing model is analyzed in detail, which can give a solid criterion for ensuring the effectiveness of this framework. • A novel method is developed for predictive energy management. • An intelligent decision-making algorithm based on deep reinforcement learning is proposed. • An efficient dynamic pricing modeling for natural gas market is developed. • The method can make trade-offs among economic benefit, gas supply reliability and CUs' satisfaction. [ABSTRACT FROM AUTHOR]

• Increasingly complex machine learning models lead to lower interpretability. • An attention-based neural network architecture for load prediction is proposed. • The architecture is tested using real operational data from a commercial building. • Proposed architecture shows enhanced prediction accuracy and interpretability. Machine learning has gained increasing popularity in building energy management due to its powerful capability and flexibility in model development as well as the rich data available in modern buildings. While machine learning is becoming more powerful, the models developed, especially artificial neural networks like Recurrent Neural Networks (RNN), are becoming more complex, resulting in "darker models" with lower model interpretability. The sophisticated inference mechanism behind machine learning prevents ordinary building professionals from understanding the models, thereby lowering trust in the predictions made. To address this, attention mechanisms have been widely implemented to improve the interpretability of deep learning; these mechanisms enable a deep learning-based model to track how different inputs influence outputs at each step of inference. This paper proposes a novel neural network architecture with an attention mechanism for developing RNN-based building energy prediction, and investigates the effectiveness of this attention mechanism in improving the interpretability of RNN models developed for 24-hour ahead building cooling load prediction. To better understand, explain and evaluate these neural network-based building energy prediction models, the obtained attention vectors (or metric) are used to visualize the influence of different parts of model inputs on the prediction result. This helps the users to understand why predictions are made by the model, as well as how input sequences proportionally influence the output sequences. Further analysis of attention vectors can provide interesting temporal information for understanding building thermal dynamics, like the thermal inertia of the building. The proposed attention-based architecture can be implemented in developing optimal operation control strategies and improving demand and supply management. The model developed based on this architecture is assessed using real building operational data, and shows improved accuracy and interpretability over baseline models (without adopting attention mechanisms). The research results help to bridge the gap between building professionals and advanced machine learning techniques. The insights obtained can be used as guidance for the development, fine-tuning, explanation and debugging of data-driven building energy prediction models. [ABSTRACT FROM AUTHOR]

• Expanding deep learning is developed for prediction of tunnelling-induced settlement. • The effects of karst geology on tunnelling construction are evaluated. • Expanding method is validated by ANN, LSTM, GRU and Convd1d models. • Influence of dataset size on predictive performance is analyzed. This paper presents the measurement and prediction of the tunnelling-induced surface response in karst ground, Guangzhou, China. A predictive method of ground settlement is proposed named as the expanding deep learning method. This method kinetically uses the expanding tunnelling data to predict ground settlement in real time. Four types of deep learning methods are compared, including artificial neural network (ANN), long short-term memory neural networks (LSTM), gated recurrent unit neural networks (GRU), and 1d convolutional neural networks (Conv1d). Based on static Pearson correlation coefficient, a kinetic correlation analysis method is proposed to evaluate the variable significance of input data on the ground settlement. The effect of cemented karst caves and variable geological conditions are then analysed. The results indicate that the expanding Conv1d model precisely predict the tunnelling-induced ground settlement. The kinetic correlation analysis can reflect the variable influence of geological condition and tunnelling operation parameters on the ground settlement. [ABSTRACT FROM AUTHOR]

Earthquake-induced landslides have resulted in huge casualties and considerable financial repercussions, and slope dynamic response analysis has always been a hot issue. The prediction of the dynamic response of a slope before the occurrence of future earthquakes will benefit disaster prevention and reduction. Traditional methods (such as the finite element method) are mostly based on simplified physical mechanisms and cannot accurately predict the dynamic response of complicated slope systems. This article innovatively applies novel recurrent neural networks to the prediction of the slope dynamic response. Using the results of large-scale shaking-table tests, we introduced a moving-steps strategy and established three recurrent neural network models: Simple-RNN, LSTM and GRU models. Moreover, a multi-layer perceptron prediction model was trained for comparative verification. We also conducted three experiments to investigate the effect of the data volume on the models. Results show that recurrent neural networks perform well in the analysis of the seismic dynamic response of a slope and provide better predictions than the multi-layer perceptron network. When there are many data, the LSTM and GRU models have advantages, and the confidence indexes of their predictions with normalized error within ±5% are 84.5% and 86.4%, respectively. It is concluded that recurrent neural networks are suitable for the time-series prediction of dynamic responses to seismic loads. To some extent, this paper may help reduce the future risks and losses of earthquake-triggered landslide disasters. • It's imperative to predict the co-seismic slope dynamic response sequences. • Predictive abilities of novel RNN models verified in large-scale shaking-table tests. • Normalized prediction errors of GRU within ±5% have a CI of 86.4%. • Future risks and losses of earthquake-triggered landslide disasters may be reduced. [ABSTRACT FROM AUTHOR]

• A deep learning flight time prediction model is proposed to forecast flight delay time. • Designed a spatial weighted layer to capture Origin-Destination specific spatial dependencies. • Stacked the spatial layer with LSTM to learn spatiotemporal patterns in air traffic network. • Estimated fuel consumption reduction is 0.016–1.915% without increasing fuel depletion risk. Under increasing economic and environmental pressure, airlines are constantly seeking new technologies and optimizing flight operations to reduce fuel consumption. However, the current practice on fuel loading, which has a significant impact on aircraft weight and fuel consumption, has yet to be thoroughly addressed by existing studies. Excess fuel is loaded by dispatchers and (or) pilots to handle fuel consumption uncertainties, primarily caused by flight time uncertainties, which cannot be predicted by current Flight Planning Systems (FPS). In this paper, we develop a novel spatial weighted recurrent neural network model to provide better flight time predictions by capturing air traffic information at a national scale based on multiple data sources, including Automatic Dependent Surveillance - Broadcast (ADS-B), Meteorological Aerodrome Reports (METAR), and airline records. In this model, a spatial weighted layer is designed to extract spatial dependences among network delay states (i.e. average flight delay at each airport and average flight delay of each Origin-Destination (OD) pair for a specific time interval). Then, a new training procedure associated with the spatial weighted layer is introduced to extract OD-specific spatial weights and then integrate into one model for a nationwide air traffic network. Long short-term memory (LSTM) networks are used after the spatial weighted layer to extract the temporal behavior patterns of network delay states. Finally, features from delays, weather, and flight schedules are fed into a fully connected neural network to predict the flight time of a particular flight. The proposed model was evaluated using one year of historical data from an airline's real operations. Results show that our model can provide a more accurate flight time predictions than baseline methods, especially for flights with extreme delays. We also show that, with the improved flight time prediction, fuel loading can be optimized and resulting reduced fuel consumption by 0.016%–1.915% without increasing the fuel depletion risk. [ABSTRACT FROM AUTHOR]