Search

Transformers are Deep Neural Networks (DNN) that utilize a self-attention mechanism to capture contextual relationships within sequential data. Unlike traditional neural networks and variants of Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM), Transformer models excel at managing long dependencies among input sequence elements and facilitate parallel processing. Consequently, Transformer-based models have garnered significant attention from researchers in the field of artificial intelligence. This is due to their tremendous potential and impressive accomplishments, which extend beyond Natural Language Processing (NLP) tasks to encompass various domains, including Computer Vision (CV), audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published, spotlighting the Transformer's contributions in specific fields, architectural disparities, or performance assessments, there remains a notable absence of a comprehensive survey paper that encompasses its major applications across diverse domains. Therefore, this paper addresses this gap by conducting an extensive survey of proposed Transformer models spanning from 2017 to 2022. Our survey encompasses the identification of the top five application domains for Transformer-based models, namely: NLP, CV, multi-modality, audio and speech processing, and signal processing. We analyze the influence of highly impactful Transformer-based models within these domains and subsequently categorize them according to their respective tasks, employing a novel taxonomy. Our primary objective is to illuminate the existing potential and future prospects of Transformers for researchers who are passionate about this area, thereby contributing to a more comprehensive understanding of this groundbreaking technology. • The paper presents a comprehensive survey on transformers for deep learning tasks. • The paper conducts a thorough analysis on highly effective models in five domains. • The paper classifies the models based on respective tasks using a proposed taxonomy. • The characteristics of the surveyed models are deeply explored and analyzed. • Future directions and challenges for transformer-based models are deciphered. [ABSTRACT FROM AUTHOR]

Sequential recommendation aims to predict future user interactions by analyzing dynamic patterns within their historical behavior sequences. Deep neural networks have recently become popular for learning representations of these sequences in the time domain. However, representing users' intentions in the time domain faces challenges such as noise in interactions and sparsity of data. Contrastive learning and representation learning in the frequency domain can mitigate these issues from different perspectives. In this paper, to fully integrate time-domain sequence representations, frequency-domain sequence representations and contrastive learning based on them, we propose a model called T ime– F requency C onsistency based contrastive learning for S equential Rec ommendation (TFCSRec). TFCSRec utilizes a time-domain encoder with a fully connected network and a filter network to extract high-order features and catch pure sequential patterns. Then, a learnable frequency-domain encoder with a recurrent neural network is designed to capture sequential characteristics in the frequency-domain space. Finally, TFCSRec combines a recommendation task and two contrastive learning tasks to optimize the two user representation encoders. Its contrastive learning is designed to minimize a contrastive regularization loss and a time–frequency consistency loss, which for the first time is constructed directly on the time-domain sequence representation and the frequency-domain sequence representation. Experiments on five benchmark datasets show that the proposed TFCSRec model outperforms other sequential recommendation models based on deep neural networks. • Design a time-domain encoder with a filter to extract sequential patterns. • Devise a learnable frequency-domain encoder to capture sequential characteristics. • Propose a time–frequency consistency based contrastive loss for the first time. [ABSTRACT FROM AUTHOR]

The aim of this paper is to explore the potential of the class of recurrent neural networks in developing a classification model for the stock market. For this research, the data that replicates the entire Nasdaq stock market limit order book is used. After extracting order book attributes from the raw dataset, feature selection approaches based on conditional entropy and Stochastic Universal Sampling are proposed, in order to highlight potentially informative features for our data classification task. Furthermore, the performances of nine presented recurrent neural network models for classifying the stock market data vector into one of the labels from the set S = {buy, sell, idle} are examined. In this study, the performances of the models based on different networks, namely the gated recurrent unit, the long short-term memory, and the recurrent neural network, are compared. Using a multi-criteria approach it is concluded that the model based on the GRU topology fed with the features selected using the newly proposed feature selection method outperforms other examined models. • The performances of nine different limit order book classification models are examined. • The PROMETHEE method specifies the most suitable model among the observed models. • The proposed feature demonstrates superiority compared to other selection methods. • The models based on different neural network topologies are evaluated. [ABSTRACT FROM AUTHOR]

Researchers and practitioners have attempted to predict the financial market by analyzing textual (e.g., news articles and social media) and numeric data (e.g., hourly stock prices, and moving averages). Among textual data, while many papers have been published that analyze social media, news content has gained limited attention in predicting the stock market. Acknowledging that news is critical in predicting the stock market, the focus of this systematic review is on papers investigating machine learning and text mining techniques to predict the stock market using news. Using Kitchenham's methodology, we present a systematic review of the literature on intelligent financial market prediction, examining data mining and machine learning approaches and the employed datasets. From five digital libraries, we identified 61 studies from 2015–2022 for synthesis and interpretation. We present notable gaps and barriers to predicting financial markets, then recommend future research scopes. Various input data, including numerical (stock prices and technical indicators) and textual data (news text and sentiment), have been employed for news-based stock market prediction. News data collection can be costly and time-consuming: most studies have used custom crawlers to gather news articles; however, there are financial news databases available that could significantly facilitate news collection. Furthermore, although most datasets have covered fewer than 100K records, deep learning and more sophisticated artificial neural networks can process enormous datasets faster, improving future model performance. There is a growing trend toward using artificial neural networks, particularly recurrent neural networks and deep learning models, from 2018 to 2021. Furthermore, regression and gradient-boosting models have been developed for stock market prediction during the last four years. Although word embedding approaches for feature representation have been employed recently with good accuracy, emerging language models may be a focus for future research. Advanced natural language processing methods like transformers have undeniably contributed to intelligent stock market prediction. However, stock market prediction has not yet taken full advantage of them. • Emerging articles (2015–2021) on news-based stock market prediction are reviewed. • There is a growing trend toward using news headlines for stock market prediction. • Recurrent neural networks are advantageous models for stock market prediction. • Stock market prediction not fully took advantage of Natural language processing yet. • Quarterly and annual reports of companies are authorized credible data sources. [ABSTRACT FROM AUTHOR]

This paper presents two novel approaches for uncertainty estimation adapted and extended for the multi-link bus travel time problem. The uncertainty is modeled directly as part of recurrent artificial neural networks, but using two fundamentally different approaches: one based on Deep Quantile Regression and the other on Bayesian neural network. Both approaches use a recurrent neural network to predict multiple time steps into the future, but handle the time-dependent uncertainty estimation differently. We present a novel sampling technique in order to aggregate quantile estimates for link level travel time to yield the multi-link travel time distribution needed for a vehicle to travel from its current position to a specific downstream stop point or transfer site. To motivate the relevance of uncertainty-aware models in the domain, we focus on the connection protection application as a case study: An expert system to determine whether a bus driver should hold and wait for a connecting service, thus ensuring the connection, or break the connection and reduce its own delay. Our results show that the proposed quantile sampling method performs overall best for the 80%, 90% and 95% prediction intervals, both for a 15 min time horizon into the future (t + 1), but also for the 30 and 45 min time horizon (t + 2 and t + 3), with a constant, but very small underestimation of the uncertainty interval (1–4 pp.). However, we also show, that the Bayesian model still can outperform the DQR for specific cases. Lastly, we demonstrate how a simple decision support system can take advantage of our uncertainty-aware travel time models to prioritize the difference in travel time uncertainty for bus holding at strategic points, thus reducing the introduced delay for the connection protection application. • Presents two uncertainty estimation methods for multi-link bus travel time. • Presents a novel quantile sampling technique for multi-link aggregation. • Uses a connection assurance (hold and wait) application as a case study. • Finds that the proposed quantile method performs best for most prediction horizons. [ABSTRACT FROM AUTHOR]

The ongoing COVID-19 pandemic has created an unprecedented predicament for global supply chains (SCs). Shipments of essential and life-saving products, ranging from pharmaceuticals, agriculture, and healthcare, to manufacturing, have been significantly impacted or delayed, making the global SCs vulnerable. A better understanding of the shipment risks can substantially reduce that nervousness. Thenceforth, this paper proposes a few Deep Learning (DL) approaches to mitigate shipment risks by predicting "if a shipment can be exported from one source to another", despite the restrictions imposed by the COVID-19 pandemic. The proposed DL methodologies have four main stages: data capturing, de-noising or pre-processing, feature extraction, and classification. The feature extraction stage depends on two main variants of DL models. The first variant involves three recurrent neural networks (RNN) structures (i.e., long short-term memory (LSTM), Bidirectional long short-term memory (BiLSTM), and gated recurrent unit (GRU)), and the second variant is the temporal convolutional network (TCN). In terms of the classification stage, six different classifiers are applied to test the entire methodology. These classifiers are SoftMax, random trees (RT), random forest (RF), k-nearest neighbor (KNN), artificial neural network (ANN), and support vector machine (SVM). The performance of the proposed DL models is evaluated based on an online dataset (taken as a case study). The numerical results show that one of the proposed models (i.e., TCN) is about 100% accurate in predicting the risk of shipment to a particular destination under COVID-19 restrictions. Unarguably, the aftermath of this work will help the decision-makers to predict supply chain risks proactively to increase the resiliency of the SCs. • Proposes a wide range of DL approaches to predict supply chain risks. • Introducing a temporal convolutional network (TCN) by combining RNN and CNN models. • Identification of the most promising DL approach in terms of performance. • Performance demonstration of the selected classifiers on the advanced DL networks. [ABSTRACT FROM AUTHOR]

• Gear shift strategy. • Mine truck. • Machine learning. • Bi-LSTM network. As mine trucks face severe and changeable driving conditions in mining area, it is important to formulate a general automatic gear shift strategy for reducing the intensity and difficulty of the work. Therefore, gear shift strategy performs a significant role in automatic transmission studies. However, it is difficult to devise an automatic shift strategy due to the fact that it relies on tremendous historical experiment data which is the highest classified commercial secret of the automobile manufacture. Since the artificial neural network can build self-learning model by inputting multiple parameters, adjusting weights and automatically outputting results, many automatic gear shift strategy studies are reported based on it. However, the gear shift prediction accuracy by traditional neural network is limited due to the non-existence of feedback connection, lacking of associative memory function and missing implicit connection between continuous data. Hence, this paper proposes a novel method based on Bi-directional Long Short-Term Memory (Bi-LSTM) network, which controls gear shifting with nine parameters and updates the traditional mine truck transmission system from manual to automatic. Truck status data while manual driving are captured through the Controller Area Network-bus (CAN-bus) and converted to the proper sequence data format. Some sequence data are used to train the Bi-LSTM network for gear prediction, while the rest is used for verification. In order to evaluate the proposed machine learning model performance, it is compared with the Back Propagation Neural Network (BPNN), Recurrent Neural Network (RNN) and Long-Short Term Memory (LSTM) networks with two and nine parameters, respectively. The experimental results demonstrate that machine learning networks with nine parameters reach higher gear shifting prediction accuracy than those with dual-parameter, except BPNN. In addition, the highest prediction accuracy is 95.8 %, which is achieved by the Bi-LSTM network with nine parameters. This proves that Bi-LSTM network outperforms the traditional neural networks, such as BPNN with 81.5 %, and other recurrent neural networks, such as RNN and LSTM with 87 % and 91.5 %, respectively. As for the average time consumption, the Bi-LSTM network spends 1.86 ms per prediction calculation which is longer than LSTM and RNN respectively having 1.58 ms and 1.06 ms. Thus, it still fits the requirement of real-time processing on mine truck industry. In general, the Bi-LSTM network shows its advantage in processing sequential data within limited time, and has the potential to be widely applied in the field of prediction based on data with temporal characteristic. [ABSTRACT FROM AUTHOR]

Ontology development is a challenging task that encompasses many time-consuming activities. One of these activities is the classification of the domain entities (concepts and instances) according to top-level concepts. This activity is usually performed manually by an ontology engineer. However, when the set of entities increases in size, associating each entity to the proper top-level ontological concept becomes challenging and requires a high level of expertise in both the target domain and ontology engineering. This paper proposes a deep learning approach that automatically classifies domain entities into top-level concepts using their informal definitions and the word embedding of the terms that represent them. From these inputs, we feed a deep neural network consisting of two modules: a feed-forward neural network and a bi-directional recurrent neural network with long short-term units. Our architecture combines both outputs of these modules into a dense layer and provides the probabilities of each candidate class. For validating our proposal, we have developed a dataset based on the OntoWordNet ontology, which provides a classification of WordNet synsets into concepts specified by DOLCE-lite-plus top-level ontology. Our experiments show that our proposal outperforms the baseline approaches by 6% regarding the F-score. In addition, our proposal is less affected by the polysemy in the terms that represent the domain entities than the compared approaches. Consequently, our proposal can consider more instances during its training than the baseline methods. • Word embedding and deep learning can be used to predict top-level ontology classes. • The polysemy of word vectors can be solved by using informal definitions of the word. • Combining different neural network models have better performance than a single one. [ABSTRACT FROM AUTHOR]

In general insurance companies, a correct estimation of liabilities plays a key role due to its impact on management and investing decisions. Since the Financial Crisis of 2007–2008 and the strengthening of regulation, the focus is not only on the total reserve but also on its variability, which is an indicator of the risk assumed by the company. Thus, measures that relate profitability with risk are crucial in order to understand the financial position of insurance firms. Taking advantage of the increasing computational power, this paper introduces a stochastic reserving model whose aim is to improve the performance of the traditional Mack's reserving model by applying an ensemble of Recurrent Neural Networks. The results demonstrate that blending traditional reserving models with deep and machine learning techniques leads to a more accurate assessment of general insurance liabilities. • The proposed Mack-Net model leads to a more accurate reserve prediction. • Mack-Net model generates more appropriate risk measures than Mack's model. • Mack-Net model does not increase the variance to generate an appropriate VaR. • Neural Networks estimate more accurate parameters than usual reserving models. [ABSTRACT FROM AUTHOR]

To combat the recent coronavirus disease 2019 (COVID-19), academician and clinician are in search of new approaches to predict the COVID-19 outbreak dynamic trends that may slow down or stop the pandemic. Epidemiological models like Susceptible–Infected–Recovered (SIR) and its variants are helpful to understand the dynamics trend of pandemic that may be used in decision making to optimize possible controls from the infectious disease. But these epidemiological models based on mathematical assumptions may not predict the real pandemic situation. Recently the new machine learning approaches are being used to understand the dynamic trend of COVID-19 spread. In this paper, we designed the recurrent and convolutional neural network models: vanilla LSTM, stacked LSTM, ED_LSTM, BiLSTM, CNN, and hybrid CNN+LSTM model to capture the complex trend of COVID-19 outbreak and perform the forecasting of COVID-19 daily confirmed cases of 7, 14, 21 days for India and its four most affected states (Maharashtra, Kerala, Karnataka, and Tamil Nadu). The root mean square error (RMSE) and mean absolute percentage error (MAPE) evaluation metric are computed on the testing data to demonstrate the relative performance of these models. The results show that the stacked LSTM and hybrid CNN+LSTM models perform best relative to other models. • Various variants of LSTM and CNN deep learning architectures have been designed. • Predicted COVID-19 confirmed cases for India and its four most affected states. • RMSE and MAPE are computed to compare the performance of the designed models. • Experimental results show that predicted cases are very close to actual cases. [ABSTRACT FROM AUTHOR]

• Simple, efficient and low cost accident detection system for PTW. • DSS takes very less time to detect the accident with severity class. • Irrespective of type of rider and vehicle design, the designed system will work well. • Novel accident detection algorithm is devised using ASW technique. • The system could discriminate the real fall state from false-positive cases. The two-wheeler accidents in most populated and developing countries have become vulnerable and six accidents happen every hour on average. This paper proposes an efficient automatic accident detection system that attempts to detect the occurrences of the accidents in powered two-wheelers (PTW) automatically using vehicle-dependent parameters and the physiological parameters of the rider in real-time. The proposed system builds an accident detection system in PTW using three steps namely, critical event detection system, accident detection system, and severity assessment system. The critical event detection system reads the accelerometer sensor values from the On-Board Diagnostic (OBD) unit mounted on the PTW and classifies the state of the vehicle as normal, fall-like, and fall through the enhanced decision tree algorithm. The enhanced decision tree algorithm uses a tanh function to calculate entropy values. The rules are extracted to fix the threshold by pruning the decision tree to identify the fall of the vehicle and the rider. Due to the unstable nature of PTW and the rider, a novel Adaptive Sequence Window algorithm (ASW) is proposed to substantiate and validate the occurrence of accidents based on the sequence of states identified. Once the accident is detected, the Decision Support System (DSS) running on the OBD mounted on the PTW decides the severity of the accident by combining the three parameters namely fall of the vehicle, fall of the rider, and pulse rate of the rider using the first-order predicate logic rules. The enhanced decision tree algorithm outperforms the other classifiers such as naïve Bayes, artificial neural network, and recurrent neural network with an accuracy of 99.8%. The OBD unit mounted on the PTW and the rider's helmet is used to detect the occurrence of accidents automatically along with its severity with less time. The ASW algorithm enables the system to detect the fall of the vehicle and rider within five minutes and prevents false positives. Further, the information can be communicated based on the severity of the accident to the emergency medical service for quick response. [ABSTRACT FROM AUTHOR]

• A novel method for segmentation and sequence learning of human activity sequences. • Exploiting temporal accumulated motion energy of human actions in activity sequences. • Comparison of the ASSL approach with different sequence learning methods. Discovering underlying patterns for predicting future actions from spatio-temporal human activity information is a fundamental component of research related to the development of expert systems in human activity recognition and assistive robotics. Current research focuses on classification or learning representations of activities for various applications. However, not much attention is given to the pattern discovery of activities which have a major role in the prediction of unseen actions. This paper proposes a novel Adaptive Segmentation and Sequence Learning (ASSL) framework which aims at segmenting unlabelled observations of human activities from extracted 3D joint information. Learning from these obtained segments provides information about the underlying patterns of activity sequences needed in predicting subsequent actions. In the proposed method, the temporal accumulated motion energy of body parts in an activity is utilised in the segmentation process to obtain key actions from unlabelled activity sequences since body parts show changes in acceleration and deceleration during an activity. Based on the segments obtained, the temporal sequence of transitions across activity segments are learned by employing a Long Short-Term Memory Recurrent Neural Network. This ASSL technique has been evaluated using both an experimental human activity dataset and a public activity dataset, and achieved a better performance when compared with other techniques including an Auto-regressive Integrated Moving Average, Support Vector Regression and Gaussian Mixture Regression Models in learning to predict patterns of activity sequences. [ABSTRACT FROM AUTHOR]

• Multi-hybrid neural network with EWT extraction and q -DSCID evaluation is pro-posed. • Novel algorithm based on random inheritance formula is advanced and engrafted. • Ameliorated DBLSTMNN, integrates deep bidirectional training configuration. • Variational learning rate arithmetic is put forward to optimize parameter selection. • Results show the high accuracy of the proposed model for crude oil futures. Machine learning algorithms provide feasibility for crude oil price prediction. In this paper, a novel multi-hybrid predictive neural network model is proposed based on complex deep learning algorithm, which integrates empirical wavelet transform, random inheritance formula error correction algorithm, deep bidirectional LSTM neural network and Elman recurrent neural network with variational learning rate. The prediction model is selected according to the sequence frequency after EWT feature extraction, and the prediction results are obtained by separately predicting and reintegrating. On the basis of individual model, the structure of deep bidirectional training, random inheritance formula and variational learning rate are proposed, which further ameliorate the performance of the model and achieve more effective data information capture. Simultaneously, the examination of variational learning rate provides us with a feasible parameter selection. The proposed model achieves high-precision prediction of crude oil futures price, and stands out in the multi-model comparison analysis and q -DSCID synchronous evaluation, with superior prediction accuracy. [ABSTRACT FROM AUTHOR]

• A lightweight and efficient model is proposed for aspect-based sentiment analysis. • A tradeoff is explored between model accuracy and model complexity. • We construct hierarchical gating structures to filter out noisy context words. • The proposed model achieves the state-of-the-art performance with less complexity. Aspect based sentiment analysis (ABSA) is the task of identifying fine-grained opinion polarity towards a specific target in a sentence, which is empowering experts and intelligent systems with enriched interaction capabilities. Most of approaches to date usually capture semantic relations between target and context words based on RNNs (Recurrent Neural Networks) or pre-trained models (e.g. BERT). However, due to computational complexity and size constraints, these models are often hosted in the cloud. Enabling ABSA models to run on resource-constrained end-devices with quick response time is still challenging and not yet well studied. This paper presents distillation network (DNet), a lightweight and efficient sentiment analysis model based on gated convolutional neural networks for on-device inference. Through combining stacked gated convolution with attention mechanism, DNet can distill aspect-aware context information from unstructured text progressively, achieving high performance with less inference latency and reduced model size. Experiments on SemEval 2014 Task 4 and ACL14 Twitter datasets demonstrate that our approach achieves the state-of-the-art performance. Furthermore, compared with the BERT-based model, DNet reduces the model size by more than 50 times and improves the responsiveness by 24 times. [ABSTRACT FROM AUTHOR]

• We apply the reverse subdivision form of BSd for prediction. • BSd can decompose noisy HV time series into several smooth signals. • All decomposed components are simultaneously used as new inputs of an RNN model. • The advantage of applying BSdis shown by comparing it with other common models. This paper presents a recurrent neural network (RNN) which is improved by using an efficient discrete wavelet transform (DWT) for predicting a high-frequency time series. In the combined DWT-RNN model, first, a multiresolution based on B-spline wavelet of high order d (BSd) is used to decompose the time series into several smooth data sets. Therefore, an approximation data set (with low-frequency) and several detail data sets (with high-frequency), with small wave amplitude, are obtained. Then, all decomposed components are used as RNN inputs. The proposed BSd-RNN model can approximate smooth patterns with satisfactory accuracy, and because of the local properties, BSd is a better choice than other common DWT such as Haar and Daubechies of order n (dbn), for preprocessing the high-frequency time series. According to results of performance metrics for predicting four different stock indices, the BSd-RNN model outperforms other common DWT-RNN model such as Haar-RNN and dbn-RNN. Also, the results show the BSd-RNN model outperforms other common artificial neural network (ANN) model such as multilayer feed-forward neural network (FFNN). Finally, The results show that BS3-RNN predicting model has better predictive ability than other compared models which use other wavelets or other ANNs. [ABSTRACT FROM AUTHOR]