Search

Transformers are Deep Neural Networks (DNN) that utilize a self-attention mechanism to capture contextual relationships within sequential data. Unlike traditional neural networks and variants of Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM), Transformer models excel at managing long dependencies among input sequence elements and facilitate parallel processing. Consequently, Transformer-based models have garnered significant attention from researchers in the field of artificial intelligence. This is due to their tremendous potential and impressive accomplishments, which extend beyond Natural Language Processing (NLP) tasks to encompass various domains, including Computer Vision (CV), audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published, spotlighting the Transformer's contributions in specific fields, architectural disparities, or performance assessments, there remains a notable absence of a comprehensive survey paper that encompasses its major applications across diverse domains. Therefore, this paper addresses this gap by conducting an extensive survey of proposed Transformer models spanning from 2017 to 2022. Our survey encompasses the identification of the top five application domains for Transformer-based models, namely: NLP, CV, multi-modality, audio and speech processing, and signal processing. We analyze the influence of highly impactful Transformer-based models within these domains and subsequently categorize them according to their respective tasks, employing a novel taxonomy. Our primary objective is to illuminate the existing potential and future prospects of Transformers for researchers who are passionate about this area, thereby contributing to a more comprehensive understanding of this groundbreaking technology. • The paper presents a comprehensive survey on transformers for deep learning tasks. • The paper conducts a thorough analysis on highly effective models in five domains. • The paper classifies the models based on respective tasks using a proposed taxonomy. • The characteristics of the surveyed models are deeply explored and analyzed. • Future directions and challenges for transformer-based models are deciphered. [ABSTRACT FROM AUTHOR]

Sequential recommendation aims to predict future user interactions by analyzing dynamic patterns within their historical behavior sequences. Deep neural networks have recently become popular for learning representations of these sequences in the time domain. However, representing users' intentions in the time domain faces challenges such as noise in interactions and sparsity of data. Contrastive learning and representation learning in the frequency domain can mitigate these issues from different perspectives. In this paper, to fully integrate time-domain sequence representations, frequency-domain sequence representations and contrastive learning based on them, we propose a model called T ime– F requency C onsistency based contrastive learning for S equential Rec ommendation (TFCSRec). TFCSRec utilizes a time-domain encoder with a fully connected network and a filter network to extract high-order features and catch pure sequential patterns. Then, a learnable frequency-domain encoder with a recurrent neural network is designed to capture sequential characteristics in the frequency-domain space. Finally, TFCSRec combines a recommendation task and two contrastive learning tasks to optimize the two user representation encoders. Its contrastive learning is designed to minimize a contrastive regularization loss and a time–frequency consistency loss, which for the first time is constructed directly on the time-domain sequence representation and the frequency-domain sequence representation. Experiments on five benchmark datasets show that the proposed TFCSRec model outperforms other sequential recommendation models based on deep neural networks. • Design a time-domain encoder with a filter to extract sequential patterns. • Devise a learnable frequency-domain encoder to capture sequential characteristics. • Propose a time–frequency consistency based contrastive loss for the first time. [ABSTRACT FROM AUTHOR]

The aim of this paper is to explore the potential of the class of recurrent neural networks in developing a classification model for the stock market. For this research, the data that replicates the entire Nasdaq stock market limit order book is used. After extracting order book attributes from the raw dataset, feature selection approaches based on conditional entropy and Stochastic Universal Sampling are proposed, in order to highlight potentially informative features for our data classification task. Furthermore, the performances of nine presented recurrent neural network models for classifying the stock market data vector into one of the labels from the set S = {buy, sell, idle} are examined. In this study, the performances of the models based on different networks, namely the gated recurrent unit, the long short-term memory, and the recurrent neural network, are compared. Using a multi-criteria approach it is concluded that the model based on the GRU topology fed with the features selected using the newly proposed feature selection method outperforms other examined models. • The performances of nine different limit order book classification models are examined. • The PROMETHEE method specifies the most suitable model among the observed models. • The proposed feature demonstrates superiority compared to other selection methods. • The models based on different neural network topologies are evaluated. [ABSTRACT FROM AUTHOR]