Search

• This is a comprehensive review of Transformer for Skeleton-based Action Recognition. • This paper proposes a new taxonomy of transformer-style techniques. • This survey aims to help researchers systematically select promising future directions. Skeleton-based action recognition has rapidly become one of the most popular and essential research topics in computer vision. The task is to analyze the characteristics of human joints and accurately classify their behaviors through deep learning technology. Skeleton provides numerous unique advantages over other data modalities, such as robustness, compactness, noise immunity, etc. In particular, the skeleton modality is extremely lightweight, which is especially beneficial for deep learning research in low-resource environments. Due to the non-European nature of skeleton data, Graph Convolution Network (GCN) has become mainstream in the past few years, leveraging the benefits of processing topological information. However, with the explosive development of transformer methods in natural language processing and computer vision, many works have applied transformer into the field of skeleton action recognition, breaking the accuracy monopoly of GCN. Therefore, we conduct a survey using transformer method for skeleton-based action recognition, forming of a taxonomy on existing works. This paper gives a comprehensive overview of the recent transformer techniques for skeleton action recognition, proposes a taxonomy of transformer-style techniques for action recognition, conducts a detailed study on benchmark datasets, compares the algorithm accuracy of standard methods, and finally discusses the future research directions and trends. To the best of our knowledge, this study is the first to describe skeleton-based action recognition techniques in the style of transformers and to suggest novel recognition taxonomies in a review. We are confident that Transformer-based action recognition technology will become mainstream in the near future, so this survey aims to help researchers systematically learn core tasks, select appropriate datasets, understand current challenges, and select promising future directions. [ABSTRACT FROM AUTHOR]

[Display omitted] • We propose a black-box adversarial text generation scheme called TextGuise, which can be used for almost all text classification tasks. The experiments show that TextGuise is superior to the state-of-the-art text adversarial example generation methods in terms of its attack success rate and attack efficiency when the perturbation ratio does not exceed 0.2.The adversarial examples generated using TextGuise retain the semantic information and strong readability of the text. None of the current defense methods can effectively defend against TextGuise. Adversarial examples greatly compromise the security of deep learning models. The key to improving the robustness of a natural language processing (NLP) model is to study attacks and defenses involving adversarial text. However, the current adversarial attack methods still face problems, such as the low success rates of attacks on some datasets, and the existing defense methods can already successfully defend against some attack methods. As a result, such attacks are unable to dig deeper into the flaws of NLP models to inform further defense improvements. Hence, it is necessary to design an adversarial attack method with a wider attack range and stronger performance. Aiming at the advantages and disadvantages of existing methods, this paper proposes a new adaptive black-box text adversarial example generation scheme, TextGuise. First, we design a keyword selection method in which word scores are calculated by combining context semantics to select the appropriate keywords to modify. Second, to maintain semantics, new keyword substitution rules are designed in combination with the characteristics of text and popular text expressions. Finally, the best modification strategy is adaptively selected through a querying model to reduce the magnitudes of disturbances. TextGuise can automatically select replacement keywords and replacement strategies that efficiently generate adversarial examples with good readability for various text classification tasks. Attack experiments conducted with TextGuise on 5 datasets yield high attack success rates that can surpass 80% when the perturbation ratio does not exceed 0.2. In addition, we present and discuss experiments focusing on defense, text similarity, query times, time consumption, etc., to test the attack performance of TextGuise. The results show that our attack method can achieve a good balance among various metrics. [ABSTRACT FROM AUTHOR]

• Sequence-to-sequence constituent parsers are lagging behind task-specific techniques. • We notably improve their accuracy by introducing a powerful neural architecture. • For the first time, we extend their coverage to deal with discontinuous constituent parsing. • Our text-to-parse transducer achieves a competitive performance on the main benchmarks. In order to achieve deep natural language understanding, syntactic constituent parsing is a vital step, highly demanded by many artificial intelligence systems to process both text and speech. One of the most recent proposals is the use of standard sequence-to-sequence models to perform constituent parsing as a machine translation task, instead of applying task-specific parsers. While they show a competitive performance, these text-to-parse transducers are still lagging behind classic techniques in terms of accuracy, coverage and speed. To close the gap, we here extend the framework of sequence-to-sequence models for constituent parsing, not only by providing a more powerful neural architecture for improving their performance, but also by enlarging their coverage to handle the most complex syntactic phenomena: discontinuous structures. To that end, we design several novel linearizations that can fully produce discontinuities and, for the first time, we test a sequence-to-sequence model on the main discontinuous benchmarks, obtaining competitive results on par with task-specific discontinuous constituent parsers and achieving state-of-the-art scores on the (discontinuous) English Penn Treebank. [ABSTRACT FROM AUTHOR]

[Display omitted] Named entity recognition is a critical task in the natural language processing field. Most existing methods for this task can only exploit contextual information within a sentence. However, their performance on recognizing entities in limited or ambiguous sentence-level contexts is usually unsatisfactory. Fortunately, other sentences in the same document can provide supplementary document-level contexts to help recognize these entities. In addition, words themselves contain word-level contextual information since they usually have different preferences of entity type and relative position from named entities. In this paper, we propose a semi-supervised unified framework to incorporate multi-level contexts for named entity recognition. We use bi-directional gated recurrent units and incorporate pre-trained language model embeddings to capture sentence-level contextual information. To incorporate document-level contexts, we propose to capture interactions between sentences via a multi-head self attention network. To mine word-level contexts, we propose an auxiliary task to predict the type of each word to capture its type preference. We jointly train our model in entity recognition and the auxiliary classification task via multi-task learning. We conduct experiments on two widely-used sequence taggers: CRF tagger and boundary tagger. The experimental results on the CoNLL dataset in English, Dutch, and German validate the effectiveness of our method. [ABSTRACT FROM AUTHOR]

Gated recurrent unit (GRU) is a variant of the recurrent neural network (RNN). It has been widely used in many applications, such as handwriting recognition and natural language processing. However, GRU can only memorize the sequential information, but lacks the capability of adaptively paying attention to important parts in the sequences. In this paper, we propose a novel RNN model, called recurrent attention unit (RAU), which can seamlessly integrate the attention mechanism into the interior of the GRU cell by adding an attention gate. The attention gate enhances the ability of RAU to remember long-term information and pay attention to important parts in the sequential data. Extensive experiments on adding problem, image classification, sentiment classification and language modeling show that RAU consistently outperforms GRU and other related models. [ABSTRACT FROM AUTHOR]

Pre-trained language models (PrLMs) have achieved great success on a wide range of natural language processing tasks by virtue of the universal language representation ability obtained by self-supervised learning on a large corpus. These models are pre-trained on standard plain texts with general language model (LM) training objectives, which would be insufficient to model dialogue-exclusive attributes like specificity and informativeness reflected in these tasks that are not explicitly captured by the pre-trained universal language representations. In this work, we propose dialogue-adaptive pre-training objectives (DAPO) derived from quality estimation to simulate dialogue-specific features, namely coherence, specificity, and informativeness. As the foundation for model pre-training, we synthesize a new dialogue corpus and build our training set with two unsupervised methods: 1) coherence -oriented context corruption, including utterance ordering, insertion, and replacement, to help the model capture the coherence inside the dialogue contexts; and 2) specificity -oriented automatic rescoring, which encourages the model to measure the quality of the synthesized data for dialogue-adaptive pre-training by considering specificity and informativeness. Experimental results on widely used open-domain response selection and quality estimation benchmarks show that DAPO significantly improves the baseline models and achieves state-of-the-art performance on the MuTual leaderboard, verifying the effectiveness of estimating quality evaluation factors into pre-training. [ABSTRACT FROM AUTHOR]

Numerical Table-Text Question Answering aims to predict the program over heterogeneous tabular and textual information, which has recently attracted strong attention in the NLP research community. Current state-of-the-art methods typically use a Seq2Seq-based model to predict programs, which may suffer from the inherent limitations of error propagation. Also, existing models just combine table-text context information and previously predicted operation when reasoning. This approach may lose information over long distances especially when the program is long, and fail to exploit the inter-dependencies between operations in the program. In this paper, we propose a joint Reason Path Driven and Global–Local Fusion Network (ReasonFuse) to alleviate the above problems. First of all, inspired by the intuition that operator prediction is a classification task while operand prediction is an extraction task, we propose a reason path driven architecture that predicts operators first and then guides the operands prediction jointly. After that, the global–local information fusion mechanism is designed to aggregate the context information and global–local program information to mitigate the loss of information over long distances. Last but not least, the soft-dropout mechanism is applied when training to improve the ability and generalization of the model. We conduct extensive experiments on the FinQA and MathQA datasets, our model achieves significant improvements as compared to all published results. The ablation study and case study verify the effectiveness, generalization, and interpretability of our model. [ABSTRACT FROM AUTHOR]

Self-attention networks have revolutionized Natural Language Processing (NLP) and are making impressive strides in image analysis tasks such as image classification and object detection. Inspired by this success, we specifically design a novel self-attention mechanism between local shapes and build a shape Transformer. We split the 3D mesh model into shape patches, which we call shape tokens, and provide polynomial fitting representations of these patches as input to the shape Transformer. The shape token encodes local geometric information and resembles the token (word) status in NLP. The simplification of the mesh model provides a hierarchical multiresolution structure, which allows us to realize the feature learning of a multilayer Transformer. We set high-level features formed by the shape Transformer as visual tokens and propose a vector-type self-attention mechanism to construct a 3D visual Transformer. Finally, we realized a hierarchical network structure based on local shape tokens and high-level visual tokens. Experiments show that our fusion network of 3D shape Transformer with explicit local shape context augmentation and 3D visual Transformer with multi-level structural feature learning achieves excellent performance on shape classification and part segmentation tasks. [ABSTRACT FROM AUTHOR]

• Explore two neural network-based methods to aggregate noisy sequence labels. • Provide a detailed comparison of different methods: probabilistic graphical models, neural networks and methods that combine graphical and neural network models. • Improve true label prediction performance on three real-world datasets from different domains and tasks. • Present a simple but effective method to learn meaningful annotator embeddings. • Demonstrate our annotator embedding is helpful for studying annotators' reliability and behaviour. Labelled data for training sequence labelling models can be collected from multiple annotators or workers in crowdsourcing. However, these labels could be noisy because of the varying expertise and reliability of annotators. In order to ensure high quality of data, it is crucial to infer the correct labels by aggregating noisy labels. Although label aggregation is a well-studied topic, only a number of studies have investigated how to aggregate sequence labels. Recently, neural network models have attracted research attention for this task. In this paper, we explore two neural network-based methods. The first method combines Hidden Markov Models with networks while also learning distributed representations of annotators (i.e., annotator embedding); the second method combines BiLSTM with autoencoders. The experimental results on three real-world datasets demonstrate the effectiveness of using neural networks for sequence label aggregation. Moreover, our analysis shows that annotators' embeddings not only make our model applicable to real-time applications, but also useful for studying the behaviour of annotators. [ABSTRACT FROM AUTHOR]

A growing interest in applying Natural Language Processing (NLP) models to computer vision problems has recently emerged. This interest is motivated by the success of NLP models in tasks such as translation and text summarization. In this paper, we propose a new method for applying NLP to image classification problems. We aim to represent the visual patterns of objects by using a sequence of alphabet symbols and then train a Gated Recurrent Unit (GRU), Long Short-Term Memory (LSTM), or Transformer using these sequences to classify objects. An extensive experimental evaluation using a limited number of images for training has been conducted to compare our method with the ResNet-50 deep learning architecture. The results obtained by the proposed method outperform ResNet-50 in all test scenarios. In one test, the method achieved an average accuracy of 95.3% compared to 89.9% of ResNet-50. The source code (http://git.inovisao.ucdb.br/inovisao/applying-npl-to-image-classification) and dataset (https://doi.org/10.6084/m9.figshare.20055602.v1) are publicly available. [ABSTRACT FROM AUTHOR]

Attacks with adversarial examples can tremendously worsen the performance of deep neural networks (DNNs). Hence, defending against such adversarial attacks is crucial for nearly all DNN-based applications. Adversarial training is an effective and extensively adopted approach for increasing the robustness of DNNs in which benign examples and their adversarial counterparts are considered together in the training stage. However, this may result in a decrease in accuracy on benign examples because it does not account for the inter-class distance of benign examples. To overcome the aforementioned dilemma, we devise a novel defense approach named adversarial supervised contrastive learning (ASCL), which combines adversarial training with supervised contrastive learning to enhance the robustness of DNN-based models while maintaining their clean accuracy. We validate the effectiveness of the proposed ASCL approach in the scenario of defending against word substitution attacks by means of extensive experiments on benchmark tasks and datasets. The experimental results show that ASCL reduces the attack success rate to 20% while maintaining the accuracy for clean inputs within a 2% margin. [ABSTRACT FROM AUTHOR]

Sentence pair modeling is a fundamental yet challenging issue for feature mining in natural language processing (NLP) tasks. Recently, most works have generated feature and sentence representation based on the interactive attention mechanism. However, these models have two limitations: (1) they only consider global information through attention coefficient weighting, which makes insufficient utilization of critical features; (2) they only conduct internal training by fine-tuning network parameters, in which attention results are poorly explained. In this paper, inspired by human reasoning, we propose a Commonality Aggregated approach (CA) to enhance the lightweight interaction model by considering phrase features and contextual words. Specifically, we first fuse positional encoding and employ supervised training to extract critical phrase information from the text as the commonality of sentence pairs. Then, we deploy transfer learning and utilize interaction network to combine crucial phrase features, core word features, and positional encoding to enhance sentence pair modeling. Compared with the original network, extensive experiments on multiple benchmark datasets demonstrate the effectiveness of the proposed commonality aggregated method with stronger competitiveness. Further visual analysisanalysies validated the more explicit interpretability of attention, and extended experimental results indicate the excellent generalization of our approach. [ABSTRACT FROM AUTHOR]

Named Entity Recognition (NER) is generally regarded as a sequence labeling task, and faces a serious problem when the named entities are nested. Span-based model, which enumerates all possible spans as potential entity mentions in a sentence and classifies them, is straightforward for nested NER but faces negative samples problems. In this paper, we propose a span-based nested NER model with BERT and try to solve the negative samples problems. In view of the phenomenon that there are too many negative samples in all spans, we employ a multi-task learning method, which divides NER task into entity identification and entity classification task. In addition, we propose the entity IoU loss function to focus our model on the hard negative samples. Our model is evaluated on three nested NER datasets: GENIA, ACE2004 and ACE2005, and the results show that our model outperforms other state-of-the-art models with the same pretrained language model, achieving 79.46%, 87.30% and 85.24% respectively in terms of F1 score. [ABSTRACT FROM AUTHOR]

• Propose a novel enhanced distance-aware self-attention network for sentence modeling. • Fusing enhanced semantic information with interactive information for sentence semantic matching. • Analysis of a variety of matching functions and proposed multi-level matching function. • Empirical insights into the effect of distance-aware self-attention on sentence semantic matching tasks. Sentence semantic matching is a core research area in natural language processing, which is widely used in various natural language tasks. In recent years, attention mechanism has shown good performance in deep neural networks for sentence semantic matching. Most of the attention-based deep neural networks focus on sentences interaction which ignore modeling the core semantic of the sentence. In other words, they do not consider the importance of the relative distance of words when modeling the sentence semantics, which leads to deviations in modeling the core semantics of the sentence and unstable sentence interaction. Usually, people tend to associate words that are relatively close together when they read and believe that there is a deeper connection between them. Besides, the current interactive matching method after sentence modeling is relatively simple and it may be inadequate. In this paper, we build a well-performed distance-aware self-attention and multi-level matching model (DSSTM) for sentence semantic matching tasks. By considering the importance of different distance tokens, it can get the better original semantics of sentences and hold interactive matching method in multiple level after sentence modeling. To be specific, given two input sentences, we first encode them as contextual embeddings. Then, the contextual embeddings are handled by enhanced distance-aware self-attention to further strengthen the sentence semantic modeling from the whole and local aspect. At the same time, we apply the co-attention layer to extract cross-sentence interaction features while simplifying all the remaining components. Finally, we fuse them into the multi-level matching function to obtain the aggregation vector and learn divers matching representations, which is helpful to capture the diversity of sentence pairs. We conduct experiments on three sentence semantic matching tasks. Experimental results on these public datasets demonstrate that our model outperforms competitive baseline methods and our model has fewer parameters. Our source code is publicly available at https://github.com/xiaodeng-1/DSSTM. [ABSTRACT FROM AUTHOR]

Machine reading comprehension (MRC), as an important task in natural language processing (NLP), is to automatically answer the question after reading a passage. In this aspect, dominant studies mainly focus on domain-specific models. However, domain-specific models trained only on single domain data often cannot achieve satisfactory performance. Although using data of other domains can bring improvement to some extent, building MRC models specific to each domain also makes deployment more difficult in practice. In this paper, we propose a multi-domain MRC model based on knowledge distillation (KD) with domain interference mitigation. Specifically, we employ KD to train a joint model by simultaneously using the multi-domain data and the output distributions of all domain-specific models. In this way, our joint model can better exploit multi-domain data while enabling simpler deployment at the same time. Moreover, to deal with the gradient conflict caused by using data of different domains, we resort to measuring domain-level gradient similarity, based on which an improved PCGrad (short for p rojecting c onflicting g radients) algorithm with adaptive learning rate is proposed. The algorithm mitigates domain interference to improve our joint model across domains. Experimental results and in-depth analysis demonstrate the effectiveness of our joint model and mitigating domain interference further improves the overall performance of our model on a set of benchmark datasets. [ABSTRACT FROM AUTHOR]

Adversarial attacks in NLP are difficult to ward off because of the discrete and highly abstract nature of human languages. Prior works utilize different word replacement strategies to generate semantic-preserving adversarial texts. These query-based methods, however, have limited exploration of the search space. To fully explore the search space, an improved beam search with multiple random perturbing positions is used. Besides, we use the transferable vulnerability from surrogate models to choose vulnerable candidate words for target models. We empirically show that beam search with multiple random attacking positions works better than the commonly used greedy search with word importance ranking. Extensive experiments on three popular datasets demonstrate that our method can outperform three advanced attacking methods under black-box settings. We provide ablation studies to clearly show the effectiveness of our improved beam search which can achieve a higher success rate than the greedy approach under the same query budget. [ABSTRACT FROM AUTHOR]

Natural language understanding (NLU) has two core tasks: intent classification and slot filling. The success of pre-training language models resulted in a significant breakthrough in the two tasks. The architecture based on autoencoding (BERT-based model) can optimize the two tasks jointly. However, we note that BERT-based models convert each complex token into multiple sub-tokens by the Wordpiece algorithm, which generates an out-of-alignment between the length of the tokens and the labels. This leads to BERT-based models not performing well in label prediction, limiting model performance improvement. Many existing models can address this issue, but some hidden semantic information is discarded during fine-tuning. We addressed the problem by introducing a novel joint method on top of BERT. This method explicitly models multiple sub-token features after the Wordpiece tokenization, contributing to both tasks. Our proposed method effectively extracts contextual features from complex tokens using the Sub-words Attention Adapter (SAA), preserving overall utterance information. Additionally, we propose an Intent Attention Adapter (IAA) to acquire comprehensive sentence features, assisting users in predicting intent. Experimental results confirm that our proposed model exhibits significant improvements on two public benchmark datasets. Specifically, the slot-filling F1 score improves from 96.5 to 98.2 (an absolute increase of 1.7%) on the Airline Travel Information Systems (ATIS) dataset. [ABSTRACT FROM AUTHOR]

Natural Language Processing (NLP) models are known vulnerable to adversarial text attacks. Various word-level attacks have been proposed to modify input words by a pre-calculated word saliency order or the heuristic optimization algorithm. However, the predefined fixed order usually leads to a low attack success ratio, and the heuristic algorithm is often inefficient due to many model queries. In this paper, an Adaptive Gradient-based Word Saliency (AGWS) is proposed to improve the query-efficiency and simultaneously preserve a high attack success ratio for the word-level adversarial text attack. Firstly, the AGWS calculates the word saliency with a one-time gradient to all words instead of iterative queries, so the query-efficiency is enhanced. Secondly, the AGWS adaptively updates the word saliency via the variable neighborhood search algorithm to avoid the fixed modification order, which is significant in improving the attack success ratio. Additionally, the AGWS collect substitutes with a hybrid sememe and synonym strategy to enlarge adversary options. Extensive experiments and ablation studies manifest that the AGWS achieves the highest or second highest attack success ratio with lowest word perturbation percentage and improves the query-efficiency compared with baselines. Besides, the AGWS also shows superiorities in adversarial training and transferability. [ABSTRACT FROM AUTHOR]

Visual question answering (VQA) holds the potential to enhance artificial intelligence proficiency in understanding natural language, stimulate advances in computer vision technologies, and expand the range of practical applications. In the current domain of VQA, single-tower architectures suffer from huge parameter issues, whereas dual-tower architectures face some challenges due to insufficient cross-modal data interactions. To address these issues, we propose a novel Mix-Tower model, which is a simple, lightweight yet effective VQA model. Our model uses the self-attention mechanism Transformer as the base unit. In the pre-training phase, we train the model with only 1.15M data. In addition, we analyze the effect of three factors on the model performance, namely the number of different layers in Transformer, the number of layers in FeedForward Networks, and the combination of different class features. For downstream tasks, the OK-VQA and COCO-QA datasets are applied for performance validation. Experimental results show that our model outperforms the best-known baselines by 7.61% and 7.89% on the OK-VQA and COCO-QA datasets, respectively. Meanwhile, our minimal model has only 35M parameters, significantly smaller than the other baseline models. Various results demonstrate that our model achieves lightweight while maintaining superior performance. Our codes is available at: https://github.com/jianruichen/MixTower. • The Mix-Tower model combines the strengths of both single-tower and dual-tower models. • The same model architecture is used to process different types of data. • A lightweight FFN module has been incorporated into the Transformer. • Compared to other baseline models, our model achieves superior performance. [Display omitted] [ABSTRACT FROM AUTHOR]

Recently, graph-based deep learning models have exhibited remarkable performance in generating solution expressions for the math word problem (MWP). However, most of these models have not taken into account the limitations and errors in constructing prior knowledge graphs, which may affect their accuracy and reliability in practical applications. In addition, during graph learning, they focus on extracting information from each given graph, while neglecting the adaptability and unification of graph representation learning. In this paper, we propose a novel multi-view graph learning-to-tree model with dual-strategy (MVG-DS-T), in which it performs adaptive and consistent multi-view representation learning through two benchmark graphs. Specifically, we construct benchmark graphs via semantic dependency parsing of MWP text, considering both semantic and quantitative aspects, i.e., semantic graph and quantitative graph. Then, the reconstruction strategy is employed to reconstruct the structure of the benchmark graphs to capture the adaptive representation information suitable for downstream tasks, while the alignment strategy is utilized to overcome the limitation of independent view representations by unifying the semantic and quantity embedding information through graph structure. Also, an adaptive length normalized loss balancing term for the tree-based decoder is introduced to control the model focus on label length during training, resulting in better equation generation. Extensive experiments demonstrate the effectiveness of the proposed approach on the MWP task. The empirical results show that MVG-DS-T achieves performance comparable to that of the state-of-the-art graph-based models in the existing literature. [ABSTRACT FROM AUTHOR]

The realm of aspect-based sentiment analysis (ABSA), which delves into the nuanced sentiment expressions individuals hold towards specific services or products, has demonstrated immense potential in real-world applications. Recently, ABSA has evolved into the development of aspect-based sentiment quadruple extraction (ASQP). ASQP's objective is to predict four crucial sentiment elements: aspect , sentiment , opinion and category , where such comprehensive approach paints a holistic description of sentiment, facilitating downstream applications. However, prevailing ASQP models suffer from various limitations, such as inefficiency in decoding, inadequate handling of implicit aspects and opinions, and underutilization of structural information. In this paper, we explore an innovative approach to enhance ASQP. Firstly, we adopt a pointer-based non-autoregressive generative framework, enabling the parallel generation of all sentiment quadruples. This approach preserves the advantages of generative methods while significantly boosting decoding efficiency. Additionally, we introduce latent variable learning to model the aspect and opinion elements, effectively enhancing our ability to reason about implicit ASQP components. Furthermore, we propose an aspect-and-opinion-guided latent structure to bolster sentiment-aware context learning. This dynamically induced graph structure adapts to the specific requirements of the task, offering optimal support for ASQP. Our method outperforms current state-of-the-art models on four benchmark ASQP datasets, demonstrating its significant superiority. A detailed analysis highlights the benefits of non-autoregressive decoding in achieving high inference efficiency, the effectiveness of the variational module in capturing implicit sentiment elements, and the value of the dynamically induced latent structure in accurate sentiment feature learning. Moreover, our system excels in producing interpretable predictions. • We for the first time model the task of aspect-based sentiment quadruple extraction (ASQP) with a non-autoregressive generative paradigm with a pointer network, generating all the sentiment quadruples in parallel, By this, we keep the advantage of the generative scheme meanwhile advancing in high decoding efficiency. • We propose to model the aspect and opinion elements as latent variables, effectively enhancing the learning of implicit ASQP elements via unsupervised aspect-opinion reasoning. • We introduce an aspect&opinion-guided latent structure to aid the sentiment-aware context learning, which is dynamically induced according to the end task's need for offering the best helpfulness. • Our framework outperforms the current best-performing baseline systems on the benchmark datasets of ASQP. Further in-depth analyses are conducted for a better understanding of our method. [ABSTRACT FROM AUTHOR]

As an important foundational task in the field of natural language processing, the Chinese named entity recognition (NER) task has received widespread attention in recent years. Self-distillation plays a role in exploring the potential of the knowledge carried by internal parameters in the BERT NER model, but few studies have noticed the impact of different granularity semantic information during the distillation process. In this paper, we propose a multi-level semantic enhancement approach based on self-distillation BERT for Chinese named entity recognition. We first design a feasible data augmentation method to improve the training quality for handling complex entity compositions, then construct a boundary smoothing module to achieve the model's moderate learning on entity boundaries. Besides, we utilize the distillation reweighting method to let the model acquire balanced entity and context knowledge. Experimental results on two Chinese named entity recognition benchmark datasets Weibo and Resume have 72.09% and 96.93% F1 scores, respectively. Compared to three different basic distillation BERT models, our model can also produce better results. The source code is available at https://github.com/lookmedandan/MSE. [ABSTRACT FROM AUTHOR]

As one of the key research topics of scientific text mining, relation extraction for scientific text has received great attention. Unfortunately, existing scientific relation (SRE) suffers from the pipeline extraction scheme and under-exploration of structural interactions, thus hindering the improvement of task performance. In this paper, we introduce an end-to-end SRE model that effectively addresses these challenges. Our approach first conceptualizes the task as a grid-tagging paradigm, enabling the extraction of all entities, attributes, and relations in a single step. Additionally, we incorporate syntactic dependency structure features into the task. To achieve this, we devise a novel label-aided graph convolutional network model, which simultaneously encodes both the topological structure and edge label features for best resolving the long-range dependence issue. Moreover, we propose two interactive learning mechanisms: intra-structure argument correlation and inter-structure argument correlation. These mechanisms significantly enhance the modeling of interactions within and between relational triplet structures, effectively improving the task performance. Empirically, our system outperforms existing methods and achieves new state-of-the-art performances on two benchmark datasets. Further in-depth analyses are shown to shed light on our approach's strengths. • We for the first time propose an end-to-end framework for the Scientific Relation Extraction task with a grid-tagging scheme. This advancement facilitates the construction of high-performance, high-quality scientific knowledge graphs. • We introduce the integration of semantic dependency structure features and devise a novel label-aided graph convolutional network model to simultaneously capture topological structure and edge label features, effectively addressing long-range supervision challenges. • We devise two interactive learning mechanisms, which significantly enhance the modeling of interaction within and between the relational triplet structures, resulting in improved performance. [ABSTRACT FROM AUTHOR]

The integration of Deep Neural Networks (DNN) with Natural Language Processing (NLP) technologies has opened new avenues in financial market prediction, particularly through the utilization of textual information. This study represents a significant advancement, which offers two primary contributions to stock trend prediction: (i) the exploitation of textual data (news, comments, microblogs) using advanced DNN architectures, enhancing market information utilization; (ii) significant improvement of the accuracy of predicting the direction of stock volatility by integrating textual and neural network technologies. Meanwhile, we have crawled, filtered, and constructed a large-scale microblog dataset. This dataset includes approximately 114,992 microblog textual data from 40 Science and Technology Innovation Board (STIB) companies in China during 2021. We conducted a comprehensive analysis using various DNN techniques, including Feedback Neural Networks (FNN), Supervised Contrastive Learning (SCL), Cross Entropy (CE), and Dual Contrastive Learning (DualCL), in conjunction with bag of words models, BERT, and Roberta compilers. Our findings reveal that the SCL method, when combined with microblog data, significantly increases prediction accuracy, particularly during the COVID-19 period. Furthermore, we discovered that using a cross-stock dataset enhances the accuracy of all prediction methods, and random allocation of microblog data leads to better results than sequential allocation. Additionally, we compared the efficacy of traditional models like the CAPM, three-factor, and five-factor models against neural network-based methods. Our results suggest a notable superiority of the SCL method in increasing prediction accuracy. Finally, applying our findings to real-world trading strategies, we demonstrated the practical advantages of using the SCL method in trading, evidenced by significant improvements across all performance indicators. [ABSTRACT FROM AUTHOR]

Referring image segmentation is guided by natural language descriptions to separate the target objects in an image. This task is different from semantic segmentation and instance segmentation in that it involves unique challenges such as multimodal information fusion, variability of natural language expressions, and model robustness. In recent years, the emergence of deep learning techniques has led to innovative ideas and methods for solving these problems. We systematically analyze the main challenges of referring image segmentation and summarize the existing solutions. These include strategies such as multimodal fusion, expression query, multimodal pre-training, and robustness. In addition, we provide an overview of several datasets commonly used in referring image segmentation and analyze the performance of various representative approaches in comparison to different datasets, visual backbone models and threshold settings. Our focus also extends to the challenges and future developments in the field of referring image segmentation. Our survey paper will provide a comprehensive technical reference for future researchers. • Examine the challenges associated with referring image segmentation and encapsulate the corresponding solutions. • Based on the constraints of current methods, scrutinize the avenues for further enhancements. • Summarise the existing findings to provide an important reference for future research. [ABSTRACT FROM AUTHOR]

Transformers have become the state-of-the-art architectures for various tasks in Natural Language Processing (NLP) and Computer Vision (CV); however, their space and computational complexity present significant challenges for real-world applications. A promising approach to address these issues is the introduction of sparsity, which involves the deliberate removal of certain parameters or activations from the neural network. In this systematic literature review, we aimed to provide a comprehensive overview of current research on sparsity in transformers. We analyzed the different sparsity techniques applied to transformers, their impact on model performance, and their efficiency in terms of time and space complexity. Moreover, we identified the major gaps and challenges in the existing literature. Our study also highlighted the importance of investigating sparsity in transformers for computational efficiency, reduced resource requirements, scalability, environmental impact, and hardware-algorithm co-design. By synthesizing the current state of research on sparsity in transformer-based models, we also provided valuable insights into their efficiency, impact on model performance, and potential trade-offs, contributing to advancing the field further. [ABSTRACT FROM AUTHOR]

• The current progress of adversarial attack and defense technologies in the NLP field is systematically summarized. • The textual adversarial attacks are classified according to the semantic granularity at the top level, and each class are further classified into subclasses according to the example generation strategy. • The defense strategies against textual adversarial attacks are summarized in the light of passive and active methods. • Significant challenges and potential development directions are concluded from the attack and defense perspective. Recently, the adversarial attack and defense technology has made remarkable achievements and has been widely applied in the computer vision field, promoting its rapid development in other fields, primarily the natural language processing domain. However, discrete semantic texts bring additional restrictions and challenges to successfully implementing adversarial attacks and defenses. This survey systematically summarizes the current progress of adversarial techniques in the natural language processing field. We first briefly introduce the textual adversarial example's particularity, vectorization, and evaluation metrics. More importantly, we categorize textual adversarial attacks according to the combination of semantic granularity and example generation strategy. Next, we present commonly used datasets and adversarial attack applications in diverse natural language processing tasks. Besides, we classify defense strategies as passive and active methods considering both input data and victim models. Finally, we present several challenging issues and future research directions in this domain. [ABSTRACT FROM AUTHOR]

Accurate traffic prediction is critical for enhancing the performance of intelligent transportation systems. The key challenge to this task is how to properly model the complex dynamics of traffic while respecting and exploiting both spatial and temporal heterogeneity in data. This paper proposes a novel framework called Meta Graph Transformer (MGT) to address this problem. The MGT framework is a generalization of the original transformer, which is used to model vector sequences in natural language processing. Specifically, MGT has an encoder-decoder architecture. The encoder is responsible for encoding historical traffic data into intermediate representations, while the decoder predicts future traffic states autoregressively. The main building blocks of MGT are three types of attention layers named Temporal Self-Attention (TSA), Spatial Self-Attention (SSA), and Temporal Encoder-Decoder Attention (TEDA), respectively. They all have a multi-head structure. TSAs and SSAs are employed by both the encoder and decoder to capture temporal and spatial correlations. TEDAs are employed by the decoder, allowing every position in the decoder to attend all positions in the input sequence temporally. By leveraging multiple graphs, SSA can conduct sparse spatial attention with various inductive biases. To facilitate the model's awareness of temporal and spatial conditions, Spatial–Temporal Embeddings (STEs) are learned from external attributes, which are composed of temporal attributes (e.g. sequential order, time of day) and spatial attributes (e.g. Laplacian eigenmaps). These embeddings are then utilized by all the attention layers via meta-learning, hence endowing these layers with Spatial–Temporal Heterogeneity-Aware (STHA) properties. Experiments on three real-world traffic datasets demonstrate the superiority of our model over several state-of-the-art methods. Our code and data are available at (http://github.com/lonicera-yx/MGT). [ABSTRACT FROM AUTHOR]

The pervasiveness of aggressive content in social media has become a serious concern for government organizations and tech companies because of its pernicious societal effects. In recent years, social media has been repeatedly used as a tool to incite communal aggression, spread distorted propaganda, damage social harmony and demean the identity of individuals or a community in the public spaces. Therefore, restraining the proliferation of aggressive content and detecting them has become an urgent duty. Studies of the identification of aggressive content have mostly been done for English and other high-resource languages. Automatic systems developed for those languages can not accurately identify detrimental contents written in regional languages like Bengali. To compensate this insufficiency, this work presents a novel Bengali aggressive text dataset (called 'BAD') with two-level annotation. In level-A, 14158 texts are labeled as either aggressive or non-aggressive. While in level-B, 6807 aggressive texts are categorized into religious, political, verbal and gendered aggression classes each having 2217, 2085, 2043 and 462 texts respectively. This paper proposes a weighted ensemble technique including m-BERT, distil-BERT, Bangla-BERT and XLM-R as the base classifiers to identify and classify the aggressive texts in Bengali. The proposed model can readdress the softmax probabilities of the participating classifiers depending on their primary outcomes. This weighting technique has enabled the model to outdo the simple average ensemble and all other machine learning (ML), deep learning (DL) baselines. It has acquired the highest weighted f 1 -score of 93.43% in the identification task and 93.11% in the categorization task. Dataset developed as the part of this work is available at https://github.com/BAD-Bangla-Aggressive-Text-Dataset [ABSTRACT FROM AUTHOR]

With the rapid development of Internet technology and the explosive growth of digital text, opinion mining has become one of the important research hotspots in the field of natural language processing (NLP). In recent years, neural network based deep learning algorithms have been applied in the field of opinion mining. Considering the relation between temporal and spatial dimensions of text data and the characteristics of natural language itself, traditional deep learning algorithms cannot be comprehensive in the processing of fully feature extraction. In this paper, we propose a new deep learning framework for opinion mining, which includes a temporal feature extraction layer that consists of two layers of bidirectional simple recurrent unit (Bi-SRU) networks extracting features at the word and grammar levels; a semantic feature extraction layer that mainly contains a multi-head attention module; a spatial feature extraction layer with dilated convolution that is used to extract opinion preference features. The Internet movie database (IMDb) is used to verify the performance of the proposed framework. The experiment results show that the proposed framework can effectively improve the classification accuracy, whose performance is better than that of the compared algorithms. [ABSTRACT FROM AUTHOR]

• A review of epistemic (model) uncertainty learning in supervised learning is provided. • The predictive uncertainty is discussed from the prospective generalization error. • A hierarchical categorization of epistemic uncertainty learning techniques are introduced. • Applications of epistemic uncertainty learning in CV and NLP are presented. • Main research gaps along with future research directions are pointed out. Quantifying the uncertainty of supervised learning models plays an important role in making more reliable predictions. Epistemic uncertainty, which usually is due to insufficient knowledge about the model, can be reduced by collecting more data or refining the learning models. Over the last few years, scholars have proposed many epistemic uncertainty handling techniques which can be roughly grouped into two categories, i.e., Bayesian and ensemble. This paper provides a comprehensive review of epistemic uncertainty learning techniques in supervised learning over the last five years. As such, we, first, decompose the epistemic uncertainty into bias and variance terms. Then, a hierarchical categorization of epistemic uncertainty learning techniques along with their representative models is introduced. In addition, several applications such as computer vision (CV) and natural language processing (NLP) are presented, followed by a discussion on research gaps and possible future research directions. [ABSTRACT FROM AUTHOR]

Machine reading comprehension is an important topic in natural language processing. However, the redundancy of text and the diversity of answer types are neglected in existing machine reading comprehension. To tackle these issues, a multi-task joint training scheme is proposed for the multi-type answer machine reading comprehension task. In this scheme, a feature-based paragraph extraction mechanism is first designed in the training stage to extract various text features from answers and sentences. For efficiency, the redundancy of the text is greatly reduced, while the effective information of the text is retained. And then, a reading comprehension module is optimized by enhancing the representation of the pre-trained language model, as well as an answer type classification module is optimized through the capsule network. Finally, a multi-task joint training model is designed to simultaneous obtaining answer text and answer type. Experiments are conducted to evaluate the proposed approach with two open datasets, CJRC and Natural Questions, and demonstrate that our model is efficient and promising, in terms of both the reading comprehension and answer classification tasks. In particular, our model won the best method that is winner of the first prize in the CAIL-2020 machine reading comprehension contest. [ABSTRACT FROM AUTHOR]

Convolutional Neural Networks (CNNs) have won tremendous success in various applications, such as image recognition and natural language processing. Due to the computing-intensive and memory-intensive features of CNN models, it is challenging to deploy them on devices with limited resources and tight power budgets. To address this limitation, pruning, quantization, weights sharing, and other methods have been proposed as efficient solutions to compress CNN models. However, these works usually do not optimize the hardware design when compressing the CNNs. In this paper, we introduce Conv-Inheritance, a hardware-efficient compression method for CNNs by reducing the number of convolution operations to simultaneously reduce the inference time and hardware computing resources on chip of CNN edge devices. We also develop an extended version, called adaptive Conv-Inheritance, to further improve the inference performance by considering the similarity between pixels. We applied our Conv-Inheritance method to compressing various CNN models and performed extensive experiments on different image datasets. Experimental results demonstrate that Conv-Inheritance is both hardware-efficient and time-efficient on edge devices. Besides, Conv-Inheritance is a plug-and-play compression method for CNNs, and we can run Conv-Inheritance on top of other compression methods without any conflicts. [ABSTRACT FROM AUTHOR]

Knowledge-based question answering (KBQA) is an interesting but challenging task in the field of natural language processing. And in recent years, there is increasing interest in introducing deep learning models for answering complex-factoid questions, which associate with multiple facts and require multi-hop inference. However, the complex-factoid question answering mainly faces two challenges: (1) multiple entities are involved in these questions, which bring multiple initial states for knowledge reasoning; (2) a number of complex-factoid questions require the intersection of multiple related sub-paths in knowledge bases, which demands repeated explorations of path reasoning, matching and assembling. To address the above challenges, we propose a motif-based Memory Network for answering complex-factoid questions, which introduces motifs as the basic constituents for semantic representation, and meanwhile includes a specific-designed memory network for knowledge reasoning and matching. Extensive experiments on real datasets demonstrate that our model significantly outperforms the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

This paper shows a computational model that classifies Cybergrooming attacks in the context of COP (child online protection) using Natural Language Processing (NLP) and Convolutional Neural Networks (CNN). The model predicts a high number of false positives, therefore low precision and F-score, but a high accuracy. In this issue, where the number of messages in the context of grooming are so low compared to the number of conversations and messages from other contexts, it can be concluded that is a very consistent and useful result as it captures a high number of true positives, considering that the classifier works for messages. Performing the training of machine learning algorithms with neural networks, semantic analysis and NLP, allows approximate representation of knowledge contributing to discovery of pseudo-intelligent information in these environments and reducing human intervention for characterization of underlying abnormal behavior and detecting messages that potentially represent these attacks. [ABSTRACT FROM AUTHOR]

Event Detection (ED) is a pivotal sub-task of Event Extraction(EE). It aims to locate triggers and categorize them into specific event types. Recent researches on ED have shown that graph convolutional neural networks with syntactic information can achieve advanced performance. However, these methods ignore the implicit importance score of tokens. This will weaken their ability of identifying trigger words. In addition, due to the long-tailed distribution in the corpus, previous methods perform poorly on sparsely labeled trigger words and are prone to overfitting on densely labeled ones. In this paper, we propose a Syntax-Enhanced GCN framework with a Decoupled Classification Rebalance mechanism (SEGCN-DCR) to address the above issues. Specifically, we exploit a tree-structured module based on dependency structure to reduce the noise by capturing global hierarchical syntactic information, and DCR mechanism to rescale the classifier weights, which makes classifier decision boundaries more reasonable. Experiments on benchmark ACE2005 show that the proposed method acquires state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Graph Convolutional Networks (GCNs) are general deep representation learning models for graph-structured data. In this paper, we propose a simple Plug-in Attention Module (PLAM) to improve the representation power of GCNs, inspired by the recent success of the query-key mechanism in computer vision and natural language processing. With this module, our network is able to adaptively learn the weights from a node towards its neighbors. Different from existing attention-based GCNs, the proposed PLAM has several important properties. First, the parameter space for the attention module is isolated from that for feature learning. This ensures that the proposed approach can be conveniently applied to existing GCNs as a plug-in module. Second, the anchor node and neighbor nodes are treated separately when learning the attention weights, which further enhances the flexibility of our structure. Third, our attention module extracts higher-level information by computing the inner product of the features between the anchor node and neighbor nodes, leading to significantly increased representation power. Last, we take a step forward and propose a novel structural encoding technique for the graph attention module to inject local and global structure information. Although being simple, our PLAM models have achieved state-of-the-art performances on graph-structured datasets under both the transductive and inductive settings. Additionally, experiments on image and point cloud datasets show potential applications of PLAM on several computer vision tasks. [ABSTRACT FROM AUTHOR]

Named Entity Recognition(NER), one of the most fundamental problems in natural language processing, seeks to identify the boundaries and types of entities with specific meanings in natural language text. As an important international language, Chinese has uniqueness in many aspects, and Chinese NER (CNER) is receiving increasing attention. In this paper, we give a comprehensive survey of recent advances in CNER. We first introduce some preliminary knowledge, including the common datasets, tag schemes, evaluation metrics and difficulties of CNER. Then, we separately describe recent advances in traditional research and deep learning research of CNER, in which the CNER with deep learning is our focus. We summarize related works in a basic three-layer architecture, including character representation, context encoder, and context encoder and tag decoder. Meanwhile, the attention mechanism and adversarial-transfer learning methods based on this architecture are introduced. Finally, we present the future research trends and challenges of CNER. [ABSTRACT FROM AUTHOR]

Sarcasm prevalent in social media poses challenges for sentiment analysis applications by flipping polarity, thus increasing the demand for sarcasm detection. In this article, we present a systematic survey of the research on sarcasm detection. We discuss the definition, problem formulation, datasets, as well as comprehensively review and evaluate methods that can detect sarcasm from three perspectives: the incongruity it contains, the sentimental cues it conveys, and the commonsense knowledge it implies. Specially, we detail sarcasm-related fundamental theories across disciplines, which may enhance sarcasm detection by leveraging interdisciplinary research and hope to facilitate collaborative efforts across research fields. We also discuss a variety of open problems, along with future opportunities for sarcasm detection. [ABSTRACT FROM AUTHOR]

Deep learning (DL) has been shown to be very effective for many application domains of machine learning (ML), including image classification, voice recognition, natural language processing, and bioinformatics. The success of DL techniques is directly related to the availability of large amounts of training data. However, in many cases, the data are sensitive to the users and should be protected to preserve the privacy. Privacy-preserving deep learning (PPDL) has thus become a very active research field to ensure the training process and use of DL models are productive without exposing or leaking information about the data. This paper aims to provide a comprehensive survey of PPDL. We concentrate on the risks that affect data privacy in DL and conduct a detailed investigation into the models that ensure privacy. Finally, we propose a set of evaluation criteria, detailing the advantages and disadvantages of the solutions. Based on the analyzed strengths and weaknesses, the paper has highlighted some important research problems and application cases that have not been studied and these point to certain open research directions. [ABSTRACT FROM AUTHOR]

Abstractive dialogue summarization requires distilling and rephrasing key information from noisy multi-speaker documents. Combining pre-trained language models with input augmentation techniques has recently led to significant research progress. However, existing solutions still struggle to select relevant chat segments, primarily relying on open-domain and unsupervised annotators not tailored to the actual needs of the summarization task. In this paper, we propose DearWatson , a task-aware utterance-level annotation framework for improving the effectiveness and interpretability of pre-trained dialogue summarization models. Precisely, we learn relevant utterances in the source document and mark them with special tags, that then act as supporting evidence for the generated summary. Quantitative experiments are conducted on two datasets made up of real-life messenger conversations. The results show that DearWatson allows model attention to focus on salient tokens, achieving new state-of-the-art results in three evaluation metrics, including semantic and factuality measures. Human evaluation proves the superiority of our solution in semantic consistency and recall. Finally, extensive ablation studies confirm each module's importance, also exploring different annotation strategies and parameter-efficient fine-tuning of large generative language models. [ABSTRACT FROM AUTHOR]

Abstractive question-answering (QA) has emerged as a prominent area in Natural Language Processing (NLP) due to its ability to produce concise and human-like responses, particularly with the advancement of Large Language Models. Despite its potential, abstractive QA suffers from challenges like the need for extensive training data and the generation of incorrect entities and out-of-context words in the responses. In safety–critical domains like medical and clinical settings, such issues are unacceptable and may compromise the accuracy and reliability of generated answers. We proposed KI-MAG (Knowledge-Infused Medical Abstractive Generator) model, a novel Knowledge-Infused Abstractive Question Answering System specifically designed for the medical domain. KI-MAG aims to address the aforementioned limitations and enhance the correctness of generated responses while mitigating data sparsity concerns. The KI-MAG system produces more precise and informative answers by incorporating relevant medical entities into the model's generation process. Furthermore, we adopt a synthetic data generation approach using question–answer pairs to overcome the challenge of limited training data in the medical domain. These synthetic pairs augment the original dataset, resulting in better model generalization and improved performance. Our extensive experimental evaluations demonstrate the effectiveness of the KI-MAG system. Compared to traditional abstractive QA models, our approach exhibits a substantial increase of approximately 15% in Blue-1, Blue-2, Blue-3, and Blue-4 scores, indicating a remarkable improvement in answer accuracy and overall quality of responses. Overall, our Knowledge-Infused Abstractive Question Answering System in the Medical Domain (KI-MAG) presents a promising solution to enhance the performance and reliability of abstractive QA models in safety–critical medical applications where precision and correctness of answers are of utmost importance. [ABSTRACT FROM AUTHOR]

Multilingual neural machine translation (MNMT) has attracted more and more attention in recent days because it can use a single neural machine translation (NMT) model to translate between multiple languages. As several languages are involved in MNMT, recent studies have shown that using part of these languages rather than all of them to train the model leads to comparable results. However, previous work on this topic mainly focuses on language clustering and features defined by linguists. The semantic relationship and language distance are not fully considered. How to select the most related language pairs to current low-resource pair to optimize the performance of MNMT is still an open question. In this paper, we propose to take language relatedness computation as a ranking problem, where features such as language distance, linguistic typological information and semantic relatedness features are incorporated into a random decision forest to improve the language relatedness evaluation (LRE) for MNMT. Since the model only focuses on monolingual LRE in general cross-lingual natural language processing tasks, we also propose two features related to machine translation (data size and bilingual relatedness) to predict the final language pairs. Experimental results on IWSLT and WMT datasets show that our proposed LRE method can achieve significant improvements compared with other models. We also conducted several groups of experiments on IWSLT and WMT datasets to further evaluate the effectiveness of the proposed method on MNMT. The results show that the MNMT model trained on language pairs predicted by the LRE method outperforms other language selection methods. [ABSTRACT FROM AUTHOR]

Variational recurrent autoencoder model (VRAE) is an appealing technique for capturing the variabilities underlying complex sequential data, which is realized by introducing high-level latent random variables as hidden states. Existing models suffer from the well-known 'posterior collapse' problem, meaning that a powerful autoregressive decoder equipped in the model itself could capture all the variabilities and hence leave the latent variables learning nothing from the data. From the perspective of model training, the posterior collapse problem can result in a very low Kullback–Leibler divergence (KL-divergence) value, which means the posteriors of the latent variables tend to be just the priors. In this paper, we address this problem by proposing a Bayesian variational recurrent neural network (BVRNN) model, in which two additional decoders are added into the original VRAE. These extra decoders can force the latent variables to learn meaningful knowledge during the training process. We conduct experiments on MNIST and Fashion-MNIST dataset. The experimental results show that the proposed model outperforms several baseline models. We further adapt the proposed model to a very challenging task in natural language processing, namely Named-Entity Recognition (NER). Experimental results show that our model is competitive to the state-of-the-art models on NER. [ABSTRACT FROM AUTHOR]

Empathy involves comprehending and sharing the emotions of another person. In the realm of conversational AI, empathy pertains to the AI's capacity to understand and respond suitably to the user's emotions and needs. Conversational AI with empathetic capabilities can heighten the user experience by making interactions more personalized and natural. At present, machine learning algorithms are commonly utilized in existing conversational AI systems to recognize emotions and corresponding empathetic intents from annotated data. Nonetheless, this approach is not without limitations, being expensive and time-consuming. Our present work takes a holistic approach to empathy in conversational AI, where we propose a novel zero-shot multitask framework, the Zero-shot Intent Emotion Detection (ZIED) network, identifies both emotions and intents in a multimodal setting. We developed an end-to-end model that concurrently captures textual, audio, and visual representations and integrates the different modalities using cross-attention mechanisms. Our experimental results, based on the EmoInt-MD dataset, show that incorporating all three modalities results in the best performance for both emotion and empathetic intent detection. We observed a noteworthy improvement of over 6% and 4% for intent and emotion, respectively, for various ratios of seen and unseen classes. [ABSTRACT FROM AUTHOR]

Fine-grained action recognition is one of the critical problems in video processing, which aims to recognize similar actions of subtle interactions between humans and objects. Inspired by the remarkable performance of the Transformer in natural language processing, Transformer has been applied to the fine-grained action recognition task. However, Transformer needs abundant training data and extra supervision to achieve comparable results with convolutional neural networks (CNNs). To address these issues, we propose a Convolutional Transformer Network (CTN), which integrates the merits of CNN (e.g. , sharing weights, capturing low-level features in videos and locality) and the benefits of Transformer (e.g., dynamic attention and learning long-range dependencies). In this paper, we propose two modifications to the original Transformer: (i) We propose a video-to-tokens module that can extract tokens from extracted spatial-temporal features in videos by 3D convolutions instead of the direct token embedding from raw input video clips; (ii) We completely replace the linear mapping in multi-head self-attention layer with depth-wise convolutional mapping, which applies a depth-wise separable convolution operation on embedded token maps. With these two modifications, our approach can extract effective spatial-temporal features from videos and process the long sequences of tokens encountered in videos. Experimental results demonstrate that our proposed CTN can achieve state-of-the-art accuracy on two fine-grained action recognition datasets (i.e., Epic-Kitchens and Diving 48) with a small computational increase. [ABSTRACT FROM AUTHOR]

As an important development direction of natural language processing, emotion recognition in conversation (ERC) remains a challenge in sentiment analysis. Given the large-scale dialogue datasets and their wide application in the fields of recommendation systems and human–machine dialogue systems, researchers have begun to pay more attention to the issue of ERC. In recent research, the task of ERC has been largely based on the graph structure to model the speaker level. However, most existing studies simply splice multimodal features, and the heterogeneity of multimodal features tends to be overlooked. Hence, this paper proposes a multivariate messaging framework to embed heterogeneous information into multimodal relational graphs. In the process of aggregating graph node information, we take into account the homogeneity of nodes and assign different weights to different nodes so as to better aggregate semantic information. In order to improve the robustness of the model, we utilize the mechanism of sharing weights among neighbors to reduce the number of network parameters and improve the generalization ability of the model. In so doing, the node information is aggregated through the constructed graph network, and the final semantic vector representation is obtained. Experiments over two benchmark datasets for ERC show that our proposed model achieves improved performance in accuracy and F1 value. [ABSTRACT FROM AUTHOR]

Position encoding has recently been shown to be effective in transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding (RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in the self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: https://huggingface.co/docs/transformers/model_doc/roformer. [ABSTRACT FROM AUTHOR]

Every day, enormous amounts of biomedical texts discussing various biomedical topics are produced. Revealing strong semantic connections hidden in those unstructured data is essential for many interesting applications such as knowledge base development for the biomedical domain as well as drug repurposing and drug–disease associations. Literature based discovery (LBD) is a well-known paradigm that refers to the issues of finding new hidden knowledge in scientific literature by connecting pieces of semantically-related information belonging to independent documents. This challenging research area has been extensively investigated by the research community and different proposals adopting natural language processing, text mining, machine learning and recently deep learning have been developed. This paper exploits a very focused task, it surveys a collection of research papers published in the recent years that have adopted Deep Learning for literature based discovery as an effective technique to discover new relationships between existing knowledge in biomedical domain. The study provides an analysis of the key characteristics of each work surveyed, including the Literature based discovery application area, the deep learning method used, the type of analyzed data, and the results obtained. Recognizing the significance of Pre-trained Language Models (PLMs), another primary aim of this paper is to offer an extensive overview of the latest developments in pre-trained language models within the field of biomedicine. This focus will primarily be on how they are applied to downstream tasks associated with Literature-Based Discovery in the biomedical domain. Additionally, the survey highlights the key drawbacks of the current state-of-the-art proposals, as well as the challenges that require further study by the research community. [ABSTRACT FROM AUTHOR]

Official gazettes are documents published by governments to publicize their actions, spanning long periods of time and making an important transparency mechanism. These documents have information on laws, contracts, and bidding processes, as well as on civil servants and their careers in public service. Automatic information extraction of these documents may contribute to public transparency, with two tasks being especially useful: the classification of the different segments of these documents, the so called acts; and the Named Entity Recognition (NER) within the acts. The variety of official gazettes and their patterns brings up the necessity of constructing different tools for specific gazettes. In this paper, we propose DODFMiner, a command-line interface tool to classify acts and extract named entities from the Official Gazette of the Federal District. The tool follows a 3-step approach: the pre-processing of the input data; text classification using rule-based systems with regular expressions; and NER with Machine Learning algorithms. It allows users to input JSON files and receive CSV as output, providing information that allows users to track government procurements through years, contracts duration and total amount, among others. We also propose a set of experiments to support the choice of models included in the tool, covering the classification and NER steps. Text classification achieved a mean F1-score of 0.778, while to the NER, we compared 3 different architectures, CRF with a mean F1-score of 0.851, CNN-biLSTM-CRF with 0.787 and CNN-CNN-LSTM with 0.841. [ABSTRACT FROM AUTHOR]