Search

For dialogue systems to function effectively, accurate natural language understanding is vital, relying on precise intent recognition and slot filling to ensure smooth and meaningful interactions. Previous studies have primarily focused on addressing each subtask individually. However, it has been discovered that these subtasks are interconnected and achieving better results requires solving them together. One drawback of the joint learning model is its inability to apply learned patterns to unseen data, which stems from a lack of large, annotated data. Recent approaches have shown that using pretrained embeddings for effective text representation can help address the issue of generalization. However, pretrained embeddings are merely trained on corpus that typically consist of commonly discussed matters, which might not necessarily contain domain specific vocabularies for the task at hand. To address this issue, the paper presents a joint model for intent detection and slot filling, harnessing pretrained embeddings and domain specific embeddings using canonical correlation analysis to enhance the model performance. The proposed model consists of convolutional neural network along with bidirectional long short-term memory (BiLSTM) for efficient joint learning classification. The results of the experiment show that the proposed model performs better than the baseline models. [ABSTRACT FROM AUTHOR]

Intent detection and slot filling are core tasks in natural language understanding (NLU) for task-oriented dialogue systems. However, current models face challenges with numerous intent categories, slot types, and domain classifications, alongside a shortage of well-annotated datasets, particularly in Chinese. Therefore, we propose a domain-aware model with multi-perspective, multi-positive contrastive learning. First, we adopt a self-supervised contrastive learning with multiple perspectives and multiple positive instances, which is capable of spacing the vectors of positive and negative instances from the domain, intent, and slot perspectives, and fusing more positive instance information to increase the classification effectiveness of the model. Our proposed domain-aware model defines domain-level units at the decoding layer, allowing the model to predict intent and slot information based on domain features, which greatly reduces the search space for intent and slot. In addition, we design a dual-stage attention mechanism for capturing implicitly shared information between intents and slots. We propose a data augmentation method that adds noise to the embedding layer, applies fine-grained augmentation techniques, and filters biased samples based on a similarity threshold. Our model is applied to real task-oriented dialogue systems and compared with other NLU models. Experimental results demonstrate that our proposed model outperforms other models in terms of NLU performance. [ABSTRACT FROM AUTHOR]

Natural language understanding (NLU) includes temporal text understanding, which can be complex and encompasses temporal common sense understanding. There are many challenges in comprehending common sense within a text. Currently, there is a limited number of datasets containing temporal common sense in English and there is an absence of such datasets specifically for the Arabic language. In this study, an Arabic dataset was constructed based on an available English dataset. This dataset is considered a valuable resource for the Arabic community. Consequently, different multilingual pre-trained language models (PLMs) were applied to both the English and new Arabic datasets. Based on this, the effectiveness of these models in Arabic and English is compared and discussed. After analyzing the errors, a new categorization of errors was proposed. Finally, the ability of the PLMs to understand the input text and predict temporal features was evaluated. Through this detailed categorization of errors and classification of temporal elements, this study establishes a comprehensive framework aimed at clarifying the specific challenges encountered by PLMs in temporal common sense understanding (TCU). This methodology underscores the urgent need for further research on PLMs' capabilities for TCU tasks. [ABSTRACT FROM AUTHOR]

Text summarization is a process of condensing lengthy texts while preserving their essential information. Previous studies have predominantly focused on high-resource languages, while low-resource languages like Thai have received less attention. Furthermore, earlier extractive summarization models for Thai texts have primarily relied on the article’s body, without considering the headline. This omission can result in the exclusion of key sentences from the summary. To address these limitations, we propose CHIMA, an extractive summarization model that incorporates the contextual information of the headline for Thai news articles. Our model utilizes a pre-trained language model to capture complex language semantics and assigns a probability to each sentence to be included in the summary. By leveraging the headline to guide sentence selection, CHIMA enhances the model’s ability to recover important sentences and discount irrelevant ones. Additionally, we introduce two strategies for aggregating headline-body similarities, simple average and harmonic mean, providing flexibility in sentence selection to accommodate varying writing styles. Experiments on publicly available Thai news datasets demonstrate that CHIMA outperforms baseline models across ROUGE, BLEU, and F1 scores. These results highlight the effectiveness of incorporating the headline-body similarities as model guidance. The results also indicate an enhancement in the model’s ability to recall critical sentences, even those scattered throughout the middle or end of the article. With this potential, headline-guided extractive summarization offers a promising approach to improve the quality and relevance of summaries for Thai news articles.

Question matching attempts to determine whether the intentions of two different questions are similar. Recently, with the development of large-scale pretrained DNN (Deep neural network) language models, state-of-the-art question-matching performance has been achieved. However, due to the independent and identically distributed assumption, the performance of these DNN models in real-world scenarios is limited by the adequacy of the training data and the distribution drift between the target and training data. In this study, we propose a novel gradual machine learning (GML)-based approach for Chinese question matching. Beginning with initially labeled instances, this approach gradually labels target instances in order of increasing hardness via iterative factor inference on a factor graph. The proposed solution first extracts diverse semantic features from different perspectives and then constructs a factor graph by fusing the extracted features to facilitate gradual learning from easy to hard. In feature modeling, we extract and model two complementary types of features: 1) TF-IDF-based keyword features, which can capture the shallow semantic similarity between two questions; 2) DNN-based deep semantic features, which can capture the latent semantic similarity between two questions. We model keyword features as unary factors in a factor graph, which define their influence on the matching status of the two questions. The DNN-based features contain global and local features, where the global features correspond to a question pair’s matching probability as estimated by a DNN model, and the local features correspond to the semantic similarity between two neighboring question pairs estimated by their vector representations in a DNN’s embedding space. To facilitate gradual inference, we model the DNN-based global and local features as unary and binary factors, respectively, in a factor graph. Finally, we implement a GML solution for question matching based on an open-sourced GML inference engine. We validated the efficacy of the proposed approach through a comparative study on two open-sourced Chinese benchmark datasets, LCQMC and the BQ corpus. Extensive experiments demonstrate that compared with pure deep learning models, the proposed solution effectively improves the accuracy of question matching, and its performance advantage generally increases with a decrease in labeled training data. Our experiments also demonstrate that the performance of the proposed solution is very robust w.r.t key algorithmic parameters, indicating its applicability in real-world scenarios. In addition, our work on the GML solution is orthogonal to existing deep learning-based question-matching algorithms because our solution can easily accommodates and leverages other deep language models.

Abstract In human speakers’ daily conversations, what we do not say matters. We not only compute the literal semantics but also go beyond and draw inferences from what we could have said but chose not to. How well is this pragmatic reasoning process represented in pre-trained large language models (LLM)? In this study, we attempt to address this question through the lens of manner implicature, a pragmatic inference triggered by a violation of the Grice manner maxim. Manner implicature is a central member of the class of context-sensitive phenomena. The current work investigates to what extent pre-trained LLMs are able to identify and tease apart different shades of meaning in manner implicature. We constructed three metrics to explain LLMs’ behavior, including LLMs-surprisals, embedding vectors’ similarities, and natural language prompting. Results showed no striking evidence that LLMs have explainable representations of meaning. First, the LLMs-surprisal findings suggest that some LLMs showed above chance accuracy in capturing different dimensions of meaning, and they were able to differentiate neutral relations from entailment or implications, but they did not show consistent and robust sensitivities to more nuanced comparisons, such as entailment versus implications and equivalence versus entailment. Second, the similarity findings suggest that the perceived advantage of contextual over static embeddings was minimal, and contextual LLMs did not notably outperform static GloVe embeddings. LLMs and GloVe showed no significant difference, though distinctions between entailment and implication were slightly more observable in LLMs. Third, the prompting findings suggest no further supportive evidence indicating LLM’s competence in fully representing different shades of meaning. Overall, our study suggests that current dominant pre-training paradigms do not seem to lead to significant competence in manner implicature within our models. Our investigation sheds light on the design of datasets and benchmark metrics driven by formal and distributional linguistic theories.

Emojis, integral to digital communication, often encapsulate complex emotional layers that enhance text beyond mere words. This research leverages the expressive power of emojis to predict Myers-Briggs Type Indicator (MBTI) personalities, diverging from conventional text-based approaches. We developed a unique dataset, EmoMBTI, by mapping emojis to specific MBTI traits using diverse posts scraped from Reddit. This dataset enabled the integration of Natural Language Processing (NLP) techniques tailored for emoji analysis. Large Language Models (LLMs) such as FlanT5, BART, and PEGASUS were trained to generate contextual linkages between text and emojis, further correlating these emojis with MBTI personalities. Following the creation of this dataset, these LLMs were applied to understand the context conveyed by emojis and were subsequently fine-tuned. Additionally, transformer models like RoBERTa, DeBERTa, and BART were specifically fine-tuned to predict MBTI personalities based on emoji mappings from MBTI dataset posts. Our methodology significantly enhances the capability of personality assessments, with the fine-tuned BART model achieving an impressive accuracy of 0.875 in predicting MBTI types, which notably exceeds the performances of RoBERTa and DeBERTa, at 0.82 and 0.84 respectively. By leveraging the nuanced communication potential of emojis, this approach not only advances personality profiling techniques but also deepens insights into digital behaviour, highlighting the substantial impact of emotive icons in online interactions. [ABSTRACT FROM AUTHOR]

The different types of implicit reviews should be well understood so that the developed extraction technique can solve all problems in implicit reviews and produce precise terms of aspects and opinions. We propose an aspectbased sentiment analysis (ABSA) method with natural language understanding for implicit reviews based on sentence and word structure. We built a text extraction method using a machine learning algorithm rule with a deep understanding of different types of sentences and words. Furthermore, the aspect category of each review is determined by measuring the word similarity between the aspect terms contained in each review and aspect keywords extracted from Wikipedia. Bidirectional encoder representations from transformers (BERT) embedding and semantic similarity are used to measure the word similarity value. Moreover, the proposed ABSA method uses BERT, a hybrid lexicon, and manual weighting of opinion terms. The purpose of the hybrid lexicon and the manual weighting of opinion terms is to update the existing lexicon and solve the problem of weighting words and phrases of opinion terms. The evaluation results were very good, with average F1-scores of 93.84% for aspect categorization and 92.42% for ABSA. [ABSTRACT FROM AUTHOR]

In human speakers' daily conversations, what we do not say matters. We not only compute the literal semantics but also go beyond and draw inferences from what we could have said but chose not to. How well is this pragmatic reasoning process represented in pre-trained large language models (LLM)? In this study, we attempt to address this question through the lens of manner implicature, a pragmatic inference triggered by a violation of the Grice manner maxim. Manner implicature is a central member of the class of context-sensitive phenomena. The current work investigates to what extent pre-trained LLMs are able to identify and tease apart different shades of meaning in manner implicature. We constructed three metrics to explain LLMs' behavior, including LLMs-surprisals, embedding vectors' similarities, and natural language prompting. Results showed no striking evidence that LLMs have explainable representations of meaning. First, the LLMs-surprisal findings suggest that some LLMs showed above chance accuracy in capturing different dimensions of meaning, and they were able to differentiate neutral relations from entailment or implications, but they did not show consistent and robust sensitivities to more nuanced comparisons, such as entailment versus implications and equivalence versus entailment. Second, the similarity findings suggest that the perceived advantage of contextual over static embeddings was minimal, and contextual LLMs did not notably outperform static GloVe embeddings. LLMs and GloVe showed no significant difference, though distinctions between entailment and implication were slightly more observable in LLMs. Third, the prompting findings suggest no further supportive evidence indicating LLM's competence in fully representing different shades of meaning. Overall, our study suggests that current dominant pre-training paradigms do not seem to lead to significant competence in manner implicature within our models. Our investigation sheds light on the design of datasets and benchmark metrics driven by formal and distributional linguistic theories. [ABSTRACT FROM AUTHOR]

Multi-label unknown intent detection is a challenging task where each utterance may contain not only multiple known but also unknown intents. To tackle this challenge, pioneers proposed to predict the intent number of the utterance first, then compare it with the results of known intent matching to decide whether the utterence contains unknown intent(s). Though they have made remarkable progress on this task, their methods still suffer from two important issues: (1) It is inadequate to extract multiple intents using only utterance encoding; (2) Optimizing two sub-tasks (intent number prediction and known intent matching) independently leads to inconsistent predictions. In this article, we propose to incorporate segment augmentation rather than only use utterance encoding to better detect multiple intents. We also design a prediction consistency module to bridge the gap between the two sub-tasks. Empirical results on MultiWOZ2.3 and MixSNIPS datasets show that our method achieves state-of-the-art performance and significantly improves the best baseline. [ABSTRACT FROM AUTHOR]

Intelligent voice assistants, such as Apple Siri and Amazon Alexa, are widely used nowadays. These task-oriented dialogue systems require a semantic parsing module in order to process user utterances and understand the action to be performed. This semantic parsing component was initially implemented by rule-based or statistical slot-filling approaches for processing simple queries; however, the appearance of more complex utterances demanded the application of shift-reduce parsers or sequence-to-sequence models. Although shift-reduce approaches were initially considered the most promising option, the emergence of sequence-to-sequence neural systems has propelled them to the forefront as the highest-performing method for this particular task. In this article, we advance the research on shift-reduce semantic parsing for task-oriented dialogue. We implement novel shift-reduce parsers that rely on Stack-Transformers. This framework allows to adequately model transition systems on the transformer neural architecture, notably boosting shift-reduce parsing performance. Furthermore, our approach goes beyond the conventional top-down algorithm: we incorporate alternative bottom-up and in-order transition systems derived from constituency parsing into the realm of task-oriented parsing. We extensively test our approach on multiple domains from the Facebook TOP benchmark, improving over existing shift-reduce parsers and state-of-the-art sequence-to-sequence models in both high-resource and low-resource settings. We also empirically prove that the in-order algorithm substantially outperforms the commonly used top-down strategy. Through the creation of innovative transition systems and harnessing the capabilities of a robust neural architecture, our study showcases the superiority of shift-reduce parsers over leading sequence-to-sequence methods on the main benchmark. [ABSTRACT FROM AUTHOR]

Natural Language Processing (NLP) is a branch of artificial intelligence technologies that has made interacting with computers more akin to natural language. This study aimed to define NLP, present its history from the sixties to the present, clarify the fundamental terms used in the field of NLP, as well as identify the constituent elements of NLP technology, the linguistic levels in the field of NLP, the stages involved in NLP technology, and the applications of NLP in library and information science. The study relied on a descriptive-analytical approach in reviewing the intellectual production related to the field of NLP, based on available foreign databases on the Egyptian Knowledge Bank (EKB). The study reached several conclusions, notably anticipating the emergence of numerous graphical user interface software that will allow the use of NLP techniques without the need for encoding, thereby facilitating beginners in applying NLP techniques easily without relying on algorithms. [ABSTRACT FROM AUTHOR]

In the era of large-scale pretrained models, artificial neural networks (ANNs) have excelled in natural language understanding (NLU) tasks. However, their success often necessitates substantial computational resources and energy consumption. To address this, we explore the potential of spiking neural networks (SNNs) in NLU—a promising avenue with demonstrated advantages, including reduced power consumption and improved efficiency due to their event-driven characteristics. We propose the SpikingMiniLM, a novel spiking Transformer model tailored for natural language understanding. We first introduce a multistep encoding method to convert text embeddings into spike trains. Subsequently, we redesign the attention mechanism and residual connections to make our model operate on the pure spike-based paradigm without any normalization technique. To facilitate stable and fast convergence, we propose a general parameter initialization method grounded in the stable firing rate principle. Furthermore, we apply an ANN-to-SNN knowledge distillation to overcome the challenges of pretraining SNNs. Our approach achieves a macro-average score of 75.5 on the dev sets of the GLUE benchmark, retaining 98% of the performance exhibited by the teacher model MiniLMv2. Our smaller model also achieves similar performance to BERTMINI with fewer parameters and much lower energy consumption, underscoring its competitiveness and resource efficiency in NLU tasks. [ABSTRACT FROM AUTHOR]

The project aims to develop an intelligent system that has the potential to significantly improve patient care by examining and supporting the emotional state of people with chronic diseases or undergoing rehabilitation. The system will use advanced technologies such as image analysis, natural language processing and understanding (NLP/NLU), and a personal assistant module. Acting as a virtual companion, this module will support the user in everyday tasks such as setting reminders and managing schedules and provide emotional support through interactive conversations. The project also involves the development of an analytical module that automatically generates analyses and reports based on the collected data. A patient-oriented system will be created during the work to collect essential data and support him in rehabilitation. On the other hand, the system will cooperate with a doctor who can make a preliminary diagnosis and develop further treatment based on the patient's data. As feedback, the patient will receive health reports along with their interpretation and treatment recommendations created by the doctor.

Natural Language Understanding (NLU) components are used in Dialog Systems (DS) to perform intent detection and entity extraction. In this work, we introduce a technique that exploits the inherent relationships between intents and entities to enhance the performance of NLU systems. The proposed method involves the utilization of a carefully crafted set of rules that formally express these relationships. By utilizing these rules, we effectively address inconsistencies within the NLU output, leading to improved accuracy and reliability. We implemented the proposed method using the Rasa framework as an NLU component and used our own conversational dataset AWPS to evaluate the improvement. Then, we validated the results in other three commonly used datasets: ATIS, SNIPS, and NLU-Benchmark. The experimental results show that the proposed method has a positive impact on the semantic accuracy metric, reaching an improvement of 12.6% in AWPS when training with a small amount of data. Furthermore, the practical application of the proposed method can easily be extended to other Task-Oriented Dialog Systems (T-ODS) to boost their performance and enhance user satisfaction. [ABSTRACT FROM AUTHOR]

The article focuses on improving Named Entity Recognition (NER) within Information Extraction (IE), crucial for enhancing precision in Information Retrieval (IR) systems. It introduces Single-Stream Reasoner (SSR), a type-agnostic solution for NER tasks, addressing challenges such as unconventional predictions and inefficient processing, achieving state-of-the-art results across multiple benchmarks and demonstrating efficiency in convergence and low-resource scenarios.

The article focuses on the limitations of current natural language understanding (NLU) models due to redundant hidden neurons and input noise, which hinder consistent performance improvements despite model scaling. Beyond existing approaches, the study proposes an intrinsic method to enhance neuron utility through a cross-model comparative loss.

In this study, we explore the implications of dataset limitations in semantic knowledge-driven machine translation (MT) for intelligent virtual assistants (IVA). Our approach diverges from traditional single-best translation techniques, utilizing a multi-variant MT method that generates multiple valid translations per input sentence through a constrained beam search. This method extends beyond the typical constraints of specific verb ontologies, embedding within a broader semantic knowledge framework. We evaluate the performance of multi-variant MT models in translating training sets for Natural Language Understanding (NLU) models. These models are applied to semantically diverse datasets, including a detailed evaluation using the standard MultiATIS++ dataset. The results from this evaluation indicate that while multivariant MT method is promising, its impact on improving intent classification (IC) accuracy is limited when applied to conventional datasets such as MultiATIS++. However, our findings underscore that the effectiveness of multivariant translation is closely associated with the diversity and suitability of the datasets utilized. Finally, we provide an in-depth analysis focused on generating variant-aware NLU datasets. This analysis aims to offer guidance on enhancing NLU models through semantically rich and variant-sensitive datasets, maximizing the advantages of multi-variant MT. [ABSTRACT FROM AUTHOR]

• Artificial Intelligence (AI) is transforming medical writing, raising ethical and quality concerns. • Med-PaLM 2 achieved 86.5 % on the USMLE but had low scientific consensus about adherence. • While AI tools can undoubtedly enhance literature review, statistical analysis, and translation processes, they faces several criticisms. • Threats include increased research waste, low-quality papers, and fraudulent publications. • Regulation and ethical frameworks are essential for managing AI's impact on science. The debate surrounding the integration of artificial intelligence (AI) into scientific writing has already attracted significant interest in medical and life sciences. While AI can undoubtedly expedite the process of manuscript creation and correction, it raises several criticisms. The crossover between AI and health sciences is relatively recent, but the use of AI tools among physicians and other scientists who work in the life sciences is growing very fast. Within this whirlwind, it is becoming essential to realize where we are heading and what the limits are, including an ethical perspective. Modern conversational AIs exhibit a context awareness that enables them to understand and remember any conversation beyond any predefined script. Even more impressively, they can learn and adapt as they engage with a growing volume of human language input. They all share neural networks as background mathematical models and differ from old chatbots for their use of a specific network architecture called transformer model [ 1 ]. Some of them exceed 100 terabytes (TB) (e.g., Bloom, LaMDA) or even 500 TB (e.g., Megatron-Turing NLG) of text data, the 4.0 version of ChatGPT (GPT-4) was trained with nearly 45 TB, but stays updated by the internet connection and may integrate with different plugins that enhance its functionality, making it multimodal. [ABSTRACT FROM AUTHOR]

Conversational assistants (CAs) and Task‐oriented ones, in particular, are designed to interact with users in a natural language manner, assisting them in completing specific tasks or providing relevant information. These systems employ advanced natural language understanding (NLU) and dialogue management techniques to comprehend user inputs, infer their intentions, and generate appropriate responses or actions. Over time, the CAs have gradually diversified to today touch various fields such as e‐commerce, healthcare, tourism, fashion, travel, and many other sectors. NLU is fundamental in the natural language processing (NLP) field. Identifying user intents from natural language utterances is a sub‐task of NLU that is crucial for conversational systems. The diversity in user utterances makes intent detection (ID) even a challenging problem. Recently, with the emergence of Deep Neural Networks. New State of the Art (SOA) results have been achieved for different NLP tasks. Recurrent neural networks (RNNs) and Transformer architectures are two major players in those improvements. RNNs have significantly contributed to sequence modelling across various application areas. Conversely, Transformer models represent a newer architecture leveraging attention mechanisms, extensive training data sets, and computational power. This review paper begins with a detailed exploration of RNN and Transformer models. Subsequently, it conducts a comparative analysis of their performance in intent recognition for Task‐oriented (CAs). Finally, it concludes by addressing the main challenges and outlining future research directions. [ABSTRACT FROM AUTHOR]

Numerical reasoning-based machine reading comprehension is a challenging task that involves language understanding with arithmetic operations such as addition, subtraction, comparison, and counting. Various studies on numeric-based reading comprehension have been conducted in English, but low-resource languages such as Vietnamese need to be considered more positively. The online COVID-19 news contains much numerical data and is the appropriate data source for this task. To overcome this problem, we propose COVIDROP, the first challenging Vietnamese machine reading comprehension corpus with numerical reasoning for online COVID-19 news articles. The corpus comprises 6594 human-generated question–answer pairs in 841 Vietnamese COVID-19 online news articles. Furthermore, we evaluated the performance of two numerical reasoning-based machine reading comprehension models, NAQANet and NumNet on COVIDROP. NAQANet performed best on the test set with 22.37% exact match (EM) and 26.58% F1. However, human performance (85.47%) is much higher, indicating that the corpus presents a good challenge for future research. Our corpus is available for evaluating numerical reasoning based on machine reading comprehension and question answering. [ABSTRACT FROM AUTHOR]

The Government of India offers various schemes for various classes of citizens. Most of the application forms of schemes to be filled are in English and it is observed that monolingual individuals find it difficult to access and fill the forms. This paper addresses the challenges faced by monolingual individuals in India, particularly the elderly, people with impairments, and those from marginalized communities. The proposed work is to create an interactive system called "Dhvani" voicebot, specifically designed for the Kannada language. It helps users in identifying suitable government schemes and fills forms in English. The proposed system is developed using the RASA chatbot framework and NLP techniques to comprehend user utterances. RNN and SVM algorithms are employed to ensure smooth conversation flow and interaction with the users. To enhance scheme suggestion accuracy, a knowledge graph is created, containing relevant data on government schemes. The intent classification model achieves an accuracy of 97%, indicating its ability to accurately understand user intentions. The integration of a knowledge graph improves the accuracy of scheme identification and suggestion to users. The system automates the process of filling out government scheme forms based on user inputs. Dhvani voicebot system presents a practical solution to address the challenges faced by monolingual individuals in accessing government schemes in India. The high accuracy of intent classification and the use of a knowledge graph contribute to the system's effectiveness. The study suggests that this system can be extended to other languages. An automated tool called "Dhvani" will solve the problem of aged, illiterate and physically challenged persons filling forms in post offices and banks. Most of the schemes, pension funds, cash withdrawal, cash deposit is through these organizations. So. the tool makes the process easier for the above mention persons without the help of others. An intent recognition and interactive tool developed in Kannada Language which is widely spoken in Karnataka, India. The digital resources available in Kannada Language is very sparce. Use of technology like interactive tool, Knowledge graph, RNN and SVM are used in the development of the tool. Government scheme recommendation interactively makes the users to choose the scheme faster in an interactive way. The form is filled automatically and can be edited to rectify mistakes. [ABSTRACT FROM AUTHOR]

Multitask learning (MTL) is a machine learning paradigm where a single model is trained to perform several tasks simultaneously. Despite the considerable amount of research on MTL, the majority of it has been centered around English language, while other language such as Arabic have not received as much attention. Most existing Arabic NLP techniques concentrate on single or multitask learning, sharing just a limited number of tasks, between two or three tasks. To address this gap, we present ArMT-TNN, an Arabic Multi-Task Learning using Transformer Neural Network, designed for Arabic natural language understanding (ANLU) tasks. Our approach involves sharing learned information between eight ANLU tasks, allowing for a single model to solve all of them. We achieve this by fine-tuning all tasks simultaneously and using multiple pre-trained Bidirectional Transformer language models, like BERT, that are specifically designed for Arabic language processing. Additionally, we explore the effectiveness of various Arabic language models (LMs) that have been pre-trained on different types of Arabic text, such as Modern Standard Arabic (MSA) and Arabic dialects. Our approach demonstrated outstanding performance compared to all current models on four test sets within the ALUE benchmark, namely MQ2Q, OOLD, SVREG, and SEC, by margins of 3.9%, 3.8%, 10.1%, and 3.7%, respectively. Nonetheless, our approach did not perform as well on the remaining tasks due to the negative transfer of knowledge. This finding highlights the importance of carefully selecting tasks when constructing a benchmark. Our experiments also show that LMs which were pretrained on text types that differ from the text type used for finetuned tasks can still perform well. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Modern Science is the property of Alcide De Gasperi University of Euroregional Economy and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Self-attention, which allows transformers to capture deep bidirectional contexts, plays a vital role in BERT-like pre-trained language models. However, the maximum likelihood pre-training objective of BERT may produce an anisotropic word embedding space, which leads to biased attention scores for high-frequency tokens, as they are very close to each other in representation space and thus have higher similarities. This bias may ultimately affect the encoding of global contextual information. To address this issue, we propose TOCOL, a TOken-Level COntrastive Learning framework for improving the contextual representation of pre-trained language models, which integrates a novel self-supervised objective to the attention mechanism to reshape the word representation space and encourages PLM to capture the global semantics of sentences. Results on the GLUE Benchmark show that TOCOL brings considerable improvement over the original BERT. Furthermore, we conduct a detailed analysis and demonstrate the robustness of our approach for low-resource scenarios. [ABSTRACT FROM AUTHOR]

Stress is a major issue in modern society. Researchers focus on identifying stress in individuals, linking language with mental health, and often utilizing social media posts. However, stress classification systems encounter data scarcity issues, necessitating data augmentation. Approaches like Back-Translation (BT), Easy Data Augmentation (EDA), and An Easier Data Augmentation (AEDA) are common. But, recent studies show the potential of generative AI, notably ChatGPT. This article centers on stress identification using the DREADDIT dataset and A Robustly Optimized BERT Pretraining Approach (RoBERTa) transformer, emphasizing the use of generative AI for augmentation. We propose two ChatGPT prompting techniques: same-intent and opposite-intent 1-shot intent-aware data augmentation. Same-intent prompts yield posts with similar topics and sentiments, while opposite-intent prompts produce posts with contrasting sentiments. Results show a 2% and 3% performance increase for opposing and same sentiments, respectively. This study pioneers intent-based data augmentation for stress detection and explores advanced mental health text classification methods with generative AI. It concludes that data augmentation has limited benefits and highlights the importance of diverse Reddit data and further research in this field. [ABSTRACT FROM AUTHOR]