Search

Text summarization is a widely-researched problem among scholars in the field of natural language processing. Multiple techniques have been proposed to help tackle this problem, yet some of these methodologies may still exhibit limitations such as the requirements for large training datasets, which might not always be possible, but more importantly, the lack of interpretability or transparency of the model. In this paper, we propose using meta-learning algorithm to train a deep learning model for extractive text summarization and then using various explanatory techniques such as SHAP (Shapley, 1953), linear regression (Lederer, 2022), decision trees (Fürnkranz, 2010), and input modification to gain insights into the model's decision making process. To evaluate the effectiveness of our approach, we will compare it to other popular natural language processing models like BERT (Miller, 2019) or XLNET (Yang et al., 2020) using the ROUGE metrics (Lin, 2004). Overall, our proposed approach provides a promising solution to the limitations of existing methods and a framework for improving the explainability of deep learning models in other natural language processing tasks. • Bi-LSTM and Meta-learning produce better results compared to other popular models. • Meta-learning can be utilized to improve performance in data-scarce scenarios. • Use SHAP, regression, decision tree, and text mutation to enhance interpretability. [ABSTRACT FROM AUTHOR]

Deep learning has emerged as a powerful tool in various domains, revolutionising machine learning research. However, one persistent challenge is the scarcity of labelled training data, which hampers the performance and generalisation of deep learning models. To address this limitation, researchers have developed innovative methods to overcome data scarcity and enhance deep model learning capabilities. Two prevalent techniques that have gained significant attention are transfer learning and self-supervised learning. Transfer learning leverages knowledge learned from pre-training on a large-scale dataset, such as ImageNet, and applies it to a target task with limited labelled data. This approach allows models to benefit from the learned representations and effectively transfer knowledge to new tasks, resulting in improved learning performance and generalisation. On the other hand, self-supervised learning focuses on training models using pretext tasks that do not require manual annotation, allowing them to learn valuable representations from large amounts of unlabelled data. These learned representations can then be fine-tuned for downstream tasks, mitigating the need for extensive labelled data. In recent years, transfer and self-supervised learning have found applications in various fields, including medical image processing, video recognition, and natural language processing. These approaches have demonstrated remarkable achievements, enabling breakthroughs in areas such as disease diagnosis, object recognition, and language understanding. However, while these methods offer numerous advantages, they also have limitations. For example, transfer learning may face domain mismatch issues between the pre-training and target domains, while self-supervised learning requires careful design of pretext tasks to ensure meaningful representations. This review paper explores the recent applications of these pre-training methods in various fields within the past three years. It delves into the advantages and limitations of each approach, assesses the performance of models employing these techniques, and identifies potential directions for future research. By providing a comprehensive review of current pre-training methods, this article offers guidance for selecting the best technique for specific deep learning applications to address the data scarcity issue. • Differences between Transfer Learning and Self-Supervised Learning. • Applications of Transfer Learning and Self-Supervised Learning. • Advantages and Limitations of Transfer Learning and Self-Supervised Learning. • Recommendations for the Use of Each Approach in Specific Applications. [ABSTRACT FROM AUTHOR]

Automated Essay Scoring (AES) is a challenging topic in Natural Language Processing. Recently, deep learning models have achieved remarkable performance for the AES task. However, applying deep learning models to the AES system in practice is expensive when both data collection and model training are taken into consideration. This paper aims to tackle this problem by proposing the Crowdsourcing-based Automated Essay Scoring (CAES) framework. The proposed framework gradually collects data through crowdsourcing and incrementally trains the AES models. In particular, we propose the Incremental Learning with Dynamic Exemplar Herding (ILDEH) approach to simultaneously tackle catastrophic forgetting and concept drift. The proposed approach dynamically updates the exemplar set by the Dynamic Exemplar Herding algorithm to obtain the best approximation of the overall data distribution and selectively apply knowledge distillation on the model outputs by Linear Outlier Suppression loss to retain the learned knowledge. Moreover, we use a lightweight AES model for effective and efficient essay scoring. The experimental results show that our proposed ILDEH approach outperforms other strong baseline approaches for the AES task. Moreover, the CAES framework is able to steadily improve the AES performance in the crowdsourcing environment with only 10.6% training time of the conventional approach. Further analysis shows that one single CPU server can support daily updates of more than 300 AES models, which is sufficient for most practical AES systems. • We proposed the Crowdsourcing-based Automated Essay Scoring (CAES) framework. • We proposed an incremental learning approach to efficiently train AES models. • Our proposed approach outperforms strong baseline approaches. • The CAES framework requires only 10.6% of the conventional training time. • The CAES framework supports daily update of 300 AES models on a single CPU server. [ABSTRACT FROM AUTHOR]

Researchers and practitioners have attempted to predict the financial market by analyzing textual (e.g., news articles and social media) and numeric data (e.g., hourly stock prices, and moving averages). Among textual data, while many papers have been published that analyze social media, news content has gained limited attention in predicting the stock market. Acknowledging that news is critical in predicting the stock market, the focus of this systematic review is on papers investigating machine learning and text mining techniques to predict the stock market using news. Using Kitchenham's methodology, we present a systematic review of the literature on intelligent financial market prediction, examining data mining and machine learning approaches and the employed datasets. From five digital libraries, we identified 61 studies from 2015–2022 for synthesis and interpretation. We present notable gaps and barriers to predicting financial markets, then recommend future research scopes. Various input data, including numerical (stock prices and technical indicators) and textual data (news text and sentiment), have been employed for news-based stock market prediction. News data collection can be costly and time-consuming: most studies have used custom crawlers to gather news articles; however, there are financial news databases available that could significantly facilitate news collection. Furthermore, although most datasets have covered fewer than 100K records, deep learning and more sophisticated artificial neural networks can process enormous datasets faster, improving future model performance. There is a growing trend toward using artificial neural networks, particularly recurrent neural networks and deep learning models, from 2018 to 2021. Furthermore, regression and gradient-boosting models have been developed for stock market prediction during the last four years. Although word embedding approaches for feature representation have been employed recently with good accuracy, emerging language models may be a focus for future research. Advanced natural language processing methods like transformers have undeniably contributed to intelligent stock market prediction. However, stock market prediction has not yet taken full advantage of them. • Emerging articles (2015–2021) on news-based stock market prediction are reviewed. • There is a growing trend toward using news headlines for stock market prediction. • Recurrent neural networks are advantageous models for stock market prediction. • Stock market prediction not fully took advantage of Natural language processing yet. • Quarterly and annual reports of companies are authorized credible data sources. [ABSTRACT FROM AUTHOR]

Regret is a universal emotion that arises from sadness, disappointment, or remorse about something that has occurred or that one has done or failed to do in the past. It typically involves wishing that a different decision had been made or that a different action had been taken. Although regret has been studied in various contexts such as psychology, neuroscience, and philosophy, its expression and analysis on social media using natural language processing techniques is a relatively recent research topic that warrants further investigation. In this paper, we present a study of regret and its expression on social media platforms. Specifically, we present a novel dataset of Reddit texts that have been classified into three classes: Regret by Action, Regret by Inaction, and No Regret. We then use this dataset to investigate the language used to express regret on Reddit and to identify the domains of text that are most commonly associated with regret. Our findings show that Reddit users are most likely to express regret for past actions, particularly in the domain of relationships. We also found that deep learning models using GloVe embedding outperformed other models in all experiments, indicating the effectiveness of GloVe for representing the meaning and context of words in the domain of regret. Overall, our study provides valuable insights into the nature and prevalence of regret on social media, as well as the potential of deep learning and word embeddings for analyzing and understanding emotional language in online text. These findings have implications for the development of natural language processing algorithms and the design of social media platforms that support emotional expression and communication. • Regret is an emotion that has a significant impact on behavior and decision-making. • Regret is often triggered by actions or inactions in the past. • Reddit's users often regret their actions about relationships in the past. • Effectiveness of GloVe in representing the meaning and context of words. [ABSTRACT FROM AUTHOR]

Hope is characterized as openness of spirit towards the future, a desire, expectation, and wish for something to happen or to be true that remarkably affects human's state of mind, emotions, behaviors, and decisions. Hope is usually associated with concepts of desired expectations and possibility/probability concerning the future. Despite its importance, hope has rarely been studied as a social media analysis task. This paper presents a hope speech dataset that classifies each tweet first into "Hope" and "Not Hope", then into three fine-grained hope categories: "Generalized Hope", "Realistic Hope", and "Unrealistic Hope" (along with "Not Hope"). English tweets in the first half of 2022 were collected to build this dataset. Furthermore, we describe our annotation process and guidelines in detail and discuss the challenges of classifying hope and the limitations of the existing hope speech detection corpora. In addition, we reported several baselines based on different learning approaches, such as traditional machine learning, deep learning, and transformers, to benchmark our dataset. We evaluated our baselines using averaged-weighted and averaged-macro F1-scores. Observations show that a strict process for annotator selection and detailed annotation guidelines enhanced the dataset's quality. This strict annotation process yielded promising performance for simple machine learning classifiers with only uni-grams; however, binary and multiclass hope speech detection results reveal that contextual embedding models have higher performance in this dataset. • Hope is fine-grained and classified into different categories. • Hope is directed towards either specific or general outcome. • Simple n-grams feature are not sufficient for multiclass hope speech detection. • Effectiveness of contextual embeddings and transformers on hope. • Problems with the current understanding of hope. [ABSTRACT FROM AUTHOR]

Deep learning techniques for Sentiment Analysis have become very popular. They provide automatic feature extraction and both richer representation capabilities and better performance than traditional feature based techniques (i.e., surface methods). Traditional surface approaches are based on complex manually extracted features, and this extraction process is a fundamental question in feature driven methods. These long-established approaches can yield strong baselines, and their predictive capabilities can be used in conjunction with the arising deep learning methods. In this paper we seek to improve the performance of deep learning techniques integrating them with traditional surface approaches based on manually extracted features. The contributions of this paper are sixfold. First, we develop a deep learning based sentiment classifier using a word embeddings model and a linear machine learning algorithm. This classifier serves as a baseline to compare to subsequent results. Second, we propose two ensemble techniques which aggregate our baseline classifier with other surface classifiers widely used in Sentiment Analysis. Third, we also propose two models for combining both surface and deep features to merge information from several sources. Fourth, we introduce a taxonomy for classifying the different models found in the literature, as well as the ones we propose. Fifth, we conduct several experiments to compare the performance of these models with the deep learning baseline. For this, we use seven public datasets that were extracted from the microblogging and movie reviews domain. Finally, as a result, a statistical study confirms that the performance of these proposed models surpasses that of our original baseline on F1-Score. [ABSTRACT FROM AUTHOR]

• Three datasets are merged to build a larger, more diversified spam dataset. • Data balancing is done using SmoteTomek and NearMiss data resampling techniques. • Proposed Hard and Soft voting ensemble models for performance improvement. • Several DL approaches are experimented with GloVe and FastText word embeddings. • Proposed FCB-SA hybrid DL approach outperformed other DL approaches. Cybercriminals use social media platforms to disseminate spam, misleading facts, fake news, and malicious links. Blocking such deceptive social media spam is essential. However, extracting relevant features from social networks is challenging due to privacy and time constraints. Traditional frequency-based word representation techniques are time-consuming and inefficient in producing contextual word vectors. Word embeddings and deep learning models have recently shown good results in text classification. Also, most existing approaches assumed balanced class distribution, which is false for most real-world datasets. In this paper, an attempt is made to advance the performance of the social spam detection system by leveraging dataset balancing, advanced word embedding techniques, machine learning, and deep learning approaches with the self-attention mechanism. In the proposed framework, the datasets are balanced using NearMiss and SmoteTomek techniques to feed several machine-learning models. Later, the baseline ML models and proposed voting-based ensemble models are evaluated on imbalanced and balanced datasets. For the proposed deep learning-based hybrid approaches, embeddings are generated using GloVe and FastText word embeddings on the balanced combined dataset and passed into the deep neural network comprised of Conv1D and Bi-directional recurrent neural network layers with the self-attention mechanism for improved context understanding and effective results. This study examines hybrid approaches for detecting social spam using imbalanced social network data and picks the optimum combination. Besides, Machine learning ensembles, word embeddings, deep learning with hyper-parameter optimization, and a self-attention method are compared thoroughly. Experiments and comparisons with other techniques show that the proposed hybrid framework with deep learning-based approaches achieves better performance. [ABSTRACT FROM AUTHOR]

Many intelligent systems in business, government or academy process natural language as an input during inference or they might even communicate with users in natural language. The natural language processing is currently often done with machine learning models. However, machine learning needs training data and such data are often scarce for low-resource languages. The lack of data and resulting poor performance of natural language processing can be solved with cross-lingual learning. Cross-lingual learning is a paradigm for transferring knowledge from one natural language to another. The transfer of knowledge can help us overcome the lack of data in the target languages and create intelligent systems and machine learning models for languages, where it was not possible previously. Despite its increasing popularity and potential, no comprehensive survey on cross-lingual learning was conducted so far. We survey 173 text processing cross-lingual learning papers and examine tasks, datasets and languages that were used. The most important contribution of our work is that we identify and analyze four types of cross-lingual transfer based on "what" is being transferred. Such insight might help other NLP researchers and practitioners to understand how to use cross-lingual learning for wide range of problems. In addition, we identify what we consider to be the most important research directions that might help the community to focus their future work in cross-lingual learning. We present a comprehensive table of all the surveyed papers with various data related to the cross-lingual learning techniques they use. The table can be used to find relevant papers and compare the approaches to cross-lingual learning. To the best of our knowledge, no survey of cross-lingual text processing techniques was done in this scope before. [ABSTRACT FROM AUTHOR]

• Document representation in the topic space improves the summarization compared to BOW. • Feature learning using unsupervised neural networks improves the summarization task. • Unsupervised neural networks trained on sentence/topic vectors give promising results. • Ensemble learning with topic representation obtains the best results. Humans must easily handle the vast amounts of data being generated by the revolution of information technology. Thus, Automatic Text summarization has been applied to various domains in order to find the most relevant information and make critical decisions quickly. In the context of Arabic, text summarization techniques suffer from several problems. First, most existing methods do not consider the context or domain to which the document belongs. Second, the majority of the existing approaches are based on the traditional bag-of-words representation, which involves high dimensional and sparse data, and makes it difficult to capture relevant information. Third, research in Arabic Text summarization is fairly small and only recently compared to that on Anglo-Saxon and other languages due to the shortage of Arabic corpora, resources, and automatic processing tools. In this paper, we try to overcome these limitations by proposing a new approach using documents clustering, topic modeling, and unsupervised neural networks in order to build an efficient document representation model. First, a new document clustering technique using Extreme learning machine is performed on large text collection. Second, topic modeling is applied to documents collection in order to identify topics present in each cluster. Third, each document is represented in a topic space by a matrix where rows represent the document sentences and columns represent the cluster topics. The generated matrix is then trained using several unsupervised neural networks and ensemble learning algorithms in order to build an abstract representation of the document in the concept space. Important sentences are ranked and extracted according to a graph model with a redundancy elimination component. The proposed approach is evaluated on Essex Arabic Summaries Corpus and compared against other Arabic text summarization approaches using ROUGE measure. Experimental results showed that the models trained on topic representation learn better representations and improve significantly the summarization performance. In particular, ensemble learning models demonstrated an important improvement on Rouge recall and promising results on F-measure. [ABSTRACT FROM AUTHOR]

This paper presents a novel architecture for dealing with Automatic Fake News detection. The architecture factors in the discourse structure of news in traditional digital media and is based on two premises. First, fake news tends to mix true and false information with the purpose of confusing readers. Second, this research is focused on fake news delivered in traditional digital media, so our approach considers the influence of the journalistic structure of news, and the way journalists tend to introduce the essential content in a news story using 5W1H answer. Considering both premises, this proposal deals with the news components separately because some may be true or false, instead of considering the veracity value of the news article as a unit. A two-layer architecture is proposed, Structure and Veracity layers. To demonstrate the validity of the proposal, a new dataset was created and annotated with a new fine-grained annotation scheme (FNDeepML) that considers the different elements of the news document and their veracity. Due to the severity of the COVID-19 pandemic crisis, health is the chosen domain, and Spanish is the language used to validate the architecture, given the lack of research in this language. However, the proposal can be applied to any other language or domain. The performance of the Veracity layer of our proposal, which factors in the traditional news article structure and the 5W1H annotation, is capable of delivering a result of F 1 =0.807. This represents a strong improvement when compared to the baseline, which uses the whole document with a single veracity value, obtaining F 1 =0.605. These findings validate the suitability and effectiveness of our approach. • A novel Automatic Fake News detection proposal based on determining the veracity of the essential content of news articles. • A new benchmark Spanish Fake News dataset focused on health news is presented. • A new Fake News Detection architecture comprising two layers (Structure Layer and Veracity Layer) is presented. • Each layer of the architecture involves a set of phases and each phase is thoroughly described. • Performance of each layer of the architecture is measured and analyzed. [ABSTRACT FROM AUTHOR]