Search

Phishing and spam are examples of unsolicited emails, result in significant financial losses for businesses and individuals every year. Numerous methodologies and strategies have been devised for the automated identification of spam, yet they have not demonstrated complete predictive precision. Within the spectrum of suggested methodologies, ML and DL algorithms have shown the most promising results. This article scrutinizes the outcomes of assessing the efficacy of three transformation-based models - BERT, AlBERT, and RoBERTa - in scrutinizing both textual and numerical data. The proposed models achieved higher accuracy and efficiency in classification tasks, which was a notable improvement above traditional models such as KNN, NB, BiLSTM, and LSTM. Interestingly, in several criteria the Roberta model achieved almost perfect accuracy, suggesting that it is very flexible on a variety of datasets. [ABSTRACT FROM AUTHOR]

In the field of natural language processing (NLP), there have been various pre-training language models in recent years, with question answering systems gaining significant attention. However, as algorithms, data, and computing power advance, the issue of increasingly larger models and a growing number of parameters has surfaced. Consequently, model training has become more costly and less efficient. To enhance the efficiency and accuracy of the training process while reducing themodel volume, this paper proposes a first-order pruningmodel PAL-BERT based on the ALBERT model according to the characteristics of question-answering (QA) system and language model. Firstly, a first-order network pruning method based on the ALBERT model is designed, and the PAL-BERT model is formed. Then, the parameter optimization strategy of the PAL-BERT model is formulated, and the Mish function was used as an activation function instead of ReLU to improve the performance. Finally, after comparison experiments with traditional deep learning models TextCNN and BiLSTM, it is confirmed that PALBERT is a pruning model compression method that can significantly reduce training time and optimize training efficiency. Compared with traditional models, PAL-BERT significantly improves the NLP task's performance. [ABSTRACT FROM AUTHOR]

Research on uni-modal sentiment analysis has achieved great success, but emotions in real life are mostly multi-modal; there are not only texts but also images, audio, video, and other forms. The various modes play a role in mutual promotion. If the connection between various modalities can be mined, the accuracy of sentiment analysis will be further improved. To this end, this paper introduces a cross-attention-based multi-modal fusion model for images and text, namely, MCAM. First, we use the ALBert pre-training model to extract text features for text; then, we use BiLSTM to extract text context features; then, we use DenseNet121 to extract image features for images; and then, we use CBAM to extract specific areas related to emotion in images. Finally, we utilize multi-modal cross-attention to fuse the extracted features from the text and image, and we classify the output to determine the emotional polarity. In the experimental comparative analysis of MVSA and TumEmo public datasets, the model in this article is better than the baseline model, with accuracy and F1 scores reaching 86.5% and 75.3% and 85.5% and 76.7%, respectively. In addition, we also conducted ablation experiments, which confirmed that sentiment analysis with multi-modal fusion is better than single-modal sentiment analysis. [ABSTRACT FROM AUTHOR]

Copyright of International Journal of InformaticsTechnologies is the property of Institute of Informatics, Gazi University and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Summary: In today's world, there is a lot of anxiety about suicidal thoughts that is conveyed on social media platforms, and people are now sharing all kinds of feelings on social media forums. The majority of them utilize social forums because they feel uncomfortable sharing privately. The study's objective is to analyze suicidal thoughts and identify them at an early stage by utilizing deep learning and transfer learning techniques. These algorithms are used to data gathered from users of Reddit forums who have suicidal thoughts as well as regular users who have non‐suicidal thoughts. For the aforementioned goal, we use techniques such as the transfer learning algorithms BERT, RoBERTa, and ALBERT, as well as BiLSTM and other deep learning algorithms. In contrast to the sequence processing model, our study demonstrates that the bidirectional long‐short term algorithm provides the best validation accuracy, while the pretrained models BERT and ALBERT also provide satisfactory accuracy. [ABSTRACT FROM AUTHOR]

Football is one of the most popular sports in the world, arousing a wide range of research topics related to its off- and on-the-pitch performance. The extraction of football entities from football news helps to construct sports frameworks, integrate sports resources, and timely capture the dynamics of the sports through visual text mining results, including the connections among football players, football clubs, and football competitions, and it is of great convenience to observe and analyze the developmental tendencies of football. Therefore, in this paper, we constructed a 1000,000-word Chinese corpus in the field of football and proposed a BiLSTM-based model for named entity recognition. The ALBERT-BiLSTM combination model of deep learning is used for entity extraction of football textual data. Based on the BiLSTM model, we introduced ALBERT as a pre-training model to extract character and enhance the generalization ability of word embedding vectors. We then compared the results of two different annotation schemes, BIO and BIOE, and two deep learning models, ALBERT-BiLSTM-CRF and ALBERT BiLSTM. It was verified that the BIOE tagging was superior than BIO, and the ALBERT-BiLSTM model was more suitable for football datasets. The precision, recall, and F-Score of the model were 85.4%, 83.47%, and 84.37%, correspondingly. [ABSTRACT FROM AUTHOR]

In recent years, numerous achievements have been made in the field of deep learning, particularly in text processing. In the wave of intelligence, people's demand for intelligent communication is becoming increasingly higher. Therefore, we consider utilizing deep learning models to design and optimize transceiver of semantic communication system. The research of semantic communication is in a booming stage, but there are still few applications in multi-user scenario. In general, the parameters of the semantic communication system transceiver based on the deep learning model are very large. Therefore, we study the multi-user semantic communication system based on the ALBERT model. The goal of the proposed semantic communication system is to intelligently and correctly send the corresponding text classification to the receiver. The channel state information (CSI) is very important for information transmission. Considering the multi-antenna multi-user uplink scenario, we adopt the conditional generative adversarial network (cGAN) model to estimate CSI and apply it to the proposed semantic communication system. In order to reduce the influence of channel estimation on the delay of communication system, we quantify the pilot at the receiver. The simulation results show that the performance of the semantic communication system proposed in this paper is better than that of the semantic communication system based on Transformer model and the traditional semantic communication system in the intelligent text classification task. Moreover, in the case of low signal-to-noise ratio, traditional communication is difficult to complete intelligent tasks. [ABSTRACT FROM AUTHOR]

Intent classification and named entity recognition of medical questions are two key subtasks of the natural language understanding module in the question answering system. Most existing methods usually treat medical queries intent classification and named entity recognition as two separate tasks, ignoring the close relationship between the two tasks. In order to optimize the effect of medical queries intent classification and named entity recognition tasks, a multi-task learning model based on ALBERT-BILSTM is proposed for intent classification and named entity recognition of Chinese online medical questions. The multi-task learning model in this paper makes use of encoder parameter sharing, which enables the model's underlying network to take into account both named entity recognition and intent classification features. The model learns the shared information between the two tasks while maintaining its unique characteristics during the decoding phase. The ALBERT pre-training language model is used to obtain word vectors containing semantic information and the bidirectional LSTM network is used for training. A comparative experiment of different models was conducted on Chinese medical questions dataset. Experimental results show that the proposed multi-task learning method outperforms the benchmark method in terms of precision, recall and F1 value. Compared with the single-task model, the generalization ability of the model has been improved. [ABSTRACT FROM AUTHOR]

With the world's interaction moving more and more toward using online social media platforms, the advent of cyberbullying has also raised its head. Multiple forms of cyberbullying exist from the more common text based to images or even videos, and this paper will explore the context of textual comments. Even in the niche area of considering only text-based data, several approaches have already been worked upon such as n-grams, recurrent units, convolutional neural networks (CNNs), gated recurrent unit (GRU) and even a combination of the mentioned architectures. While all of these produce workable results, the main point of contention is that true contextual understanding is quite a complex concept. These methods fail due to two simple reasons: (i) lack of large datasets to properly utilize these architectures and (ii) the fact that understanding context requires some mechanism of remembering history that is only present in the recurrent units. This paper explores some of the recent approaches to the difficulties of contextual understanding and proposes an ALBERT-based fine-tuned model that achieves state-of-the-art results. ALBERT is a transformer-based architecture and thus even at its untrained form provides better contextual understanding than other recurrent units. This coupled with the fact that ALBERT is pre-trained on a large corpus allowing the flexibility to use a smaller dataset for fine-tuning as the pre-trained model already has deep understanding of the complexities of the human language. ALBERT showcases high scores in multiple benchmarks such as the GLUE and SQuAD showing that high levels of contextual understanding are inherently present and thus fine-tuning for the specific case of cyberbullying allows to use this to our advantage. With this approach, we have achieved an F1 score of 95% which surpasses current approaches such as the CNN + wordVec, CNN + GRU and BERT implementations. [ABSTRACT FROM AUTHOR]

Football is one of the most popular sports in the world, arousing a wide range of research topics related to its off- and on-the-pitch performance. The extraction of football entities from football news helps to construct sports frameworks, integrate sports resources, and timely capture the dynamics of the sports through visual text mining results, including the connections among football players, football clubs, and football competitions, and it is of great convenience to observe and analyze the developmental tendencies of football. Therefore, in this paper, we constructed a 1000,000-word Chinese corpus in the field of football and proposed a BiLSTM-based model for named entity recognition. The ALBERT-BiLSTM combination model of deep learning is used for entity extraction of football textual data. Based on the BiLSTM model, we introduced ALBERT as a pre-training model to extract character and enhance the generalization ability of word embedding vectors. We then compared the results of two different annotation schemes, BIO and BIOE, and two deep learning models, ALBERT-BiLSTM-CRF and ALBERT BiLSTM. It was verified that the BIOE tagging was superior than BIO, and the ALBERT-BiLSTM model was more suitable for football datasets. The precision, recall, and F-Score of the model were 85.4%, 83.47%, and 84.37%, correspondingly.

The upsurge of prolific blogging and microblogging platforms enabled the abusers to spread negativity and threats greater than ever. Negative and hateful comments are averting users from sharing their opinion freely on social media platforms. It often breaks people's confidence and causes extensive damage to their mental health. Hence, identifying these toxic contents and taking appropriate measures against them is crucial to preserve a safe environment on social media. Numerous state-of-the-art approaches classify the whole content as toxic or non-toxic, but they don't distinguish the precise toxic portion from the whole content. Detecting the toxic portions is essential as it substantially aids to moderate the toxic contents through excluding the abusive parts. This paper describes our proposed approach to detect the toxic portions from text contents efficiently and accurately. We explore an ensemble of sequence labeling models including the word embedding-based Spark NLP NER (named entity recognition) deep learning model, spaCy NER model with custom toxic tags, and ALBERT NER model to identify the toxic spans. The NER-based models usually intend to capture the contextual attributes of phrases and spans that are essential for named entity recognition. As the toxic span detection task also requires us to apprehend the phrasal context for detecting toxic span, the similarity between these two tasks inspires us to exploit these NER models. Finally, we determine the final toxic spans using a prevalence-based fusion of the predictions generated by these models. The fusion strategy enables us to consolidate the diversity of these models for perceiving the phrasal context in all aspects. Experimental results achieved on the SemEval-2021 toxic spans detection dataset depict that our model meticulously captures the toxic fragment and achieves a competitive result among the other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Aspect-level sentiment classification aims to solve the problem, which is to judge the sentiment tendency of each aspect in a sentence with multiple aspects. Previous works mainly employed Long Short-Term Memory (LSTM) and Attention mechanisms to fuse information between aspects and sentences, or to improve large language models such as BERT to adapt aspect-level sentiment classification tasks. The former methods either did not integrate the interactive information of related aspects and sentences, or ignored the feature extraction of sentences. This paper proposes a novel multi-grained attention representation with ALBERT (MGAR-ALBERT). It can learn the representation that contains the relevant information of the sentence and the aspect, while integrating it into the process of sentence modeling with multi granularity, and finally get a comprehensive sentence representation. In Masked LM (MLM) task, in order to avoid the influence of aspect words being masked in the initial stage of the pre-training, the noise linear cosine decay is introduced into $n-gram$ . We implemented a series of comparative experiments to verify the effectiveness of the method. The experimental results show that our model can achieve excellent results on Restaurant dataset with numerous number of parameters reduced, and it is not inferior to other models on Laptop dataset.

Deep learning is advancing rapidly; it has aided in solving problems that were thought impossible. Natural language understanding is one such task that has evolved with the advancement of deep learning systems. There have been several sentiment analysis attempts, but they aim to classify it as a single emotion. Human emotion in natural language is generally a complex combination of emotions, which may be indeterminate or neutral at times. Neutrosophy is a branch of philosophy that identifies neutralities and uses membership functions (positive, negative, neutral) to quantify a sample into Single Valued Neutrosophic Set (SVNS) values. Our work aims to combine the power of deep learning with SVNS to represent a sample's sentiment into membership functions of SVNS. We have worked on the Offensive Language Identification Dataset (OLID). Combining the power of state-of-the-art neural network techniques with neutrosophy allowed us to quantify the sentiments and identify the transition phase between positive and negative ones. We used the transition phase to capture neutral samples, which is beneficial if we want to obtain purely positive/negative samples. We performed experiments using Bi-directional Long Short Term Memory (BiLSTM) with attention, Bidirectional Encoder Representations from Transformers (BERT), A Lite BERT (ALBERT), A Robustly Optimised BERT Approach (RoBERTa), and MPNet. Our SVNS model performed equivalent to state-of-the-art neural network models on the OLID dataset. Here, we propose a novel framework that can integrate with any neural network model and quantify sentiments using SVNS. • Sentiment analysis model using deep learning, neutrosophy, and transfer learning. • Analysing tweets as a combination of sentiments identifying neutralities. • Quantifying tweets into Single Valued Neutrosophic Sets (SVNS) for OLID dataset. • Experimental analysis using BiLSTM, BERT, RoBERTa, ALBERT, and MPNet. • Gaussian Mixture Model and k-means clustering algorithm for SVNS calculation. [ABSTRACT FROM AUTHOR]

Deep learning has been widely used in numerous real-world engineering applications and for classification problems. Real-world data is present with neutrality and indeterminacy, which neutrosophic theory captures clearly. Though both are currently developing research areas, there has been little study on their interlinking. We have proposed a novel framework to implement neutrosophy in deep learning models. Instead of just predicting a single class as output, we have quantified the sentiments using three membership functions to understand them better. Our proposed model consists of two blocks, feature extraction, and feature classification. Having a separate feature extraction block enables us to use any model as a feature extractor. We experimented with BiLSTM using GloVe (Global Vectors for word representation), BERT (Bidirectional Encoder Representations from Transformers), ALBERT (A Lite BERT), RoBERTa (Robustly optimized BERT approach), MPNet, and stacked ensemble models. Feature classification performs prediction and dimensionality reduction of features. Experimental analysis was done on the SemEval 2017 Task 4 dataset (Subtask A). We used the intermediate layer features to define membership functions of Single Valued Neutrosophic Sets (SVNS). We used these membership functions for prediction as well. We have compared our models with the top five teams of the task and recent state-of-the-art systems. Our proposed stacked ensemble model achieved the best recall (0.733) score. • Architecture for combining neutrosophy and deep learning for sentiment analysis. • Experimental analysis using LSTM, BERT, RoBERTa, ALBERT, MPNet, and Ensemble models. • Feature extraction using intermediate layers of deep learning models. • Using K-means and Cosine distance for calculating SVNS from generated features. • SVNS as an alternative to Softmax for multi-label sentiment analysis problem. [ABSTRACT FROM AUTHOR]