Search

Phishing and spam are examples of unsolicited emails, result in significant financial losses for businesses and individuals every year. Numerous methodologies and strategies have been devised for the automated identification of spam, yet they have not demonstrated complete predictive precision. Within the spectrum of suggested methodologies, ML and DL algorithms have shown the most promising results. This article scrutinizes the outcomes of assessing the efficacy of three transformation-based models - BERT, AlBERT, and RoBERTa - in scrutinizing both textual and numerical data. The proposed models achieved higher accuracy and efficiency in classification tasks, which was a notable improvement above traditional models such as KNN, NB, BiLSTM, and LSTM. Interestingly, in several criteria the Roberta model achieved almost perfect accuracy, suggesting that it is very flexible on a variety of datasets. [ABSTRACT FROM AUTHOR]

Transfer learning models have proven superior to classical machine learning approaches in various text classification tasks, such as sentiment analysis, question answering, news categorization, and natural language inference. Recently, these models have shown exceptional results in natural language understanding (NLU). Advanced attention‐based language models like BERT and XLNet excel at handling complex tasks across diverse contexts. However, they encounter difficulties when applied to specific domains. Platforms like Facebook, characterized by continually evolving casual and sophisticated language, demand meticulous context analysis even from human users. The literature has proposed numerous solutions using statistical and machine learning techniques to predict the sentiment (positive or negative) of online customer reviews, but most of them rely on various business, review, and reviewer features, which leads to generalizability issues. Furthermore, there have been very few studies investigating the effectiveness of state‐of‐the‐art pre‐trained language models for sentiment classification in reviews. Therefore, this study aims to assess the effectiveness of BERT, RoBERTa, ALBERT, DistilBERT, and XLNet in sentiment classification using the Yelp reviews dataset. The models were fine‐tuned, and the results obtained with the same hyperparameters are as follows: 98.30 for RoBERTa, 98.20 for XLNet, 97.40 for BERT, 97.20 for ALBERT, and 96.00 for DistilBERT. [ABSTRACT FROM AUTHOR]

Aspect-level sentiment analysis is a critical component of sentiment analysis, aiming to determine the sentiment polarity associated with specific aspect words. However, existing methodologies have limitations in effectively managing aspect-level sentiment analysis. These limitations include insufficient utilization of syntactic information and an inability to precisely capture the contextual nuances surrounding aspect words. To address these issues, we propose an Aspect-Oriented Graph Attention Network (AOGAT) model. This model incorporates syntactic information to generate dynamic word vectors through the pre-trained model ALBERT and combines a graph attention network with BiGRU to capture both syntactic and semantic features. Additionally, the model introduces an aspect-focused attention mechanism to retrieve features related to aspect words and integrates the generated representations for sentiment classification. Our experiments on three datasets demonstrate that the AOGAT model outperforms traditional models. [ABSTRACT FROM AUTHOR]

As an extension of the transformer architecture, the BERT model has introduced a new paradigm for natural language processing, achieving impressive results in various downstream tasks. However, high-performance BERT-based models—such as ELECTRA, ALBERT, and RoBERTa—suffer from limitations such as poor continuous learning capability and insufficient understanding of domain-specific documents. To address these issues, we propose the use of an attention mechanism to combine BERT-based models with neural topic models. Unlike traditional stochastic topic modeling, neural topic modeling employs artificial neural networks to learn topic representations. Furthermore, neural topic models can be integrated with other neural models and trained to identify latent variables in documents, thereby enabling BERT-based models to sufficiently comprehend the contexts of specific fields. We conducted experiments on three datasets—Movie Review Dataset (MRD), 20Newsgroups, and YELP—to evaluate our model's performance. Compared to the vanilla model, the proposed model achieved an accuracy improvement of 1–2% for the ALBERT model in multiclassification tasks across all three datasets, while the ELECTRA model showed an accuracy improvement of less than 1%. [ABSTRACT FROM AUTHOR]

Sentiment analysis is a critical component in understanding customer opinions and reactions. This study explores the application of sentiment analysis using Python on the Amazon Fine Food Reviews dataset to classify customer reviews as positive or negative, enabling businesses to gain valuable insight into customer sentiments. This study used and compared the efficiency of Logistic Regression, Support Vector Machines, Random Forest, XGBoost, LSTM, and ALBERT. The comparison results showed that the LSTM and ALBERT classifiers stand out with remarkable accuracy (96%) and substantial support for positive and negative reviews. On the other hand, although the Random Forest classifier had similar accuracy (96%), it exhibited lower support for positive and negative sentiments. [ABSTRACT FROM AUTHOR]

In the field of natural language processing (NLP), there have been various pre-training language models in recent years, with question answering systems gaining significant attention. However, as algorithms, data, and computing power advance, the issue of increasingly larger models and a growing number of parameters has surfaced. Consequently, model training has become more costly and less efficient. To enhance the efficiency and accuracy of the training process while reducing themodel volume, this paper proposes a first-order pruningmodel PAL-BERT based on the ALBERT model according to the characteristics of question-answering (QA) system and language model. Firstly, a first-order network pruning method based on the ALBERT model is designed, and the PAL-BERT model is formed. Then, the parameter optimization strategy of the PAL-BERT model is formulated, and the Mish function was used as an activation function instead of ReLU to improve the performance. Finally, after comparison experiments with traditional deep learning models TextCNN and BiLSTM, it is confirmed that PALBERT is a pruning model compression method that can significantly reduce training time and optimize training efficiency. Compared with traditional models, PAL-BERT significantly improves the NLP task's performance. [ABSTRACT FROM AUTHOR]

Research on uni-modal sentiment analysis has achieved great success, but emotions in real life are mostly multi-modal; there are not only texts but also images, audio, video, and other forms. The various modes play a role in mutual promotion. If the connection between various modalities can be mined, the accuracy of sentiment analysis will be further improved. To this end, this paper introduces a cross-attention-based multi-modal fusion model for images and text, namely, MCAM. First, we use the ALBert pre-training model to extract text features for text; then, we use BiLSTM to extract text context features; then, we use DenseNet121 to extract image features for images; and then, we use CBAM to extract specific areas related to emotion in images. Finally, we utilize multi-modal cross-attention to fuse the extracted features from the text and image, and we classify the output to determine the emotional polarity. In the experimental comparative analysis of MVSA and TumEmo public datasets, the model in this article is better than the baseline model, with accuracy and F1 scores reaching 86.5% and 75.3% and 85.5% and 76.7%, respectively. In addition, we also conducted ablation experiments, which confirmed that sentiment analysis with multi-modal fusion is better than single-modal sentiment analysis. [ABSTRACT FROM AUTHOR]

With the advancement of technology in recent years, the application of artificial intelligence in real life has become more extensive. Graphic recognition is a hot spot in the current research of related technologies. It involves machines extracting key information from pictures and combining it with natural language processing for in-depth understanding. Existing methods still have obvious deficiencies in fine-grained recognition and deep understanding of contextual context. Addressing these issues to achieve high-quality image-text recognition is crucial for various application scenarios, such as accessibility technologies, content creation, and virtual assistants. To tackle this challenge, a novel approach is proposed that combines the Mask R-CNN, DCGAN, and ALBERT models. Specifically, the Mask R-CNN specializes in high-precision image recognition and segmentation, the DCGAN captures and generates nuanced features from images, and the ALBERT model is responsible for deep natural language processing and semantic understanding of this visual information. Experimental results clearly validate the superiority of this method. Compared to traditional image-text recognition techniques, the recognition accuracy is improved from 85.3% to 92.5%, and performance in contextual and situational understanding is enhanced. The advancement of this technology has far-reaching implications for research in machine vision and natural language processing and open new possibilities for practical applications.

In natural language processing (NLP), a Question Answering System (QAS) refers to a system or model that is designed to understand and respond to user queries in natural language. As we navigate through the recent advancements in QAS, it can be observed that there is a paradigm shift of the methods used from traditional machine learning and deep learning approaches towards transformer-based language models. While significant progress has been made, the utilization of these models for historical QAS and the development of QAS for Malay language remain largely unexplored. This research aims to bridge the gaps, focusing on developing a Multilingual QAS for history of Malaysia by utilizing a transformer-based language model. The system development process encompasses various stages, including data collection, knowledge representation, data loading and pre-processing, document indexing and storing, and the establishment of a querying pipeline with the retriever and reader. A dataset with a collection of 100 articles, including web blogs related to the history of Malaysia, has been constructed, serving as the knowledge base for the proposed QAS. A significant aspect of this research is the use of the translated dataset in English instead of the raw dataset in Malay. This decision was made to leverage the effectiveness of well-established retriever and reader models that were trained on English data. Moreover, an evaluation dataset comprising 100 question-answer pairs has been created to evaluate the performance of the models. A comparative analysis of six different transformer-based language models, namely DeBERTaV3, BERT, ALBERT, ELECTRA, MiniLM, and RoBERTa, has been conducted, where the effectiveness of the models was examined through a series of experiments to determine the best reader model for the proposed QAS. The experimental results reveal that the proposed QAS achieved the best performance when employing RoBERTa as the reader model. Finally, the proposed QAS was deployed on Discord and equipped with multilingual support through the incorporation of language detection and translation modules, enabling it to handle queries in both Malay and English. Doi: 10.28991/ESJ-2024-08-02-019 Full Text: PDF

Copyright of International Journal of InformaticsTechnologies is the property of Institute of Informatics, Gazi University and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Spam emails, which are unsolicited bulk emails, pose a significant threat in digital communication security. To counter spam emails, learning-based spam email filtering methods have been extensively studied. However, as spam patterns evolve, these methods face challenges in maintaining the accuracy of models trained on outdated patterns. To demonstrate these limitations empirically and gain insight into the classification patterns of spam email filtering models, we propose an advanced analysis method to analyze the performance degradation of spam email filtering models. The proposed analysis method involves text preprocessing, embedding model training, spam email filtering model training, evaluation, and analysis of the classification patterns of the learning-based spam email filtering models. From the experimental results under various datasets and spam email filtering models, we show that the accuracy of spam email filtering models significantly decreases when the feature distribution of the test dataset is different from the training dataset. We also provides valuable insights for improving the model architecture, dataset structure, and training strategies by analysis of various factors such as confusion matrix, performance metrics, mean sequence length, out-of-vocabulary (OOV) rate, and top-20 tokens.

As an extension of the transformer architecture, the BERT model has introduced a new paradigm for natural language processing, achieving impressive results in various downstream tasks. However, high-performance BERT-based models—such as ELECTRA, ALBERT, and RoBERTa—suffer from limitations such as poor continuous learning capability and insufficient understanding of domain-specific documents. To address these issues, we propose the use of an attention mechanism to combine BERT-based models with neural topic models. Unlike traditional stochastic topic modeling, neural topic modeling employs artificial neural networks to learn topic representations. Furthermore, neural topic models can be integrated with other neural models and trained to identify latent variables in documents, thereby enabling BERT-based models to sufficiently comprehend the contexts of specific fields. We conducted experiments on three datasets—Movie Review Dataset (MRD), 20Newsgroups, and YELP—to evaluate our model’s performance. Compared to the vanilla model, the proposed model achieved an accuracy improvement of 1–2% for the ALBERT model in multiclassification tasks across all three datasets, while the ELECTRA model showed an accuracy improvement of less than 1%.

In today's interconnected world, where individuals can create and receive information freely, the proliferation of fake news has become a significant issue. This type of false information frequently appears in areas such as business or politics, and its widespread dissemination on the internet can disrupt the normal social order and create a biased net- work atmosphere, ultimately leading to the destruction of the normal network environment. The evolution of fake news, from early plain text to complex images and texts, has made its detection more difficult. To address this, we propose an Albert ResNet50 hybrid deep neural net- work model that combines implicit features of both text and images for detecting multimodal fake news. We tested our model on three fake news datasets, and the results showed an accuracy rate of 90.51%, 79.87%, and 92.93%, respectively. Compared to traditional models that only use text data, our multimodal model can better identify fake news. [ABSTRACT FROM AUTHOR]

Named entity recognition of wheat diseases and pests is a key step to building a knowledge graph. Aiming at the problems of lack of training data, complex entity structure, diverse entity types and uneven entity distribution in wheat diseases and pests field, under the promise of fully mining the implicit knowledge, two data augmentation methods are used to expand sentence semantic information, and to construct the corpus WpdCNER (wheat pests and diseases Chinese named entity recognition) and the field lexicon WpdDict (wheat pests and diseases dictionary). And 16 categories of entities are defined with the field experts’ guidance. Meanwhile, Chinese named entity recognition model based on rules amendment WPD-RA (wheat pests and disease-rules amendment model) is proposed. This model is carried out entity recognition based on ALBERT+BiLSTM+CRF (a lite bi-directional encoder representation from transformer + bi-directional long short-term memory + conditional random field), and specific rules are defined to amend entity boundaries after recognition. The WPD-RA model achieves the best results with 94.72% precision, 95.23% recall, and 94.97% F1. Its precision is increased by 1.71 percentage points, recall is increased by 0.34 percentage points, and F1 is increased by 1.03 percentage points, compared with the model without rules. Experimental results show that the model can effectively recognize named entities in wheat diseases and pests field, and its performance is better than other models. The proposed model provides a reference idea for named entity recognition task in other fields such as food safety and biology.

Information retrieval and text mining fields extensively utilize text semantic matching models. In this paper, civil litigation Q&A under the environmental justice system is taken as a specific research field, and after constructing a civil litigation Q&A system based on deep learning, two of the key techniques—question categorization and semantic matching—are selected as the main research content. Specifically, the ALBERT algorithm is used to extract word vectors, and the hidden feature vectors are obtained through BiLSTM modeling of contextual relationships and then combined with the Attention mechanism for scoring and weighting to obtain the final text-level vectors for classification so as to establish the civil litigation question classification model based on ALBERT. Then, we establish the BERT-based civil litigation question and answer matching model by sorting the set of candidate answers by semantic matching degree based on the BERT algorithm. Selected datasets and comparison algorithms are experimented with, and the analysis shows that the question classification model has a better effect than civil litigation question text classification, and the values of each index have been improved by 0.75%~3.00% on the basis of the baseline model. The MAP and MRR values (0.76~0.86) of the question-matching model are higher than those of the comparison model, verifying its superior performance in semantically assigning characters. The model proposed in this paper is more useful because it can provide civil litigation counseling to the public.

A model is utilized in this paper to analyze the textual emotions of ideology in Marxist theory. Long and short-term memory networks are chosen as the main method of text analysis to construct the main process of Marxist ideology education. Combined with the hybrid self-attention mechanism, the efficiency of extracting data features from the text was improved. The results show that the ALBERT-SABL-based sentiment analysis model is 86.9% accurate in extracting the sentiment of the ideology, and the F1 value is 87.6%. Compared with TextCnn, the accuracy has improved by 1.8%. Different schools have different levels of identification with Marx’s ideology, and under the identification dimension, the highest sentiment identification dimension among the eight school samples is School 2, with a dimension of 100. This study provides reference data in the ideological education of Marxist theory and promotes the development of Marx’s ideology.

Summary: In today's world, there is a lot of anxiety about suicidal thoughts that is conveyed on social media platforms, and people are now sharing all kinds of feelings on social media forums. The majority of them utilize social forums because they feel uncomfortable sharing privately. The study's objective is to analyze suicidal thoughts and identify them at an early stage by utilizing deep learning and transfer learning techniques. These algorithms are used to data gathered from users of Reddit forums who have suicidal thoughts as well as regular users who have non‐suicidal thoughts. For the aforementioned goal, we use techniques such as the transfer learning algorithms BERT, RoBERTa, and ALBERT, as well as BiLSTM and other deep learning algorithms. In contrast to the sequence processing model, our study demonstrates that the bidirectional long‐short term algorithm provides the best validation accuracy, while the pretrained models BERT and ALBERT also provide satisfactory accuracy. [ABSTRACT FROM AUTHOR]

The increasing application of medical robots in the healthcare sector underscores the critical importance of intent recognition in enhancing the interaction and assistance capabilities of these robots. Traditional intent recognition methods utilize convolutional neural networks (CNNs) for text analysis but often fall short in capturing global features, resulting in incomplete information. To address this challenge, this paper introduces an innovative approach by combining an enhanced CNN with bidirectional gated recurrent units (BiGRU) to construct a dual-channel short-text intent recognition model. This model effectively leverages both local and global features to more accurately comprehend user needs and intentions. Experimental results demonstrate that this model excels, achieving an accuracy rate of 96.68% and an F1 score of 96.67% on the THUCNews_Title dataset. In comparison to conventional intent recognition models, it exhibits significantly improved performance, thereby providing substantial support for medical robots in patient care and assisting healthcare professionals. [ABSTRACT FROM AUTHOR]

Football is one of the most popular sports in the world, arousing a wide range of research topics related to its off- and on-the-pitch performance. The extraction of football entities from football news helps to construct sports frameworks, integrate sports resources, and timely capture the dynamics of the sports through visual text mining results, including the connections among football players, football clubs, and football competitions, and it is of great convenience to observe and analyze the developmental tendencies of football. Therefore, in this paper, we constructed a 1000,000-word Chinese corpus in the field of football and proposed a BiLSTM-based model for named entity recognition. The ALBERT-BiLSTM combination model of deep learning is used for entity extraction of football textual data. Based on the BiLSTM model, we introduced ALBERT as a pre-training model to extract character and enhance the generalization ability of word embedding vectors. We then compared the results of two different annotation schemes, BIO and BIOE, and two deep learning models, ALBERT-BiLSTM-CRF and ALBERT BiLSTM. It was verified that the BIOE tagging was superior than BIO, and the ALBERT-BiLSTM model was more suitable for football datasets. The precision, recall, and F-Score of the model were 85.4%, 83.47%, and 84.37%, correspondingly. [ABSTRACT FROM AUTHOR]

In recent years, numerous achievements have been made in the field of deep learning, particularly in text processing. In the wave of intelligence, people's demand for intelligent communication is becoming increasingly higher. Therefore, we consider utilizing deep learning models to design and optimize transceiver of semantic communication system. The research of semantic communication is in a booming stage, but there are still few applications in multi-user scenario. In general, the parameters of the semantic communication system transceiver based on the deep learning model are very large. Therefore, we study the multi-user semantic communication system based on the ALBERT model. The goal of the proposed semantic communication system is to intelligently and correctly send the corresponding text classification to the receiver. The channel state information (CSI) is very important for information transmission. Considering the multi-antenna multi-user uplink scenario, we adopt the conditional generative adversarial network (cGAN) model to estimate CSI and apply it to the proposed semantic communication system. In order to reduce the influence of channel estimation on the delay of communication system, we quantify the pilot at the receiver. The simulation results show that the performance of the semantic communication system proposed in this paper is better than that of the semantic communication system based on Transformer model and the traditional semantic communication system in the intelligent text classification task. Moreover, in the case of low signal-to-noise ratio, traditional communication is difficult to complete intelligent tasks. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Frontiers of Computer Science & Technology is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Intelligent educational diagnosis can effectively promote the development of artificial intelligence in education. The knowledge diagnosis of specific domains (e.g., mathematics, physics) plays an important role in intelligent educational diagnosis but typically relies on complex semantic information. Most existing methods only produce single sentence representations that have difficulty detecting multiple knowledge points from text. The resources of knowledge point diagnosis of specific domains are also relatively sparse. In this study, we build a dataset about mathematics that is collected from real mathematical examination and artificially annotated 18 knowledge points. We also propose the MultiHop Attention mechanism (MHA) model to focus on different important information in mathematical questions using a multiple attention mechanism. Each attention mechanism obtains different attention weights for different parts of mathematical questions. The MHA allows us to effectively obtain a comprehensive semantic representation of mathematical questions. Additionally, because the ALBERT model is advanced and efficient, we use it for word embedding in this study. The proposed method synthetically considers multiple keywords related to knowledge points in mathematical questions for knowledge diagnosis research. Experimental results with the proposed mathematical dataset show that MHA achieves marked improvements compared to existing methods. [ABSTRACT FROM AUTHOR]

The purpose of this essay is threefold. Firstly, it attempts to introduce the diplomatic and collecting careers of the Austro-Hungarian diplomat Baron Albert Eperjesy (1848–1916), who was the highest representative of his country in numerous European capitals and –between 1895 and 1901– Tehran. Secondly, an attempt will be made to contextualise his collecting habits by drawing attention to the peculiarities of Austro-Hungarian collector diplomats. Finally, and perhaps most importantly, the Persian element of this collection will be discussed within the previously outlined framework, namely, what artworks it did include, how and where he obtained them, and what would be their subsequent fate.

Cyber threat intelligence entity identification is the key to cyber threat intelligence analysis. In view of the fact that traditional word embedding cannot represent the polysemy of a word, it is difficult to effectively identify the key information of cyber threat intelligence entities, and at the same time, facing the exponential growth of threat intelligence, the efficiency of the identification model needs to be improved urgently. A network threat intelligence named entity recognition model based on ALBERT is proposed. The model first uses ALBERT to extract threat intelligence dynamic feature word vectors. Then, the feature word vector is input to the bidirectional long short-term memory network (BiLSTM) layer to obtain the corresponding label of each word in the sentence. Finally, the conditional random field (CRF) layer is modified and the sequence label is output with the maximum probability. The experimental results of identification model comparison show that the F1 value of the proposed model is 92.21%, which is obviously better than other models. In the case of the same recognition accuracy, the time and resource costs of the proposed model are also lower, which is suitable for massive and efficient entity recognition tasks in the field of cyber threat intelligence. [ABSTRACT FROM AUTHOR]

Named entity recognition can deeply explore semantic features and enhance the ability of vector representation of text data. This paper proposes a named entity recognition method based on multi-head attention to aim at the problem of fuzzy lexical boundary in Chinese named entity recognition. Firstly, Word2vec is used to extract word vectors, HMM is used to extract boundary vectors, ALBERT is used to extract character vectors, the Feedforward-attention mechanism is used to fuse the three vectors, and then the fused vectors representation is used to remove features by BiLSTM. Then multi-head attention is used to mine the potential word information in the text features. Finally, the text label classification results are output after the conditional random field screening. Through the verification of WeiboNER, MSRA, and CLUENER2020 datasets, the results show that the proposed algorithm can effectively improve the performance of named entity recognition. [ABSTRACT FROM AUTHOR]

Microblog is an important platform for mining public opinion, and it is of great value to conduct emotional analysis of microblog texts during the current epidemic. Aiming at the problem that most current emotional classification methods cannot effectively extract deep text features, and that traditional word vectors cannot dynamically obtain the semantics of words according to their context, which leads to classification bias, this research put forward a microblog text emotion classification algorithm based on TCN-BiGRU and dual attention (TCN-BiGRU-DATT). First, the vector representation of the text was obtained using ALBERT. Second, the TCN and BiGRU networks were used to extract the emotional information contained in the text through dual pathway feature extraction, to efficiently obtain the deep semantic features of the text. Then, the dual attention mechanism was introduced to allocate the global weight of the key information in the semantic features, and the emotional features were spliced and fused. Finally, the Softmax classifier was applied for emotion classification. The findings of a comparative experiment on a set of microblog text comments collected throughout the pandemic revealed that the accuracy, recall, and F1 value of the emotion classification method proposed in this paper reached 92.33%, 91.78%, and 91.52%, respectively, which was a significant improvement compared with other models. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)