Search

Phishing and spam are examples of unsolicited emails, result in significant financial losses for businesses and individuals every year. Numerous methodologies and strategies have been devised for the automated identification of spam, yet they have not demonstrated complete predictive precision. Within the spectrum of suggested methodologies, ML and DL algorithms have shown the most promising results. This article scrutinizes the outcomes of assessing the efficacy of three transformation-based models - BERT, AlBERT, and RoBERTa - in scrutinizing both textual and numerical data. The proposed models achieved higher accuracy and efficiency in classification tasks, which was a notable improvement above traditional models such as KNN, NB, BiLSTM, and LSTM. Interestingly, in several criteria the Roberta model achieved almost perfect accuracy, suggesting that it is very flexible on a variety of datasets. [ABSTRACT FROM AUTHOR]

Transfer learning models have proven superior to classical machine learning approaches in various text classification tasks, such as sentiment analysis, question answering, news categorization, and natural language inference. Recently, these models have shown exceptional results in natural language understanding (NLU). Advanced attention‐based language models like BERT and XLNet excel at handling complex tasks across diverse contexts. However, they encounter difficulties when applied to specific domains. Platforms like Facebook, characterized by continually evolving casual and sophisticated language, demand meticulous context analysis even from human users. The literature has proposed numerous solutions using statistical and machine learning techniques to predict the sentiment (positive or negative) of online customer reviews, but most of them rely on various business, review, and reviewer features, which leads to generalizability issues. Furthermore, there have been very few studies investigating the effectiveness of state‐of‐the‐art pre‐trained language models for sentiment classification in reviews. Therefore, this study aims to assess the effectiveness of BERT, RoBERTa, ALBERT, DistilBERT, and XLNet in sentiment classification using the Yelp reviews dataset. The models were fine‐tuned, and the results obtained with the same hyperparameters are as follows: 98.30 for RoBERTa, 98.20 for XLNet, 97.40 for BERT, 97.20 for ALBERT, and 96.00 for DistilBERT. [ABSTRACT FROM AUTHOR]

In natural language processing (NLP), a Question Answering System (QAS) refers to a system or model that is designed to understand and respond to user queries in natural language. As we navigate through the recent advancements in QAS, it can be observed that there is a paradigm shift of the methods used from traditional machine learning and deep learning approaches towards transformer-based language models. While significant progress has been made, the utilization of these models for historical QAS and the development of QAS for Malay language remain largely unexplored. This research aims to bridge the gaps, focusing on developing a Multilingual QAS for history of Malaysia by utilizing a transformer-based language model. The system development process encompasses various stages, including data collection, knowledge representation, data loading and pre-processing, document indexing and storing, and the establishment of a querying pipeline with the retriever and reader. A dataset with a collection of 100 articles, including web blogs related to the history of Malaysia, has been constructed, serving as the knowledge base for the proposed QAS. A significant aspect of this research is the use of the translated dataset in English instead of the raw dataset in Malay. This decision was made to leverage the effectiveness of well-established retriever and reader models that were trained on English data. Moreover, an evaluation dataset comprising 100 question-answer pairs has been created to evaluate the performance of the models. A comparative analysis of six different transformer-based language models, namely DeBERTaV3, BERT, ALBERT, ELECTRA, MiniLM, and RoBERTa, has been conducted, where the effectiveness of the models was examined through a series of experiments to determine the best reader model for the proposed QAS. The experimental results reveal that the proposed QAS achieved the best performance when employing RoBERTa as the reader model. Finally, the proposed QAS was deployed on Discord and equipped with multilingual support through the incorporation of language detection and translation modules, enabling it to handle queries in both Malay and English. Doi: 10.28991/ESJ-2024-08-02-019 Full Text: PDF

Copyright of International Journal of InformaticsTechnologies is the property of Institute of Informatics, Gazi University and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Information retrieval and text mining fields extensively utilize text semantic matching models. In this paper, civil litigation Q&A under the environmental justice system is taken as a specific research field, and after constructing a civil litigation Q&A system based on deep learning, two of the key techniques—question categorization and semantic matching—are selected as the main research content. Specifically, the ALBERT algorithm is used to extract word vectors, and the hidden feature vectors are obtained through BiLSTM modeling of contextual relationships and then combined with the Attention mechanism for scoring and weighting to obtain the final text-level vectors for classification so as to establish the civil litigation question classification model based on ALBERT. Then, we establish the BERT-based civil litigation question and answer matching model by sorting the set of candidate answers by semantic matching degree based on the BERT algorithm. Selected datasets and comparison algorithms are experimented with, and the analysis shows that the question classification model has a better effect than civil litigation question text classification, and the values of each index have been improved by 0.75%~3.00% on the basis of the baseline model. The MAP and MRR values (0.76~0.86) of the question-matching model are higher than those of the comparison model, verifying its superior performance in semantically assigning characters. The model proposed in this paper is more useful because it can provide civil litigation counseling to the public.

Summary: In today's world, there is a lot of anxiety about suicidal thoughts that is conveyed on social media platforms, and people are now sharing all kinds of feelings on social media forums. The majority of them utilize social forums because they feel uncomfortable sharing privately. The study's objective is to analyze suicidal thoughts and identify them at an early stage by utilizing deep learning and transfer learning techniques. These algorithms are used to data gathered from users of Reddit forums who have suicidal thoughts as well as regular users who have non‐suicidal thoughts. For the aforementioned goal, we use techniques such as the transfer learning algorithms BERT, RoBERTa, and ALBERT, as well as BiLSTM and other deep learning algorithms. In contrast to the sequence processing model, our study demonstrates that the bidirectional long‐short term algorithm provides the best validation accuracy, while the pretrained models BERT and ALBERT also provide satisfactory accuracy. [ABSTRACT FROM AUTHOR]

Cyber threat intelligence entity identification is the key to cyber threat intelligence analysis. In view of the fact that traditional word embedding cannot represent the polysemy of a word, it is difficult to effectively identify the key information of cyber threat intelligence entities, and at the same time, facing the exponential growth of threat intelligence, the efficiency of the identification model needs to be improved urgently. A network threat intelligence named entity recognition model based on ALBERT is proposed. The model first uses ALBERT to extract threat intelligence dynamic feature word vectors. Then, the feature word vector is input to the bidirectional long short-term memory network (BiLSTM) layer to obtain the corresponding label of each word in the sentence. Finally, the conditional random field (CRF) layer is modified and the sequence label is output with the maximum probability. The experimental results of identification model comparison show that the F1 value of the proposed model is 92.21%, which is obviously better than other models. In the case of the same recognition accuracy, the time and resource costs of the proposed model are also lower, which is suitable for massive and efficient entity recognition tasks in the field of cyber threat intelligence. [ABSTRACT FROM AUTHOR]

The classification performance is highly affected because hyperspectral images include many bands, have high dimensions, and have few labeled training samples. This challenge is reduced by using rich spatial information and an effective classifier. The classifiers in this study are BERT-based (Bidirectional Encoder Representations from Transformers) models, which have recently been applied in natural language processing. The BERT model and its performance-improved version, the ALBERT (A Lite BERT) model, are utilized as transformer-based models. Because of their structure, these models can also accept spatial information via ‘segment embeddings’. Segmentation algorithms are commonly used in the literature to get spatial information. Superpixel methods have shown superior results in the segmentation literature due to the utility of working at the superpixel level rather than the conventional pixel level. HyperSLIC, a modified version of the SLIC superpixel method for hyperspectral images, is employed as input to BERT-based models in this study. In addition, HyperSLIC segmentation results are merged with the DBSCAN algorithm for similar superpixels to increase the size of spatially similar areas and called as HyperSLIC-DBSCAN. The effects of segment embedding information on classification accuracy in BERT-based models is studied experimentally. Experimental results show that BERT-based models outperform conventional and deep learning-based 1D/2D convolutional neural network classifiers when spatial information is used with the help of segment embedding information.

Large-scale pre-trained language representation and its promising performance in various downstream applications have become an area of interest in the field of natural language processing (NLP). There has been huge interest in further increasing the model's size in order to outperform the best previously obtained performances. However, at some point, increasing the model's parameters may lead to reaching its saturation point due to the limited capacity of GPU/TPU. In addition to this, such models are mostly available in English or a shared multilingual structure. Hence, in this paper, we propose a lite BERT trained on a large corpus solely in the Romanian language, which we called "A Lite Romanian BERT (ALR-BERT)". Based on comprehensive empirical results, ALR-BERT produces models that scale far better than the original Romanian BERT. Alongside presenting the performance on downstream tasks, we detail the analysis of the training process and its parameters. We also intend to distribute our code and model as an open source together with the downstream task. [ABSTRACT FROM AUTHOR]

Large-scale pre-trained language representation and its promising performance in various downstream applications have become an area of interest in the field of natural language processing (NLP). There has been huge interest in further increasing the model’s size in order to outperform the best previously obtained performances. However, at some point, increasing the model’s parameters may lead to reaching its saturation point due to the limited capacity of GPU/TPU. In addition to this, such models are mostly available in English or a shared multilingual structure. Hence, in this paper, we propose a lite BERT trained on a large corpus solely in the Romanian language, which we called “A Lite Romanian BERT (ALR-BERT)”. Based on comprehensive empirical results, ALR-BERT produces models that scale far better than the original Romanian BERT. Alongside presenting the performance on downstream tasks, we detail the analysis of the training process and its parameters. We also intend to distribute our code and model as an open source together with the downstream task.

Stock price prediction can be made more efficient by considering the price fluctuations and understanding people’s sentiments. A limited number of models understand financial jargon or have labelled datasets concerning stock price change. To overcome this challenge, we introduced FinALBERT, an ALBERT based model trained to handle financial domain text classification tasks by labelling Stocktwits text data based on stock price change. We collected Stocktwits data for over ten years for 25 different companies, including the major five FAANG (Facebook, Amazon, Apple, Netflix, Google). These datasets were labelled with three labelling techniques based on stock price changes. Our proposed model FinALBERT is fine-tuned with these labels to achieve optimal results. We experimented with the labelled dataset by training it on traditional machine learning, BERT, and FinBERT models, which helped us understand how these labels behaved with different model architectures. Our labelling method’s competitive advantage is that it can help analyse the historical data effectively, and the mathematical function can be easily customised to predict stock movement.

Deep learning is advancing rapidly; it has aided in solving problems that were thought impossible. Natural language understanding is one such task that has evolved with the advancement of deep learning systems. There have been several sentiment analysis attempts, but they aim to classify it as a single emotion. Human emotion in natural language is generally a complex combination of emotions, which may be indeterminate or neutral at times. Neutrosophy is a branch of philosophy that identifies neutralities and uses membership functions (positive, negative, neutral) to quantify a sample into Single Valued Neutrosophic Set (SVNS) values. Our work aims to combine the power of deep learning with SVNS to represent a sample's sentiment into membership functions of SVNS. We have worked on the Offensive Language Identification Dataset (OLID). Combining the power of state-of-the-art neural network techniques with neutrosophy allowed us to quantify the sentiments and identify the transition phase between positive and negative ones. We used the transition phase to capture neutral samples, which is beneficial if we want to obtain purely positive/negative samples. We performed experiments using Bi-directional Long Short Term Memory (BiLSTM) with attention, Bidirectional Encoder Representations from Transformers (BERT), A Lite BERT (ALBERT), A Robustly Optimised BERT Approach (RoBERTa), and MPNet. Our SVNS model performed equivalent to state-of-the-art neural network models on the OLID dataset. Here, we propose a novel framework that can integrate with any neural network model and quantify sentiments using SVNS. • Sentiment analysis model using deep learning, neutrosophy, and transfer learning. • Analysing tweets as a combination of sentiments identifying neutralities. • Quantifying tweets into Single Valued Neutrosophic Sets (SVNS) for OLID dataset. • Experimental analysis using BiLSTM, BERT, RoBERTa, ALBERT, and MPNet. • Gaussian Mixture Model and k-means clustering algorithm for SVNS calculation. [ABSTRACT FROM AUTHOR]

Deep learning has been widely used in numerous real-world engineering applications and for classification problems. Real-world data is present with neutrality and indeterminacy, which neutrosophic theory captures clearly. Though both are currently developing research areas, there has been little study on their interlinking. We have proposed a novel framework to implement neutrosophy in deep learning models. Instead of just predicting a single class as output, we have quantified the sentiments using three membership functions to understand them better. Our proposed model consists of two blocks, feature extraction, and feature classification. Having a separate feature extraction block enables us to use any model as a feature extractor. We experimented with BiLSTM using GloVe (Global Vectors for word representation), BERT (Bidirectional Encoder Representations from Transformers), ALBERT (A Lite BERT), RoBERTa (Robustly optimized BERT approach), MPNet, and stacked ensemble models. Feature classification performs prediction and dimensionality reduction of features. Experimental analysis was done on the SemEval 2017 Task 4 dataset (Subtask A). We used the intermediate layer features to define membership functions of Single Valued Neutrosophic Sets (SVNS). We used these membership functions for prediction as well. We have compared our models with the top five teams of the task and recent state-of-the-art systems. Our proposed stacked ensemble model achieved the best recall (0.733) score. • Architecture for combining neutrosophy and deep learning for sentiment analysis. • Experimental analysis using LSTM, BERT, RoBERTa, ALBERT, MPNet, and Ensemble models. • Feature extraction using intermediate layers of deep learning models. • Using K-means and Cosine distance for calculating SVNS from generated features. • SVNS as an alternative to Softmax for multi-label sentiment analysis problem. [ABSTRACT FROM AUTHOR]