Search

Recurrent Neural Networks (RNNs) have emerged as one of the most popular neural networks for processing time-series problems, widely used in machine translation, automatic speech recognition, and other natural language processing applications. However, conventional RNNs suffered from vanishing and exploding gradients, resulting in poor network performance in applications with long-term input information. As a variant of RNN, Long Short-Term Memory (LSTM) had been proposed to tackle this issue. Nevertheless, at the same time, LSTM introduces gating units and many additional parameters, which makes it challenging to be implemented directly on resource-limited platforms, such as Field Programmable Gate Arrays (FPGAs). This work first investigated the overall maximum achievable compression rates of different gating units and their correlations. Then, Gating Units Level Balanced Compression (GBC) strategy is proposed. After Top- $k$ pruning, the proposed GBC strategy can attain a compression rate of $36.6\times $ for LSTM. Further, the theoretical analysis indicates that for the existing gating units level LSTM compression variants, the GBC strategy still has further potential for compression. A complementary compression of the GBC strategy is performed on the existing coupled-gate LSTM to verify the analysis. Experimental results show that GBC achieves an additional $32\times $ (overall $42.7\times $) compression rate with negligible accuracy loss. Finally, hardware experiments conducted on Xilinx ADM-PCIE-7V3 FPGAs also demonstrate that the accelerator designed in this paper achieves an improvement of 7.4%-191.5% in energy efficiency compared to the state-of-the-art designs. [ABSTRACT FROM AUTHOR]

• Local and global self-attention mechanisms are used for character embedding. • CNN with multi-size filters are used to extract character information for NER. • A cross-attention method that fuses character and word embedding for NER is proposed • A modified Mogrifier LSTM is presented to improve the performance of NER. • Proposed methods integrated with a transformer-based model achieve good performance. Clinical and biomedical concept extraction is critical in medical analysis using clinical and biomedical documents from professional literature, EHRs and PHRs. Named entity recognition (NER) accurately marks essential information in the literature based on the characteristics of the target entity, providing a method for extracting clinical and biomedical concepts. The performance of NER is heavily embedding-dependent, so recent studies have proposed the method of generating word embedding from character-level information, which can strengthen the representation ability for word embedding. In this paper, we present a novel neural network model including an attention mechanism network and a convolutional neural network (CNN) to further improve character-level embedding. First, an attention mechanism is applied simultaneously to the local and global character embedding. Then, a CNN with multi-size filters is used to extract more information from the character level, which can capture more meaningful features from words with various lengths. In addition, a cross-attention method is used to leverage the interaction between word embedding and character embedding to generate the final word representation. Finally, we modified Mogrifier LSTM to make it suitable for NER tasks and integrated it into our model. Experimental results show that our method is effective and that the model performs better than the baseline models. We also apply our methods proposed in this paper to the transformer-based model and obtain a 90.36 F1-score on NCBI-Disease. [ABSTRACT FROM AUTHOR]

Commonsense knowledge bases play an essential role in a wide range of natural language processing tasks. This paper studies the problem of representation learning for commonsense knowledge bases to effectively incorporate their knowledge into numerical models. Most existing knowledge base representation learning methods are difficult to apply to commonsense knowledge bases since they are much sparser than general knowledge bases. Hence, in this paper, we propose a novel method for commonsense knowledge base representation learning. Specifically, we first model the nodes from multiple views, including word/phase information, context information, and graph information. Then, we design a scoring function to measure whether the commonsense triplets are established through relation representation learning. We conduct extensive experiments on two tasks and the results show that our proposed model outperforms other knowledge base representation learning methods. [ABSTRACT FROM AUTHOR]

Among the incoming challenges in monitoring systems, the aggregation, synthesis and management of knowledge through ontological structures hold an essential place. Existing knowledge extraction systems often use a supervised approach that relies on annotated data, inducing implicitly a fastidious annotation process. Current research is towards the definition of unsupervised or semi-supervised systems, allowing a wider range of knowledge extraction. The evaluation of such systems, performing knowledge extraction using natural language processing methods requires performance indicators. The indicators usually used in such evaluations have limitations in the specific context of knowledge extraction for unsupervised ontology population. Thus, the definition of new evaluation methods becomes a need arising from the singularity of the harvested data, especially when these are not annotated. Hence, this article proposes a method for measuring performance in unsupervised context where reference data and extracted data do not overlap optimally. The proposed evaluation method is based on the exploitation of data that serve as a reference but are not specifically linked to the data used for extraction, which makes it an original evaluation method. To apply the performance measure on concrete cases, this paper also presents an unsupervised self-feeding rule-based approach for domain-independent ontology population from textual data. • Automated validation of relations depends strongly on chosen similarity measure. • ROUGE score is better at comparing ontology instances' meaning than edit distances. • Reuse of prior consensual knowledge can help reduce expert bias in validation step. • Accuracy of automated validation is sensitive to the size of reference data used. [ABSTRACT FROM AUTHOR]

It has been a long-standing goal in natural language processing (NLP) to learn a general linguistic intelligence model that can perform well on many different NLP tasks continually evolving over time while avoiding revisiting all previous data at each stage. Most existing deep neural networks suffer from catastrophic forgetting when dealing with sequential tasks in an incremental way, leading to dramatic performance degradation due to the missing training data of old tasks. In this paper, we propose a Lifelong language method with Adaptive Uncertainty Regularization (LAUR), which can adapt a single BERT model to work with continuously arriving text examples from different NLP tasks. Specifically, LAUR is built on the Bayesian online learning framework, and three uncertainty regularization terms are devised to collaboratively control the parameters so as to resolve the stability-plasticity dilemma in lifelong language learning. The previous posterior constrain parameters that strongly determine the output results, preventing these parameters from changing drastically, while other parameters are encouraged to be updated over time. In addition, we propose a task-specific residual adaptation module in parallel to each layer of BERT to endow LAUR with the capacity to learn better task-specific knowledge. This configuration makes LAUR less prone to losing the knowledge stored in the base BERT network when learning a new task. Experimental results show that LAUR outperforms state-of-the-art lifelong learning models on a variety of NLP tasks. For reproducibility, we submit the code and data at: https://github.com/kiujhytgtrfd2021/LAUR. [ABSTRACT FROM AUTHOR]

It has been a long-standing goal in natural language processing (NLP) to learn a general linguistic intelligence model that can perform well on many different NLP tasks continually evolving over time while avoiding revisiting all previous data at each stage. Most existing deep neural networks suffer from catastrophic forgetting when dealing with sequential tasks in an incremental way, leading to dramatic performance degradation due to the missing training data of old tasks. In this paper, we propose a Lifelong language method with Adaptive Uncertainty Regularization (LAUR), which can adapt a single BERT model to work with continuously arriving text examples from different NLP tasks. Specifically, LAUR is built on the Bayesian online learning framework, and three uncertainty regularization terms are devised to collaboratively control the parameters so as to resolve the stability-plasticity dilemma in lifelong language learning. The previous posterior constrain parameters that strongly determine the output results, preventing these parameters from changing drastically, while other parameters are encouraged to be updated over time. In addition, we propose a task-specific residual adaptation module in parallel to each layer of BERT to endow LAUR with the capacity to learn better task-specific knowledge. This configuration makes LAUR less prone to losing the knowledge stored in the base BERT network when learning a new task. Experimental results show that LAUR outperforms state-of-the-art lifelong learning models on a variety of NLP tasks. For reproducibility, we submit the code and data at: https://github.com/kiujhytgtrfd2021/LAUR. [ABSTRACT FROM AUTHOR]

Human behavior may be one of the most challenging phenomena to model and validate. This paper proposes a method for automatically extracting and compiling evidence on human behavior determinants into a knowledge graph. The method (1) extracts associations of behavior determinants and choice options in relation to study groups and moderators from published studies using Natural Language Processing and Deep Learning, (2) synthesizes the extracted evidence into a knowledge graph, and (3) sub-selects the model components and relationships that are relevant and robust. The method can be used to either (4a) construct a structurally valid simulation model before proceeding with calibration or (4b) to validate the structure of existing simulation models. To demonstrate the feasibility of the method, we discuss an example implementation with mode of transport as behavior choice. We find that including non-frequently studied significant behavior determinants drastically improves the model's explanatory power in comparison to only including frequently studied variables. The paper serves as a proof-of-concept which can be reused, extended or adapted for various purposes. • The structure of behavior models should be validated using existing evidence. • Deep Learning can help automatize behavior evidence extraction and synthesis. • Infrequently studied significant variables improve model performance drastically. [ABSTRACT FROM AUTHOR]

In response to fighting COVID-19 pandemic, researchers in machine learning and artificial intelligence have constructed some medical knowledge graphs (KG) based on existing COVID-19 datasets, however, these KGs contain a considerable amount of semantic relations which are incomplete or missing. In this paper, we focus on the task of knowledge graph embedding (KGE), which serves an important solution to infer the missing relations. In the past, there have been a collection of knowledge graph embedding models with different scoring functions to learn entity and relation embeddings published. However, these models share the same problems of rarely taking important features of KG like attribute features, other than relation triples, into account, while dealing with the heterogeneous, complex and incomplete COVID-19 medical data. To address the above issue, we propose a graph feature collection network (GFCNet) for COVID-19 KGE task, which considers both neighbor and attribute features in KGs. The extensive experiments conducted on the COVID-19 drug KG dataset show promising results and prove the effectiveness and efficiency of our proposed model. In addition, we also explain the future directions of deepening the study on COVID-19 KGE task. [ABSTRACT FROM AUTHOR]

We investigate the problem of structured encryption (STE) for knowledge graphs (KGs) where the knowledge of data can be efficiently and privately queried. Presently, the application of natural language processing (NLP) for knowledge-based search is gradually emerging. Compared with the traditional search based only on keywords of documents—symmetric searchable encryption (SSE), the knowledge-based search system transforms the latent knowledge contained in documents into a semantic network as a knowledge base, which greatly improves the accuracy and relevance of search results. In order to develop a knowledge-based search, the contents of documents are analyzed and extracted using KG techniques (e.g. multi-relational graph (MG) and property graph (PG)), and then all encrypted nodes and edges in a KG constitute the entire index table and database. This paper proposes the first STE for KGs with CQA2-security to search on protected knowledge, where KGs include MGs and PGs. In general, the latter is more complex than the former, but it can represent more abundant knowledge. Experimental results show that the index construction time of our schemes is about 1.9s and the query time is about 190 ms. Our sensitivity analysis shows that the performance of our proposed schemes is greatly influenced by the number of edges and nodes, but less by the number of properties. [ABSTRACT FROM AUTHOR]

Aspect-based sentiment classification has become a popular topic in natural language processing. Exploiting dependency syntactic information with graph neural networks has recently become a popular trend. Despite their success, methods that rely heavily on a dependency tree face major challenges. This concerns the alignment of aspects and their word sentiments due to the richness of the language and the fact that a dependency tree might produce noisy signals from unrelated associations. This paper introduces a Dual-Relational Graph Attention Network (DRGAT) that fully exploits syntactic structural information and then the sentiment-aware context (e.g., phrase segmentation and hierarchical structure) of the constituent tree of a sentence. Additional constituency and dependency attention mechanisms provide comprehensive syntactic information across words, thereby enabling an accurate connection between aspect words and corresponding sentiment words. Considering that the original parsed constituency tree may have a large depth, this could lead to words being far apart increasing the computational overhead. The constituency tree of each sentence is dynamically reconstructed by determining the importance of each relational node. Extensive experimental results on six English datasets demonstrated that fully exploiting syntactic information can achieve excellent sentiment classification results. [ABSTRACT FROM AUTHOR]

A primary focus of the healthcare industry is to improve patient experience and quality of service. Practitioners and health workers are generating large volumes of text that are captured in Electronic Medical Records, clinical reports, and publications. Additionally, patients post millions of comments on social media related to healthcare, on diverse topics such as hospital services, disease symptoms, and drugs effects. Unifying various data sources can guide physicians and healthcare workers to avoid unnecessary, irrelevant information and expedite access to helpful information. The main challenge to creating Biomedical Natural Language Understanding is the lack of standard datasets and the extensive computational resources needed to develop different models. This paper proposes a model trained on low-tier GPU computers, producing comparable results to larger models like BioBERT. We propose BINER, a Biomedical Named Entity Recognition architecture using limited data and computational resources. [ABSTRACT FROM AUTHOR]

• An Italian BERT model (AlBERTo) has been fine-tuned on financial sentences. • AlBERTino can determine the sentiment score of news present in financial newspapers. • The sentiment score is used to drive the parameters of GBM through a MCMC. • The average of Monte Carlo simulation paths is the predicted stock value. BERT (Bidirectional Encoder Representations from Transformers) is one of the most popular models in Natural Language Processing (NLP) for Sentiment Analysis. The main goal is to classify sentences (or entire texts) and to obtain a score in relation to their polarity: positive, negative or neutral. Recently, a Transformer-based architecture, the fine-tuned AlBERTo (Polignano et al. (2019)), has been introduced to determine a sentiment score in the financial sector through a specialized corpus of sentences. In this paper, we use the sentiment (polarity) score to improve the stocks forecasting. We apply the BERT model to determine the score associated to various events (both positive and negative) that have affected some stocks in the market. The sentences used to determine the scores are newspaper articles published on MilanoFinanza. We compute both the average sentiment score and the polarity, and we use a Monte Carlo method to generate (starting from the day the article was released) a series of possible paths for the next trading days, exploiting the Bayesian inference to determine a new series of bounded drift and volatility values on the basis of the score; thus, returning an exact "directed" price as a result. [ABSTRACT FROM AUTHOR]

• A novel target attention convolutional neural network is proposed. • The proposed method fully utilized the associated information between entities and relations. • The proposed method does not need to introduce external complicated features. • The proposed method achieves state-of-the-art performance on SemEval-2010 task 8 and Conll04 datasets. Relation classification (RC) is an essential task in natural language processing (NLP), which extracts relationships of entity pairs in sentences of text. In the paper, a novel target attention convolutional neural network (TACNN) is proposed for the RC by fully utilizing word embedding information and position embedding information. Simultaneously, a target attention mechanism (TAM) is applied into a context layer of the convolutional neural network (CNN) model, which increases the effect of the relationship matrix weights of two entities in the sentence, while ignoring the calculation of irrelevant terms. And the TACNN is essentially to modify the weight of the relationship matrix of entities in the sentence at the context layer and connect the relationship feature composed of the lexical layer feature with the target attention layer feature. Therefore, the TACNN simplifies the structure of the CNN and improves the computational efficiency. On SemEval-2010 Task 8 dataset and Conll04 dataset, the TACNN obtains 85.3% and 71.4% of the F1-score, respectively. In contrast to previously available public models, the TACNN achieves a state-of-the-art level in the F1-score of the RC. [ABSTRACT FROM AUTHOR]

• An Italian BERT model (AlBERTo) has been fine-tuned on financial sentences. • AlBERTino can determine the sentiment score of news present in financial newspapers. • The sentiment score is used to drive the parameters of GBM through a MCMC. • The average of Monte Carlo simulation paths is the predicted stock value. BERT (Bidirectional Encoder Representations from Transformers) is one of the most popular models in Natural Language Processing (NLP) for Sentiment Analysis. The main goal is to classify sentences (or entire texts) and to obtain a score in relation to their polarity: positive, negative or neutral. Recently, a Transformer-based architecture, the fine-tuned AlBERTo (Polignano et al. (2019)), has been introduced to determine a sentiment score in the financial sector through a specialized corpus of sentences. In this paper, we use the sentiment (polarity) score to improve the stocks forecasting. We apply the BERT model to determine the score associated to various events (both positive and negative) that have affected some stocks in the market. The sentences used to determine the scores are newspaper articles published on MilanoFinanza. We compute both the average sentiment score and the polarity, and we use a Monte Carlo method to generate (starting from the day the article was released) a series of possible paths for the next trading days, exploiting the Bayesian inference to determine a new series of bounded drift and volatility values on the basis of the score; thus, returning an exact "directed" price as a result. [ABSTRACT FROM AUTHOR]

• A novel target attention convolutional neural network is proposed. • The proposed method fully utilized the associated information between entities and relations. • The proposed method does not need to introduce external complicated features. • The proposed method achieves state-of-the-art performance on SemEval-2010 task 8 and Conll04 datasets. Relation classification (RC) is an essential task in natural language processing (NLP), which extracts relationships of entity pairs in sentences of text. In the paper, a novel target attention convolutional neural network (TACNN) is proposed for the RC by fully utilizing word embedding information and position embedding information. Simultaneously, a target attention mechanism (TAM) is applied into a context layer of the convolutional neural network (CNN) model, which increases the effect of the relationship matrix weights of two entities in the sentence, while ignoring the calculation of irrelevant terms. And the TACNN is essentially to modify the weight of the relationship matrix of entities in the sentence at the context layer and connect the relationship feature composed of the lexical layer feature with the target attention layer feature. Therefore, the TACNN simplifies the structure of the CNN and improves the computational efficiency. On SemEval-2010 Task 8 dataset and Conll04 dataset, the TACNN obtains 85.3% and 71.4% of the F1-score, respectively. In contrast to previously available public models, the TACNN achieves a state-of-the-art level in the F1-score of the RC. [ABSTRACT FROM AUTHOR]

In recent years, many applications are using various forms of deep learning models. Such methods are usually based on traditional learning paradigms requiring the consistency of properties among the feature spaces of the training and test data and also the availability of large amounts of training data, e.g., for performing supervised learning tasks. However, many real-world data do not adhere to such assumptions. In such situations transfer learning can provide feasible solutions, e.g., by simultaneously learning from data-rich source data and data-sparse target data to transfer information for learning a target task. In this paper, we survey deep transfer learning models with a focus on applications to text data. First, we review the terminology used in the literature and introduce a new nomenclature allowing the unequivocal description of a transfer learning model. Second, we introduce a visual taxonomy of deep learning approaches that provides a systematic structure to the many diverse models introduced until now. Furthermore, we provide comprehensive information about text data that have been used for studying such models because only by the application of methods to data, performance measures can be estimated and models assessed. [ABSTRACT FROM AUTHOR]

Recent research has shown that the unaware automation of high-risk decision-making tasks can result in unfair decisions being made. The most common approaches to address this problem adopt definitions of fairness based on protected attributes. Precise annotation of protected attributes enables the application of bias mitigation techniques to commonly unlabeled kinds of data (e.g., images, text, etc.). This paper proposes a framework to automatically annotate protected attributes in data collections. The framework focuses on providing a single interface to annotate protected attributes of different types (e.g., gender, race, etc.) and from different kinds of data. Internally, the framework coordinates multiple sensors to produce the final annotation. Several sensors for textual data are proposed. An optimization search technique is designed to tune the framework to specific domains. Additionally, a small dataset of movie reviews —annotated with gender and sentiment— was created. The evaluation in datasets of texts from diverse domains shows the quality of the annotations and their effectiveness to be used as a proxy to estimate fairness in datasets and machine learning models. The source code is available online for the research community. • A framework to automatically annotate protected attributes in datasets. • Techniques to annotate gender in textual collections with fairness considerations. • Optimization search approach for tuning the framework to custom domains. • Small dataset of movie reviews annotated with gender and sentiment. • Effective use of annotations as a proxy to estimate fairness in datasets. [ABSTRACT FROM AUTHOR]

Natural language processing (NLP) backdoor attacks have become a hidden threat to modern NLP applications. Most of the existing defense methods defend against specific types of backdoor attacks, and they generally fail to defend against invisible backdoors with syntactically correct triggers. This paper proposes NLPSweep, a comprehensive defense scheme that can defend against five common types of backdoor attacks, namely, character, word, sentence, homograph, and learnable textual attacks. Specifically, we propose a framework that can discover an effective defense solution without prior knowledge of the attacks. The defense solution is optimized from the framework and can defend against various attacks while ensuring high accuracy. Finally, we verify the effectiveness of NLPSweep on two pretrained models (BERT and XLNET) on three classic datasets (SST-2, IMDB, and OLID) and compare it with five state-of-the-art defense methods, namely, ONION, Pred, RAP, Fine-pruning, and STRIP. The experimental results demonstrate that NLPSweep has an average model accuracy (ACC) greater than 0.922 and that the average attack success rate (ASR) is only 0.202, outperforming the compared methods. Furthermore, NLPSweep is tested on the real-world Yelp dataset and it can effectively defend against backdoor attacks with the ASR less than 0.07 and the ACC greater than 0.973.1 • Existing defense methods only defend against specific backdoor attacks. • Our defense framework can discover an effective defense solution without prior knowledge of the attacks. • Our defense solution can defend against various attacks while ensuring high accuracy. • Tests on three classic datasets show that our defense scheme outperforms recent work. • Our scheme demonstrates operational success on the real-world Yelp dataset, effectively defending against backdoor attacks. [ABSTRACT FROM AUTHOR]

With the capability of modeling lighter, MLP-based models like the pNLP-Mixer and the HyperMixer demonstrate the potential for diverse tasks in NLP. However, these linguistic models are not optimized for the regularity of textual hierarchical abstraction. Here, this paper proposes the hidden bias attention (HBA), a novel attention mechanism that is lighter than self-attention and focuses on extracting hidden (topic) semanteme. Additionally, this paper introduces a series of lightweight deep learning architectures, HBA-Mixer based on HBA and MHBA-Mixers based on multi-head HBA, which both outperforms pNLP-Mixer and HyperMixer in accuracy with fewer parameters on 3 tasks, including text classification, natural language inference, and sentiment analysis. Compared with large pre-trained models, MHBA-Mixers achieve over 90% of their accuracy with one-thousandth of the parameters. [ABSTRACT FROM AUTHOR]

Named entity recognition and relation extraction are crucial tasks in natural language processing. As the traditional pipelined manners may suffer from the error propagation issue and ignore underlying interactions, joint extraction of entities and relations has become the dominant trend. However, the performance of existing joint extraction models needs improvement. This paper presents a two-stage tagging scheme that separately labels candidate head entities and multiple tail entities in specific relations. Next, it proposes a novel lightweight joint extraction neural model based on the entity-first labeling strategy. In the proposed model, the BiLSTM-based encoder combines the hidden state and global context features and feeds them as input for the next two entity labeling tasks. Further, with the input of the mixed context representation, the candidate-head-entity recognition module is adopted to identify the candidate head entity, while the multiple-tail-entities recognition module is equipped with an entity-correlated attention mechanism to identify the corresponding tail entity under a specific head entity. Comprehensive experiments on two widely used English datasets and one self-constructed Chinese dataset were performed. The experimental results showed that the proposed model outperformed the baseline approaches in the relation extraction task and achieved a competitive entity recognition effect via a lightweight architecture. [ABSTRACT FROM AUTHOR]

Deep models have achieved prominent results in pattern recognition tasks, especially computer vision and natural language processing. However, the dataset bias caused by the distribution discrepancy between the training and testing data hinders the generalization ability of deep models. Though many domain adaptation approaches have been proposed to mitigate such negative effect, most of them improve the transferability of features by aligning global distributions of deep models. Few researchers pay attention to the versatility of deep features which can play a vital role in cross-domain recognition. In this paper, we propose to enrich the classic deep learning models by capturing high-low-frequency information and multi-scale features, which deal with the domain shift that cannot be easily addressed by merely feature-level alignment. The Hierarchical Transfer Network (HTN) leverages octave convolution, pyramid features, and self-attention mechanism for revamping the classic models, which can be further integrated with any domain alignment approaches by replacing the feature extractor with the proposed HTN. Extensive experiments have been conducted on three public domain adaptation benchmarks. The results show that the proposed HTN can effectively improve adversarial-based, statistics-based, and norm-based domain adaptation approaches, achieving competitive performance without involving model complexity. [ABSTRACT FROM AUTHOR]

• Ambiguity is a challenging task in text mining addressed for word-sense disambiguation algorithms. • The lack of labeled dataset is a barrier and semi-supervised learning (SSL) addresses this problem. • Our method explored SSL and word-embeddings CBOW, SKIP-GRAM, FASTTEXT, GLOVE, BERT and ELECTRA. • The F1-score increases in several datasets like Senseval-2, Senseval-3, Semeval-2007 and Semcor. Word Sense Disambiguation (WSD) aims to determine the meaning of a word in context. Different approaches have been proposed in supervised and unsupervised domains. In most cases, supervised learning provides superior WSD performance. Since sense-annotated corpora can be difficult or time-consuming to obtain, which must be repeated for new domains, languages, and sense inventories, semi-supervised learning (SSL) methods, that combine a small amount of sense-annotated data, start to be pre-eminent. In SSL, graph-based methods are common, because they capture the relationships between terms using an undirected graph. This paper aims to investigate semi-supervised WSD by considering different graph-based SSL algorithms with features generated by word embeddings from Word2Vec, FastText, GloVe, BERT and ELECTRA models combined with parts-of-speech tags and word context. We test several combinations of word-embedding models, similarity measures for graph construction and SSL classification algorithms to disambiguate classical lexical sample WSD datasets. The results indicate our SSL algorithms achieved competitive results compared to supervised ones and the ELECTRA models performed better than other embeddings for SSL. [ABSTRACT FROM AUTHOR]

Document-level relation extraction is a complex natural language processing task that predicts relations of entity pairs by capturing the critical semantic features on entity pairs from the document. However, current methods usually consider that the entity pairs contain the vast majority of information which can represent relational facts, and thus focus on modeling the entity pair, ignoring features on whole document and sentences. In the document-level relation extraction, the distance between entity pairs is relatively long. Judging the relation between entities usually requires reading many sentences or the whole document. Therefore, sentences and documents are particularly crucial for document-level relation extraction. In order to make full use of the multi-level information of sentences and documents, this paper proposes a document-level relation extraction framework with two advantages. First, we use the encoder to obtain the semantic features about the document and use the inter-sentence attention based on entity pairs to dynamically capture the features of multiple vital sentences. Second, we design a document gating that combines sentence-level features with document-level features to predict relations. Extensive experiments on a benchmark dataset have well-validated the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Identifying causality between events is a crucial research task in natural language processing. However, existing methods either ignore background knowledge of events or do not consider interference of knowledge graph noise on event representations. In this paper, we propose a novel Graph Contrast-based Knowledge Augmented Network (GCKAN) for event causality identification task, which integrates comprehensive background knowledge of events from knowledge graphs and alleviates the knowledge graph noise problem. First, a descriptive knowledge augmentation module is proposed to aggregate one-hop neighbor information in knowledge graphs and learn meaningful descriptive knowledge of events. Then, a relational knowledge augmentation module is designed to encode multi-hop path information and learn latent reasoning knowledge between event pairs. In addition, trustworthiness-based and degree-based graph contrastive learning schemas are devised in the two modules respectively, which suppress knowledge graph noise during information aggregation and derive more robust knowledge-aware event representations. Extensive experiments on three public datasets demonstrate the consistent superiority of GCKAN over state-of-the-art knowledge-based techniques. Noise interference experiments and cross-topic adaptation experiments further verify the robustness and generalization of GCKAN. • A graph contrast-based knowledge augmented network for event causality identification is proposed. • The model aggregates descriptive and relational knowledge from knowledge graphs and alleviates labeled data scarcity issue. • Graph contrastive learning schemes are devised to suppress knowledge noise and improve robustness of event representations. • Experimental results demonstrate effectiveness and robustness of the method. [ABSTRACT FROM AUTHOR]

• An approach for cleaning and reconciliation of tags in open data portal (ODP) categories using natural language processing methods. • A data structure for connecting ODPs – cross-portal metadata alignment structure generated using ODPs' tags. • Formal context reduction based on semantic similarity measure through means of natural language processing. • Cross-portal dataset categorization - determining appropriate category for a dataset using the metadata alignment structure. • Quality assessment using datasets obtained from Canada's and New Zealand's ODP. Due to openness and transparency initiatives, a vast amount of data is being made publicly available. This data has great significance for business and society. However, it also led to challenges that need to be overcome for this data to reach its full potential. In this paper, we are focusing on the problem of connecting open data portals (ODPs) through metadata alignment. We investigate the available metadata accompanying datasets, especially the part related to categories datasets belong to and tags that closely describe datasets. The methodology we propose is relying on Formal Concept Analysis for the creation of the hierarchical structure used for determining the similarity of tags' usage in different ODPs. We propose such a structure to be used for open data portal metadata alignment. Further, we apply semantic similarity measures to reduce the complexity of the cross-portal data structure while preserving all its characteristics. We demonstrate how our approach can be used for determining dataset category across multiple ODPs aligned using the data structure our approach generates. We envision our approach to improve cross-portal search and metadata enrichment through open data categorization. Lastly, the quality of our approach was tested using datasets obtained from Canada's and New Zealand's ODPs. [ABSTRACT FROM AUTHOR]

As an emerging in-memory element, memristor has been widely used in various neural network circuits to represent the weights and accelerate the calculation. However, the Transformer Network (TN), one of the most important models for machine vision and natural language processing in recent years, has not yet been full-circuit implemented using memristors due to the complex calculation process and data storage. In order to carry out the computation of the TN more efficiently, this work proposes a memristor-based full-circuit implementation of the TN capable of: 1) a memristor crossbar module to preserve the weights of the TN and perform the vector-matrix multiplications; 2) an analog signal memory module to store the analog signal directly in near-memory mode; 3) function circuit modules to achieve five transformations, namely Softmax, Layer Normalization, ReLU, Multiply-add and Residual; 4) a timing signal generation module to schedule operations of the circuit. The proposed TN circuit can complete all calculations directly based on the analog signal without using any analog-digital converter (ADC), digital-analog converter (DAC) and digital memory. In addition, character image recognition experiments are carried out in PSPICE to verify the functional correctness of the designed circuit. The corresponding signal retention rates of the analog memory, the performance of the whole circuit, and the non-idealities of the memristors are also analyzed. The results indicate that the circuit has advantages in terms of area overhead, energy efficiency and anti-noise. [ABSTRACT FROM AUTHOR]