Search

ive summarization aims to reproduce the essential information of a source document in a summary by using the summarizer's own words. Although this approach is more similar to how humans summarize, it is more challenging to automate as it requires a complete understanding of natural language. However, the development of deep learning approaches, such as the sequence-to-sequence model with an attention-based mechanism, and the availability of pre-trained language models have led to improved performance in summarization systems. Nonetheless, abstractive summarization still suffers from issues such as hallucination and unfaithfulness. To address these issues, we propose an approach that utilizes a guidance signal using important Elementary Discourse Units (EDUs). We compare our work with previous guided summarization and two other summarization models that enhanced the faithfulness of the summary. Our approach was tested on CNN/Daily Mail dataset, and results showed an improvement in both truthfulness and good quantity coverage of the source document.

ive text summarization, in comparison to extractive text summarization, offers the potential to generate more accurate summaries. In our work, we present a stage-wise abstractive text summarization model that incorporates Elementary Discourse Unit (EDU) segmentation, EDU selection, and EDU fusion. We first segment the articles into a fine-grained form, EDUs, and build a Rhetorical Structure Theory (RST) tree for each article in order to represent the dependencies among EDUs; those EDUs are encoded in Graph Attention Networks (GATs); those with higher importance will be selected as candidates to be fused and the fusing stage is done by Bidirectional and Auto-Regressive Transformers (BART) model which merges the selected EDUs into summaries. A Greedy Method is leveraged to greedily select those EDUs whose combinations can maximize the ROUGE scores. Our model outperforms the baseline of BART (large) on the CNN/Daily Mail dataset, showing its effectiveness in abstractive text summarization.

Question generation is the parallel task of question answering, where given an input context and, optionally, an answer, the goal is to generate a relevant and fluent natural language question. Although recent works on question generation have experienced success by utilizing sequence-to-sequence models, there is a need for question generation models to handle increasingly complex input contexts to produce increasingly detailed questions. Multi-hop question generation is a more challenging task that aims to generate questions by connecting multiple facts from multiple input contexts. In this work, we apply a transformer model to the task of multi-hop question generation without utilizing any sentence-level supporting fact information. We utilize concepts that have proven effective in single-hop question generation, including a copy mechanism and placeholder tokens. We evaluate our model’s performance on the HotpotQA dataset using automated evaluation metrics, including BLEU, ROUGE and METEOR and show an improvement over the previous work.

In Natural Language Processing, multi-document summarization (MDS) poses many challenges to researchers. While advancements in deep learning approaches have led to the development of several advanced language models capable of summarization, the variety of approaches specific to the problem of multi-document summarization remains relatively limited. Current state-of-the-art models produce impressive results on multi-document datasets, but the question of whether improvements can be made via the combination of these state-of-the-art models remains. This question is particularly relevant in few-shot and zero-shot applications, in which models have little familiarity or no familiarity with the expected output, respectively. To explore one potential method, we implement a query-relevance-focused approach which combines the pretrained models' outputs using maximal marginal relevance (MMR). Our MMR-based approach shows improvement over some aspects of the current state-of-the-art results while preserving overall state-of-the-art performance, with larger improvements occurring in fewer-shot contexts.

This thesis presents studies in neural text summarization for single and multiple documents.The focus is on using sentence paraphrasing and compression for generating fluent summaries, especially in multi-document summarization where there is data paucity. A novel solution is to use transfer-learning from downstream tasks with an abundance of data. For this purpose, we pre-train three models for each of extractive summarization, paraphrase generation and sentence compression. We find that summarization datasets – CNN/DM and NEWSROOM – contain a number of noisy samples. Hence, we present a method for automatically filtering out this noise. We combine the representational power of the GRU-RNN and TRANSFORMER encoders in our paraphrase generation model. In training our sentence compression model, we investigate the impact of using different early-stopping criteria, such as embedding-based cosine similarity and F1. We utilize the pre-trained models (ours, GPT2 and T5) in different settings for single and multi-document summarization.

In Natural Language Processing, researchers find many challenges on Query Focused Abstractive Summarization (QFAS), where Bidirectional Encoder Representations from Transformers for Summarization (BERTSUM) can be used for both extractive and abstractive summarization. As there is few available datasets for QFAS, we have generated queries for two publicly available datasets, CNN/Daily Mail and Newsroom, according to the context of the documents and summaries. To generate abstractive summaries, we have applied two different approaches, which are Query focused Abstractive and Query focused Extractive then Abstractive summarizations. In the first approach, we have sorted the sentences of the documents from the most query-related sentences to the less query-related sentences, and in the second approach, we have extracted only the query related sentences to fine-tune the BERTSUM model. Our experimental results show that both of our approaches show good results on ROUGE metric for CNN/Daily Mail and Newsroom datasets.

Query-based summarization problem is an interesting problem in the text summarization field. On the other hand, the reinforcement learning technique is popular for robotics and becoming accessible for the text summarization problem in the last couple of years (Narayan et al., 2018). The lack of significant works using reinforcement learning to solve the query-based summarization problem inspired us to use this technique. While doing so, We also introduce a different approach for sentence ranking and clustering to avoid redundancy in summaries. We propose an unsupervised extractive summarization method, which provides state-of-the-art results on some metrics. We develop two abstractive multi-document summarization models using the reinforcement learning technique and the transformer model (Vaswani et al., 2017). We consider the importance of information coverage and diversity under a fixed sentence limit for our summarization models. We have done several experiments for our proposed models, which bring significant results across different evaluation metrics.

Query Focused Summarization (QFS) summarizes a long document with respect to a given input query. Creating a query-focused abstractive summary by using a neural network model is a difficult task which is yet to be fully solved. In our thesis, we propose two neural network models for the query-focused abstractive summarization task. We propose a model based on the sequence-to-sequence architecture with a pointer-generator mechanism. Furthermore, we also use the transformer architecture to design a model for the abstractive summarization. Afterward, we train both our models with the Debatepedia dataset so that the model can learn to summarize a long document with respect to a query. We evaluate the output of our models against the human-created reference summary. Our transformer model outperforms our sequence-to-sequence model in all ROUGE scores.

Query-focused abstractive document summarization (QFADS) is a process of shortening a document into a summary while keeping the context of query in mind. We implemented a model consisting of a novel selective mechanism for QFADS. A selective mechanism was used for improving the representation of a long input (passage) sequence. We conducted experiments on the Debatepedia dataset, a recently developed dataset for query-focused abstractive summarization task, which showed that our model outperforms the state-of-the-art model in all ROUGE scores. Also, we proposed three models all of which consist of a coarse-to-fine approach and a novel selective mechanism for query-focused abstractive multi document summarization (QFAMDS). The coarse-to-fine approach was used to reduce the length of the passage input from multiple documents. We conducted experiments on the MS MARCO dataset, a recently developed large scale dataset by Microsoft for reading comprehension, and have reported our scores using various evaluation metrics.

In this thesis, we have approached a technique for tackling abstractive text summarization tasks with state-of-the-art results. We have proposed a novel method to improve multidocument summarization. The lack of large multi-document human-authored summaries needed to train seq2seq encoder-decoder models and the inaccuracy in representing multiple long documents into a fixed size vector inspired us to design complementary models for two different tasks such as sentence clustering and neural sentence fusion. In this thesis, we minimize the risk of producing incorrect fact by encoding a related set of sentences as an input to the encoder. We applied our complementary models to implement a full abstractive multi-document summarization system which simultaneously considers importance, coverage, and diversity under a desired length limit. We conduct extensive experiments for all the proposed models which bring significant improvements over the state-of-the-art methods across different evaluation metrics.