Back to Search Start Over

Assessing The Factual Accuracy of Generated Text

Authors :
Goodrich, Ben
Rao, Vinay
Saleh, Mohammad
Liu, Peter J
Source :
The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '19), August 4--8, 2019, Anchorage, AK, USA
Publication Year :
2019

Abstract

We propose a model-based metric to estimate the factual accuracy of generated text that is complementary to typical scoring schemes like ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and BLEU (Bilingual Evaluation Understudy). We introduce and release a new large-scale dataset based on Wikipedia and Wikidata to train relation classifiers and end-to-end fact extraction models. The end-to-end models are shown to be able to extract complete sets of facts from datasets with full pages of text. We then analyse multiple models that estimate factual accuracy on a Wikipedia text summarization task, and show their efficacy compared to ROUGE and other model-free variants by conducting a human evaluation study.

Details

Database :
arXiv
Journal :
The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '19), August 4--8, 2019, Anchorage, AK, USA
Publication Type :
Report
Accession number :
edsarx.1905.13322
Document Type :
Working Paper
Full Text :
https://doi.org/10.1145/3292500.3330955