1. STD: An Automatic Evaluation Metric for Machine Translation Based on Word Embeddings
- Author
-
Wujie Zheng, Zibin Zheng, Fanghua Ye, Yuetang Deng, Pairui Li, and Chuan Chen
- Subjects
Matching (statistics) ,Acoustics and Ultrasonics ,Machine translation ,Computer science ,business.industry ,computer.software_genre ,Computational Mathematics ,Range (mathematics) ,Semantic similarity ,Metric (mathematics) ,Computer Science (miscellaneous) ,NIST ,Artificial intelligence ,Electrical and Electronic Engineering ,business ,computer ,Natural language processing ,Word (computer architecture) ,Word order - Abstract
Lexical-based metrics such as BLEU, NIST, and WER have been widely used in machine translation MT evaluation. However, these metrics badly represent semantic relationships and impose strict identity matching, leading to moderate correlation with human judgments. In this paper, we propose a novel MT automatic evaluation metric Semantic Travel Distance STD based on word embeddings. STD incorporates both semantic and lexical features word embeddings and n-gram and word order into one metric. It measures the semantic distance between the hypothesis and reference by calculating the minimum cumulative cost that the embedded n-grams of the hypothesis need to “travel” to reach the embedded n-grams of the reference. Experiment results show that STD has a better and more robust performance than a range of state-of-the-art metrics for both the segment-level and system-level evaluation.
- Published
- 2019