1. Natural language modeling with syntactic structure dependency.
- Author
-
Shuang, Kai, Tan, Yijia, Cai, Zhun, and Sun, Yue
- Subjects
- *
NATURAL languages , *ARTIFICIAL neural networks , *RECURRENT neural networks , *NEUROLINGUISTICS - Abstract
• The syntactic structure of natural language is import for the model to understand the sentences comprehensively. • The relative syntactic distance can be used to describe the degree of dependency between adjacent words in a sentence. • A relative syntactic distance-based language model RSD-LSTM is proposed to integrate syntactic information into LSTM. • Low-level and high-level syntactic structure features can be fused to generate hidden representations. • The experimental results of RSD-LSTM outperform existing state-of-the-art models on different tasks. In natural language, the relationship among the constituents of a sentence is usually tree-like: words, phrases, and clauses constitute a sentence hierarchically, and the dependency between different constituents induces the syntactic structure. Such a complex tree-like structure is vital for understanding natural languages. However, recurrent neural networks (RNNs) model languages sequentially and fail to encode a hierarchical syntactic dependency comprehensively, therefore causing the networks to underperform on comprehension-based tasks. In this paper, we propose a novel neural language model, called relative syntactic distance LSTM (RSD-LSTM), to capture the syntactic structure dependency dynamically. RSD-LSTM employs a convolutional neural network to compute the relative syntactic distance between sentences to represent the degree of dependency between words and modifies the gating mechanism of LSTM through the relative syntactic distance. Furthermore, we add a direct connection between hidden states to fuse high-level and low-level syntactic features. We conducted extensive experiments on language modeling. The results suggest that RSD-LSTM achieves improvements of 1.82 and 2.03 in perplexity compared with current top methods on the Penn Treebank and WikiText-2 datasets, respectively. Moreover, we conducted experiments on a machine translation application task. Experimental results of this task also show significant improvements of RSD-LSTM compared with baseline models. [ABSTRACT FROM AUTHOR]
- Published
- 2020
- Full Text
- View/download PDF