1. Toward Utilizing Bidirectional Multi-head Attention Technique for Automatic Correction of Grammatical Errors.
- Author
-
Mahmoud, Zeinab, Kryvinska, Natalia, Abdalsalm, Mohammed, Solyman, Aiman, Alfatemi, Ali, and Musyafa, Ahmad
- Subjects
LOW-resource languages ,COMPUTATIONAL linguistics ,MACHINE translating ,CAPSULE neural networks ,ARABIC language - Abstract
Automatic Grammar Error Correction (GEC) models identify and correct a wide range of grammatical errors. Various strategies have been proposed for GEC, with the Neural Machine Translation (NMT) approach being the most effective. However, NMT-based GEC models with encoderdecoder layers rely heavily on the highest layer, leading to potential inaccuracies. Additionally, during inference, exposure bias can cause the model to substitute previously targeted words with incorrect alternatives. Another challenge is data scarcity. This paper introduces a GEC model leveraging the seq-to-seq Transformer framework, specifically designed for low-resource languages like Arabic. We propose a method to generate noise in the text to create synthetic parallel data, addressing the data constraints. Inspired by Capsule Networks (CapsNet), we incorporate CapsNet in GEC to dynamically aggregate information from multiple layers. In order to mitigate exposure bias, we incorporated a bidirectional training approach and a regularization term using Kullback-Leibler divergence to align left-toright and right-to-left models. Experiments on two benchmarks demonstrate that our model outperforms current Arabic GEC models, achieving the highest scores. The code is available on GitHub (https://github.com/Zainabobied/ArabicGEC). [ABSTRACT FROM AUTHOR]
- Published
- 2024