Back to Search
Start Over
Applying large language models for automated essay scoring for non-native Japanese
- Source :
- Humanities & Social Sciences Communications, Vol 11, Iss 1, Pp 1-15 (2024)
- Publication Year :
- 2024
- Publisher :
- Springer Nature, 2024.
-
Abstract
- Abstract Recent advancements in artificial intelligence (AI) have led to an increased use of large language models (LLMs) for language assessment tasks such as automated essay scoring (AES), automated listening tests, and automated oral proficiency assessments. The application of LLMs for AES in the context of non-native Japanese, however, remains limited. This study explores the potential of LLM-based AES by comparing the efficiency of different models, i.e. two conventional machine training technology-based methods (Jess and JWriter), two LLMs (GPT and BERT), and one Japanese local LLM (Open-Calm large model). To conduct the evaluation, a dataset consisting of 1400 story-writing scripts authored by learners with 12 different first languages was used. Statistical analysis revealed that GPT-4 outperforms Jess and JWriter, BERT, and the Japanese language-specific trained Open-Calm large model in terms of annotation accuracy and predicting learning levels. Furthermore, by comparing 18 different models that utilize various prompts, the study emphasized the significance of prompts in achieving accurate and reliable evaluations using LLMs.
- Subjects :
- History of scholarship and learning. The humanities
AZ20-999
Social Sciences
Subjects
Details
- Language :
- English
- ISSN :
- 26629992
- Volume :
- 11
- Issue :
- 1
- Database :
- Directory of Open Access Journals
- Journal :
- Humanities & Social Sciences Communications
- Publication Type :
- Academic Journal
- Accession number :
- edsdoj.439a835311b94accb8aef695115998b8
- Document Type :
- article
- Full Text :
- https://doi.org/10.1057/s41599-024-03209-9