A Bi-modal Automated Essay Scoring System for Handwritten Essays

During the past few decades, Automated Essay Scoring (AES) technology has been widely used to alleviate the workload of teachers and improve the feedback cycle in educational systems. However, the scoring of handwritten essays poses great challenges for existing systems, since most of them only take text as input without consideration of errors or bias which may be introduced by Optical Character Recognition (OCR) as a necessary pre-processing step. This paper proposes VisualAES, a bimodal automated essay scoring system that utilizes both textual and visual features for handwritten essay scoring. Specifically, we first employ three powerful pre-trained transformer-based models as the backbone, and extend them to take both textual features and visual features extracted by Faster R-CNN. Then, a stacking ensemble model is subsequently adopted to robustly map their outputs to a final score. We evaluate VisualAES with the public Automated Student Assessment Prize (ASAP) dataset and our proposed handwritten Chinese Students' handwritten essay dataset (ChnStd). Results show that the proposed VisualAES outperforms all state-of-the-art methods on both datasets. More importantly, by incorporating handwritten image information, we also achieve a further performance improvement on ChnStd and reduce the side effects of OCR.