Back to Search Start Over

WordBytes: Exploring an Intermediate Constraint Format for Rapid Classification of Student Answers on Constructed Response Assessments

Authors :
Kim, Kerry J.
Meir, Eli
Pope, Denise S.
Wendel, Daniel
Source :
Journal of Educational Data Mining. 2017 9(2):45-71.
Publication Year :
2017

Abstract

Computerized classification of student answers offers the possibility of instant feedback and improved learning. Open response (OR) questions provide greater insight into student thinking and understanding than more constrained multiple choice (MC) questions, but development of automated classifiers is more difficult, often requiring training a machine learning system with many human-classified answers. Here we explore a novel intermediate constraint question format called WordBytes (WB) where students assemble one-sentence answers to two different college evolutionary biology questions by choosing, then ordering, fixed tiles containing words and phrases. We found WB allowed students to construct hundreds to thousands of different answers (= 20 tiles), with multiple ways to express correct and incorrect answers with different misconceptions. We found humans could specify rules for an automated WB grader that could accurately classify answers as correct/incorrect with Cohen's kappa = 0.88, near the measured intra-rater reliability of two human graders and the performance of machine classification of OR answers (Nehm et al., 2012). Finer-grained classification to identify the specific misconception had lower accuracy (Cohen's kappa < 0.75), which could be improved either by using a machine learner or revising the rules, but both would require considerably more development effort. Our results indicate that WB may allow rapid development of automated correct/incorrect answer classification without collecting and hand-grading hundreds of student answers.

Details

Language :
English
ISSN :
2157-2100
Volume :
9
Issue :
2
Database :
ERIC
Journal :
Journal of Educational Data Mining
Publication Type :
Academic Journal
Accession number :
EJ1166127
Document Type :
Journal Articles<br />Reports - Research