Serious Game Analytics by Design: Feature Generation and Selection Using Game Telemetry and Game Metrics--Toward Predictive Model Construction

The construction of prediction models reflecting players' learning performance in serious games currently faces various challenges for learning analytics. In this study, we design, implement, and field test a learning analytics system for a serious game, advancing the field by explicitly showing which in-game features correspond to differences in learner performance. We then deploy and test a system that provides instructors with clear signals regarding student learning and progress in the game, which instructors could depend upon for interventions. Within the study, we examined, coded, and filtered a substantial gameplay corpus, determining expertise in the game. Mission HydroSci (MHS) is a serious game that teaches middle-school students water science. Using our logging system, designed and implemented along with game design and development, we captured around 60 in-game features from the gameplay of 373 students who completed Unit 3 of MHS in its first field test. We tested eight hypotheses during the field test and presented this paper's results to participating teachers. Our findings reveal several features with statistical significance that will be critical for creating a validated prediction model. We discuss how this work will help future research establish a framework for designing analytics systems for serious games and advancing gaming design and analytics theory.