A vision-based multi-cues approach for individual students' and overall class engagement monitoring in smart classroom environments.

Student disengagement in higher education offline classroom learning has become a critical concern because of digital distractions and insufficient student engagement monitoring. As the student count increases, it becomes harder for teachers to monitor individual students' engagement and maintain class engagement throughout the lecture. Computer vision, affective computing, and deep learning technologies can potentially build vision-based intelligent applications that can automatically estimate student engagement in real-time. Existing methods have limitations like scalability, high computation, non-real-time performance, and inability to track individual students' engagement. This paper attempts to address this research gap by proposing a real-time vision-based multi-cues approach. The approach obtained three kinds of scores: First, an affect score is based on the positive and negative academic affective states. Second, an attention score is based on frontal, and non-frontal head pose estimation using Perspective-n-Point technique. Third, a head movement score is based on head displacement distance. A Facial Expression Recognition model has been developed for academic affective state classification. This was achieved by fine-tuning MobileNetV2 through an incremental unfreezing approach on our newly developed dataset named Classroom Spontaneous Facial Expression Dataset. The model achieved training, validation, and testing accuracies of 95.7%, 94.9%, and 76%, respectively. The three obtained scores are combined using weighted fusion method to estimate individual students' engagement labels: engaged or non-engaged. Subsequently, every student's labels are aggregated to estimate the overall class engagement. Additionally, when validated with students' engagement self-reports the proposed approach yielded a 75% and 80% correlation for individual students and overall class engagement, respectively. [ABSTRACT FROM AUTHOR]