In Search of Negative Moments: Multi-Modal Analysis of Teacher Negativity in Classroom Observation Videos

We explore multi-modal machine learning-based approaches (facial expression recognition, auditory emotion recognition, and text sentiment analysis) to identify "negative moments" of teacher-student interaction during classroom teaching. Our analyses on a large (957 videos, each 20min) dataset of classroom observations suggest that: (1) Negative moments occur sparsely and are laborious to find by manually watching videos from start to finish; (2) Contemporary machine perception tools for emotion, speech, and text sentiment analysis show only limited ability to capture the diverse manifestations of classroom negativity in a fully automatic way; (3) Semi-automatic procedures that combine machine perception with human annotation may hold more promise for finding authentic moments of classroom negativity; Finally, (4) even short 10 sec negative moments contain rich structure in terms of the actions and behaviors that they comprise. [For the complete proceedings, see ED630829.]