Exploring students' cognitive and affective states during problem solving through multimodal data: Lessons learned from a programming activity.

Background: Problem‐solving is a multidimensional and dynamic process that requires and interlinks cognitive, metacognitive, and affective dimensions of learning. However, current approaches practiced in computing education research (CER) are not sufficient to capture information beyond the basic programming process data (i.e., IDE‐log data). Therefore, how cognition and affect intertwine and unfold over time in programming problem‐solving activities are rarely investigated. Objectives: In this study, we examined how the theory‐informed measures from multimodal data that we have selected as proxies for cognitive and affective dimensions of learning, are associated with student performance, and in comparison, to prior‐knowledge. Methods: A high‐frequency temporal data was collected with a camera, an electroencephalogram, and an eye‐tracker from 40 computer science students (bachelor and master studies) in the context of a code‐debugging activity. To study the cognitive processes associated with learning we focused on cognitive load theory (CLT) and the human information processing model. In addition, we complemented CLT with the model of affective dynamics in learning to avoid the machine reductionism perspective. Results: Our findings demonstrated that attention, convergent thinking, and frustration were positively correlated with students' successful code‐debugging (i.e., performance), and frequently manifested by high performing participants. Cognitive load, memory load, and boredom were negatively correlated with students' performance, and typically manifested by low performing participants. Implications: Extending the context of analysis in reference to student cognitive processes and affective states, affords educators not just to identify lower performers, but also to understand the potential reasons behind their performance, making our method an important contribution in the confluence of CER and the learning technology communities. In addition, the insights extracted from our analyses allow us to discuss potential avenues for improving learning design and the design of interactive learning systems to support the development of students' problem‐solving skills. Lay Description: What is currently known about the subject matter: ● Problem‐solving interlinks cognitive, affective and metacognitive dimensions of learning. ● In digital settings instructors struggle to distinguish cognitive‐affective states of students. ● Learning technologies require interventions based on cognitive‐affective states of students. What the paper adds: ● When and what cognitive‐affective states are triggered during solving a programming problem. ● What and how measures as proxies for cognitive‐affective states are associated with performance. ● Emphasizes the benefits of augmenting IDE‐log data with sensor data in the context of learning design. Implications of study findings for practitioners: ● Methodology guidelines to identify and model variability in humans' natural behaviours. ● Importance of content and instructions personalization for student engagement and learning. ● Identification of multimodal data proxies for flexible and adaptive multimodal interfaces. [ABSTRACT FROM AUTHOR]