Search

Background: Teachers often rely on the use of open‐ended questions to assess students' conceptual understanding of assigned content. Particularly in the context of mathematics; teachers use these types of questions to gain insight into the processes and strategies adopted by students in solving mathematical problems beyond what is possible through more close‐ended problem types. While these types of problems are valuable to teachers, the variation in student responses to these questions makes it difficult, and time‐consuming, to evaluate and provide directed feedback. It is a well‐studied concept that feedback, both in terms of a numeric score but more importantly in the form of teacher‐authored comments, can help guide students as to how to improve, leading to increased learning. It is for this reason that teachers need better support not only for assessing students' work but also in providing meaningful and directed feedback to students. Objectives: In this paper, we seek to develop, evaluate, and examine machine learning models that support automated open response assessment and feedback. Methods: We build upon the prior research in the automatic assessment of student responses to open‐ended problems and introduce a novel approach that leverages student log data combined with machine learning and natural language processing methods. Utilizing sentence‐level semantic representations of student responses to open‐ended questions, we propose a collaborative filtering‐based approach to both predict student scores as well as recommend appropriate feedback messages for teachers to send to their students. Results and Conclusion: We find that our method outperforms previously published benchmarks across three different metrics for the task of predicting student performance. Through an error analysis, we identify several areas where future works may be able to improve upon our approach. Lay Description: What is already known about this topic: Open‐ended questions are used by teachers in the domain of mathematics to assess their students understanding but automated support for these types of questions are limited in online learning platforms. Recent advancements in areas of machine learning and natural language processing have led to promising results in a range of domains and applications. What this paper adds: Emulating how teachers identify similar student answers can be used to build tools that support teachers in assessing and providing feedback to student open‐ended work. Implications for practice: Developing better automated supports can increase the amount of direct feedback students receive to guide their learning. [ABSTRACT FROM AUTHOR]

Background: Students' tendencies to seek feedback are associated with improved learning. Yet, how soon this association becomes robust enough to make predictions about learning is not fully understood. Such knowledge has strong implications for early identification of students at‐risk for underachievement via digital learning platforms. Objectives: We sought to understand how early in the academic year students' end‐of‐year learning outcomes could be predicted by their performance and feedback‐seeking behaviours within a digital learning platform. We analysed data collected at different time points in the academic year and across different cohorts of students within the context of high school advanced placement (AP) Statistics courses. Methods: High school students enrolled in AP Statistics spanning three academic years between 2017 and 2020 (N = 726; Mage = 16.72 years) completed 3 or 4 homework assignments, each 2 and 3 months apart. Results and conclusions: Across the three cohorts, and even as early as the first assignment, a model consisting of demographic variables (gender, race/ethnicity, parental education), assignment performance, and interaction with the digital score report explained significant variation in students' final course grades (R2 = 0.314–0.412) and AP exam scores (κ = 0.583–0.689). Students' assignment performance was positively associated with end‐of‐year learning outcomes. Students who more frequently checked their digital score reports tended to receive better learning outcomes, though not consistently across cohorts. Implications: These findings further an understanding of how students' early performance and feedback‐seeking behaviours within a digital learning platform predict end‐of‐year learning outcomes. Lay Description: What is already known about this topic: Students' early academic performance predicts end‐of‐year success.Students' self‐regulated learning behaviours are another predictor of academic success.Clickstream data have been used as an indicator of the application of self‐regulated learning strategies, particularly behaviours associated with seeking feedback.Indicators of feedback‐seeking behaviours reflected via clickstream data have been shown to predict student learning.Yet, how soon associations between early performance and such indicators of self‐regulated learning as feedback‐seeking are robust across different groups of learners, different learning circumstances, and different types of learning outcomes has yet to be more fully understood. What this paper adds: We investigated whether performance and feedback‐seeking behaviours predict end‐of‐year learning outcomes at different points during the academic year across multiple cohorts of learners (2017–2018, 2018–2019, and 2019–2020) and using two different learning outcomes (class grades, AP exam scores).One cohort was affected by COVID‐19 during the 2019–2020 academic year, allowing us to examine whether differences emerge amidst unprecedented circumstances.We used assignment scores and online feedback‐seeking to predict end‐of‐year learning outcomes.Assignment scores were positively associated with end‐of‐year grades.First assignment performance predicted AP exam scores.Use of feedback (an indication of self‐regulated learning behaviours) tended to predict learning outcomes, though not consistently across cohorts. Implications for practice and/or policy: The findings from this study could improve understanding of how digital learning platform measurements predict end‐of‐year learning outcomes though also highlight the importance of context differences.Such findings hold implications for early identification of at‐risk students and appreciation for differences between learners and contexts.These findings also contribute to a growing understanding of the impact of the COVID‐19 pandemic on student engagement and learning. [ABSTRACT FROM AUTHOR]

The response process of problem‐solving items contains rich information about respondents' behaviours and cognitive process in the digital tasks, while the information extraction is a big challenge. The aim of the study is to use a data‐driven approach to explore the latent states and state transitions underlying problem‐solving process to reflect test‐takers' behavioural patterns, and to investigate how these states and state transitions could be associated with test‐takers' performance. We employed the Hidden Markov Modelling approach to identify test takers' hidden states during the problem‐solving process and compared the frequency of states and/or state transitions between different performance groups. We conducted comparable studies in two problem‐solving items with a focus on the US sample that was collected in PIAAC 2012, and examined the correlation between those frequencies from two items. Latent states and transitions between them underlying the problem‐solving process were identified and found significantly different by performance groups. The groups with correct responses in both items were found more engaged in tasks and more often to use efficient tools to solve problems, while the group with incorrect responses was found more likely to use shorter action sequences and exhibit hesitative behaviours. Consistent behavioural patterns were identified across items. This study demonstrates the value of data‐driven based HMM approach to better understand respondents' behavioural patterns and cognitive transmissions underneath the observable action sequences in complex problem‐solving tasks. Lay Description: What is already known about this topic?: Process data of interactive problem‐solving tasks contains rich information about test‐takers.To extract underlying information from process data is a big challenge. ​ What this paper adds?: The hidden Markov model helps identify latent states underlying problem‐solving process.Different performance groups displayed different behaviour characteristics.A consistent behavioural pattern was observed across problem‐solving items. ​ Implications for practitioners: The use of HMM holds promise in better visualize and understand behaviour patterns in response process.HMM is potential in understanding cognitive transmissions underneath the observable action sequences. ​ [ABSTRACT FROM AUTHOR]