Search

Background: Teachers often rely on the use of open‐ended questions to assess students' conceptual understanding of assigned content. Particularly in the context of mathematics; teachers use these types of questions to gain insight into the processes and strategies adopted by students in solving mathematical problems beyond what is possible through more close‐ended problem types. While these types of problems are valuable to teachers, the variation in student responses to these questions makes it difficult, and time‐consuming, to evaluate and provide directed feedback. It is a well‐studied concept that feedback, both in terms of a numeric score but more importantly in the form of teacher‐authored comments, can help guide students as to how to improve, leading to increased learning. It is for this reason that teachers need better support not only for assessing students' work but also in providing meaningful and directed feedback to students. Objectives: In this paper, we seek to develop, evaluate, and examine machine learning models that support automated open response assessment and feedback. Methods: We build upon the prior research in the automatic assessment of student responses to open‐ended problems and introduce a novel approach that leverages student log data combined with machine learning and natural language processing methods. Utilizing sentence‐level semantic representations of student responses to open‐ended questions, we propose a collaborative filtering‐based approach to both predict student scores as well as recommend appropriate feedback messages for teachers to send to their students. Results and Conclusion: We find that our method outperforms previously published benchmarks across three different metrics for the task of predicting student performance. Through an error analysis, we identify several areas where future works may be able to improve upon our approach. Lay Description: What is already known about this topic: Open‐ended questions are used by teachers in the domain of mathematics to assess their students understanding but automated support for these types of questions are limited in online learning platforms. Recent advancements in areas of machine learning and natural language processing have led to promising results in a range of domains and applications. What this paper adds: Emulating how teachers identify similar student answers can be used to build tools that support teachers in assessing and providing feedback to student open‐ended work. Implications for practice: Developing better automated supports can increase the amount of direct feedback students receive to guide their learning. [ABSTRACT FROM AUTHOR]

Background: Computer‐based assessment allows for the monitoring of reader behaviour. The identification of patterns in this behaviour can provide insights that may be useful in informing educational interventions. Objectives: Our study aims to explore what different patterns of reading activity exist, and investigates their interpretation and consistency across different task sets (units), countries, and languages. Three patterns were expected: on‐task, exploring and disengaged. Methods: Using log data from the PISA 2012 digital reading assessment (9226 students from seven countries), we conducted hierarchical cluster analyses with typical process indicators of digital reading assessments. We identified different patterns and explored whether they remained consistent across different units. To validate the interpretation of the identified patterns, we examined their relationship to performance and student characteristics (gender, socio‐economic status, print reading skills). Results and Conclusions: The results indicate a small number of transnational clusters, with unit‐specific differences. Cluster interpretation is supported by associations with student characteristics—for example, students with low print reading skills were more likely to show a disengaged pattern than proficient readers. Exploring behaviour tended to be exhibited only once across the three units: It occurred in the first unit for proficient readers and in later units for less skilled readers. Major Takeaways: Behavioural patterns can be identified in digital reading tasks that may prove useful for educational monitoring and intervention. Although task situations are designed to evoke certain behaviours, the interpretation of observed behavioural patterns requires validation based on task requirements, assessment context and relationships to other available information. Lay Description: What is already known about the subject matter?: Students differ in how they read, comprehend and use digital information.Indicators from process data provide insight about how students engage with digital reading tasks. What does this paper add to the subject matter?: Based on multiple process indicators, a small number of clusters indicating different behavioural activity can be distinguished.These clusters can be described as on‐task, passive, hasty, exploring, disengaged, persistent and lost interest.A meaningful interpretation of the clusters must consider the requirements of the underlying tasks. Implications for practice and/or policy: Knowledge of how students engage with digital resources may provide useful feedback for teachers to guide students' learning or intervene when they struggle.Educational monitoring: The high comparability of country‐specific results suggests an invariant set of solution strategies in the digital reading assessment. [ABSTRACT FROM AUTHOR]

Background: Providing students with worked out problem solutions is a beneficial instructional technique in STEM disciplines, and studying examples that have been worked out incorrectly may be especially helpful for reducing misconceptions in students with low prior content knowledge. However, past results are inconclusive and the effects of incorrect worked examples alone or in combination with correct examples remains unclear. Objectives: We aim to address whether studying incorrect examples alone or in combination with correct examples can support the reduction of students' fraction misconceptions, operationalized as errors made with high confidence. Methods: After incorrectly solving a sampling problem, 130 students in 4th through 11th grade in the U.S. were randomly assigned to a condition in an online problem set focused on fraction equivalence. Students studied either single‐type worked examples (i.e., correct or incorrect; n = 49) or combination‐type worked examples (correct and incorrect; n = 41) or engaged in a problem‐solving control (n = 50). Results: Studying a combination of correct and incorrect worked examples was as effective as the problem‐solving control with feedback at improving fraction equivalence knowledge and reducing the rate of high‐confidence errors. Students in both the combination condition and the problem‐solving with feedback condition outperformed those who studied either correct or incorrect worked examples alone. Conclusions: Results support the inclusion of a combination of correct and incorrect worked examples when teaching students with low prior content knowledge. Studying a combination of example types within an online tutor helps to reduce misconceptions about fractions, a topic students commonly struggle with. A problem‐solving task with corrective feedback worked equally well. Lay Description: What is already known about this topic: Studying worked examples of mathematical solutions improves student learning.Incorrect examples, either alone or in combination with correct examples, are particularly effective for improving learning for students with low prior knowledge in the target content.Comparing correct and incorrect examples is also effective at improving problem‐solving but this effect may be specific to those with high prior knowledge.Still unknown are the impacts of these different combinations of examples on misconceptions in particular. What this paper adds: We show that studying a combination of correct and incorrect examples reduces math misconceptions.Problem‐solving with corrective feedback within an online tutor also has this effect. Implications for practice and/or policy: Using a combination of example types will help reduce students' misconceptions about fractions. [ABSTRACT FROM AUTHOR]

Background: Valuable safeguards against fast‐spreading conjectures about learning in times of fear and uncertainty are evidence‐based approaches to the assessment of the impact of sudden and unforeseen disruptions on learning practices. The present research focused on physics learning in such times because conceptual and computational literacy in physics is critical to the development of a scientifically and technologically literate society. Objectives: The present research aimed (a) to offer an objective assessment of whether performance differences in a physics course of the general education curriculum existed between the face‐to‐face medium (familiar mode of instruction) and the online medium (unfamiliar mode of instruction) for both male and female students, and then (b) to develop a response to the evidence collected to ensure a quality education for all parties involved. The research intended to fill two critical gaps in the extant literature: mixed findings concerning students' performance in the face‐to‐face and online mediums as well as scarce coverage of specific domains of knowledge that are critical to STEM learners. Methods: Students' performance was examined as a function of the type of assessment (formative and summative), instructional mode (online and face‐to‐face), and gender. An understudied student population of STEM students of Middle Eastern descent without prior formal exposure to online instruction was targeted. Results and Conclusions: In both formative and summative assessments, male students performed better online than face‐to‐face, whereas the performance of female students was either higher online or equivalent between instructional mediums. The evidence collected suggested that consideration be given to remedies that foster academic success in the face‐to‐face instructional medium, particularly for male students. Takeaways: An evidence‐based approach to learning dismantled emotion‐driven expectations regarding the impact of the online medium on physics learning, and encouraged new perspectives about instruction. Lay Description: What is already known?: Ongoing debates on the impact of the pandemic on students' learning shape decisions regarding future reliance on the online medium for instruction. What this paper adds: The extant literature is mixed concerning performance in the face‐to‐face and online mediums, including particular domains of knowledge critical to STEM learning.This study focused on physics learning in an understudied population of STEM students who were unaccustomed to online instruction.Performance was higher online than face‐to‐face, but gender differences emerged. Implications for practice: Lessons learned focused on the flexibility of online learning and on the need to develop materials to enhance learning in males enrolled in face‐to‐face or hybrid/blended classes. [ABSTRACT FROM AUTHOR]

Background: The sudden growth in online instruction due to COVID‐19 restrictions has given renewed urgency to questions about remote learning that have remained unresolved. Web‐based assessment software provides instructors an array of options for varying testing parameters, but the pedagogical impacts of some of these variations has yet to be examined. Objectives: The current study examines whether question order affects student performance on online college math assessments. Drawing on the literature of testing and math anxiety, we hypothesized that difficult questions near the beginning of an assessment would have a destructive effect on student confidence, which would in turn have a deleterious effect on their performance. Methods: We employed an observational 'big data' methodology, analysing 23,468 final exams completed by students in 10 different courses over eight semesters at a Math Emporium in a large technical university in the eastern United States. Students were freshmen and sophomores enrolled in non‐engineering math courses. We regressed the final score on the difficulty level of the first and second questions, controlling for several other factors. Results and Conclusions: We found that several factors—day of the week, amount of time before the deadline, number of minutes spent on the exam—have more of an impact on score than question order. This pattern was consistent across sexes. Takeaways: Our findings contradict some previous studies, which have found that difficult early questions degrade student performance, and that this affect is more pronounced in females. This work enriches our understanding of how students respond to online assessment. Lay Description: What is already known about this topic: Previous studies have had mixed results about the effect of question difficulty ordering on student performance.Furthermore, few studies have examined this issue for computer assessments. What this paper adds: We employ a novel methodology with a uniquely large data set.Our findings indicate that difficulty ordering has no effect on student performance. Implications for practice: Math instructors can use the random ordering feature of computer assessment generating tools because question order does not affect performance. [ABSTRACT FROM AUTHOR]

Although the Internet is widely used by students in both formal and informal environments, little is known about how and where youth spend their time online. Using Internet search and Web analytics data, this study discovered a large-scale phenomenon associated with the poor performance of elementary school students in the USA that has been overlooked by educational researchers. This study found that approximately 10 million Internet users in the USA, many of whom are presumably youth, spend about 89 million hours in a year on a popular math game site that targets children. The number of game site users is equivalent to half of the K-5 Internet population in the USA. However, there is little evidence that the math games on the website meet the criteria for effective instruction as described in the literature. This study found a significant negative correlation between search volumes for the game site in the 50 states in the USA and 4th grade students' performance in mathematics and reading. Moreover, Internet users in the states with greater numbers of low-income families and fewer college graduates were more likely to search for the game site. The implications of these findings are discussed. [ABSTRACT FROM AUTHOR]