Search

Spatial analytics receive increased attention in educational data mining. A critical issue in stop detection (i.e., the automatic extraction of timestamped and located stops in the movement of individuals) is a lack of validation of stop accuracy to represent phenomena of interest. Next to a radius that an actor does not exceed for a certain duration to establish a stop, this study presents a reproducible procedure to optimize a range parameter for K-12 classrooms where students sitting within a certain vicinity of an inferred stop are tagged as being visited. This extension is motivated by adapting parameters to infer teacher visits (i.e., on-task and off-task conversations between the teacher and one or more students) in an intelligent tutoring system classroom with a dense layout. We evaluate the accuracy of our algorithm and highlight a tradeoff between precision and recall in teacher visit detection, which favors recall. We recommend that future research adjust their parameter search based on stop detection precision thresholds. This adjustment led to better cross-validation accuracy than maximizing parameters for an average of precision and recall (F1 = 0.18 compared to 0.09). As stop sample size shrinks with higher precision cutoffs, thresholds can be informed by ensuring sufficient statistical power in offline analyses. We share avenues for future research to refine our procedure further. Detecting teacher visits may benefit from additional spatial features (e.g., teacher movement trajectory) and can facilitate studying the interplay of teacher behavior and student learning. [For the complete proceedings, see ED630829.]

Gaming the system, a behavior in which learners exploit a system's properties to make progress while avoiding learning, has frequently been shown to be associated with lower learning. However, when we applied a previously validated gaming detector across conditions in experiments with an algebra tutor, the detected gaming was not associated with learning, challenging its construct validity. Our iterative exploratory data analysis suggested that some contextual factors that varied across and within conditions might contribute to this lack of association. We present a latent variable model, "item response theory-based gaming detection" (IRT-GD), that accounts for contextual factors and estimates latent gaming tendencies as the degree of deviation from normative behaviors across contexts. Item response theory models, widely used in knowledge assessment, account for item difficulty in estimating latent student abilities: students are estimated as having higher ability when they can get harder items correct than when they only get easier items correct. Similarly, IRT-GD accounts for contextual factors in estimating latent gaming tendencies: students are estimated as having a higher gaming tendency when they game in less commonly gamed contexts than when they only game in more commonly gamed contexts. IRT-GD outperformed the original detector on three datasets in terms of the association with learning. IRT-GD also more accurately revealed intervention effects on gaming and revealed a correlation between gaming and perceived competence in math. Our approach is not only useful for others wanting to apply a gaming assessment in their context but is also generally applicable in creating robust behavioral measures. [For the full proceedings, see ED623995.]

In numerous studies, intelligent tutoring systems (ITSs) have proven effective in helping students learn mathematics. Prior work posits that their effectiveness derives from efficiently providing eventually-correct practice opportunities. Yet, there is little empirical evidence on how learning processes with ITSs compare to other forms of instruction. The current study compares problem-solving with an ITS versus solving the same problems on paper. We analyze the learning process and pre-post gain data from N = 97 middle school students practicing linear graphs in three curricular units. We find that (i) working with the ITS, students had more than twice the number of eventually-correct practice opportunities than when working on paper and (ii) omission errors on paper were associated with lower learning gains. Yet, contrary to our hypothesis, tutor practice did not yield greater learning gains, with tutor and paper comparing differently across curricular units. These findings align with tutoring allowing students to grapple with challenging steps through tutor assistance but not with eventually-correct opportunities driving learning gains. Gaming-the-system, lack of transfer to an unfamiliar test format, potentially ineffective tutor design, and learning affordances of paper can help explain this gap. This study provides first-of-its-kind quantitative evidence that ITSs yield more learning "opportunities" than equivalent paper-and-pencil practice and reveals that the relation between opportunities and learning gains emerges only when the instruction is effective.

Prior research shows that self-explanation promotes understanding by helping learners connect new knowledge with prior knowledge. However, despite ample evidence supporting the effectiveness of self-explanation, an instructional design challenge emerges in how best to scaffold self-explanation. In particular, it is an open challenge to design self- explanation support that simultaneously facilitates performance and learning outcomes. Towards this goal, we designed anticipatory diagrammatic self-explanation, a novel form of self-explanation embedded in an Intelligent Tutoring System (ITS). In our ITS, anticipatory diagrammatic self-explanation scaffolds learners by providing visual representations to help learners predict an upcoming strategic step in algebra problem solving. A classroom experiment with 108 middle-school students found that anticipatory diagrammatic self-explanation helped students learn formal algebraic strategies and significantly improve their problem-solving performance. This study contributes to understanding of how self-explanation can be scaffolded to support learning and performance.

Constructing effective and well-balanced learning groups is important for collaborative learning. Past research explored how group formation policies affect learners' behaviors and performance. With the different classroom contexts, many group formation policies work in theory, yet their feasibility is rarely investigated in authentic class sessions. In the current work, we define "feasibility" as the ratio of students being able to find available partners that satisfy a given group formation policy. Informed by user-centered research in K-12 classrooms, we simulated pairing policies on historical data from an intelligent tutoring system (ITS), a process we refer to as "SimPairing." As part of the process for designing a pairing orchestration tool, this study contributes insights into the feasibility of four dynamic pairing policies, and how the feasibility varies depending on parameters in the pairing policies or different classes. We found that on average, dynamically pairing students based on their in-the-moment wheel-spinning status can pair most struggling students, even with moderate constraints of restricted pairings. In addition, we found there is a trade-off between the required knowledge heterogeneity and policy feasibility. Furthermore, the feasibility of pairing policies can vary across different classes, suggesting a need for customization regarding pairing policies. [For the full proceedings, see ED615472.]

Many studies have shown that visual representations can enhance student understanding of STEM concepts. However, prior research suggests that visual representations alone are not necessarily effective across a broad range of students. To address this problem, we created a novel, scaffolded form of diagrammatic self-explanation in which students "explain" their problem-solving steps in the form of diagrams. We used contrasting cases to support students' sense-making between algebraic equations and diagrams in the self-explanation activity. We conducted a classroom experiment with 41 students in grades 5 and 6 to test the effectiveness of this strategy when embedded in an Intelligent Tutoring System for algebra. We found that scaffolded diagrammatic self-explanation enhanced conceptual knowledge for students who did not have prior knowledge of formal equation-solving strategies. The study is the first experimental study showing that visual representations can enhance conceptual knowledge in early algebra. [This paper was published in: "ICLS 2020 Proceedings," ISLS, 2020, pp. 35-42.]

When designing an instructional tool and using it in pedagogical activities, it is essential that designers and users understand what pedagogical affordances and constraints the tool provides to support its successful integration into targeted pedagogical activities. Toward this end, we developed "Pedagogical Affordance Analysis (PAA)." PAA involves analyzing teachers' Pedagogical Content Knowledge and/or Technological Pedagogical Content Knowledge to elicit pedagogical affordances and constraints that are specific to a given instructional goal. Information obtained through PAA can help in designing, refining, and/or evaluating instructional tools. We present a case study in which we used PAA to successfully design a visual representation for middle-school algebra. To the best of our knowledge, PAA is the only available systematic method that leverages teachers' pedagogical knowledge in identifying pedagogical affordances and constraints. PAA can be used across a wide range of existing tools and prototypes of to-be-designed tools. [This paper was published in: "ICLS 2020 Proceedings," ISLS, 2020, pp. 1561-64.]

Extensive literature in artificial intelligence in education focuses on developing automated methods for detecting cases in which students struggle to master content while working with educational software. Such cases have often been called "wheel-spinning," "unproductive persistence," or "unproductive struggle." We argue that most existing efforts rely on operationalizations and prediction targets that are misaligned to the approaches of real-world instructional systems. We illustrate facets of misalignment using Carnegie Learning's "MATHia" as a case study, raising important questions being addressed by on-going efforts and for future work. [This paper was published in: I. Bitencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Millan (Eds.), "Proceedings of the 21st International Conference on Artificial Intelligence in Education" (AIED 2020). Lecture Notes in Computer Science (LNCS, Vol. 12164 pp.92-97). Springer, Cham.]

"Wheel spinning" is the phenomenon in which a student fails to master a Knowledge Component (KC), despite significant practice. Ideally, an intelligent tutoring system would detect this phenomenon early, so that the system or a teacher could try alternative instructional strategies. Prior work has put forward several criteria for wheel spinning and has demonstrated that wheel spinning can be detected reasonably early. Yet the literature lacks systematic comparisons among the multiple wheel spinning criteria, features, and models that have been proposed, across multiple evaluation criteria (e.g., earliness, precision, and generalizability) and datasets. In our experiments, we constructed six wheel spinning detectors and compared their performance under two different wheel spinning criteria with three datasets. The results show that two prominent criteria for wheel spinning diverge substantially, and that a Random Forest model has the most consistent performance in early detection of wheel spinning across datasets and wheel spinning criteria. In addition, we found that a simple model overlooked by previous research (Logistic Regression trained on a single feature) is able to detect wheel spinning at an early stage with decent performance. This work brings us closer to unifying strands of prior work on wheel spinning (e.g., understanding how different criteria compare) and to early detection of wheel spinning in educational practice. [For the corresponding grantee submission, see ED594575. For the full proceedings, see ED599096.]

"Wheel spinning" is the phenomenon in which a student fails to master a Knowledge Component (KC), despite significant practice. Ideally, an intelligent tutoring system would detect this phenomenon early, so that the system or a teacher could try alternative instructional strategies. Prior work has put forward several criteria for wheel spinning and has demonstrated that wheel spinning can be detected reasonably early. Yet the literature lacks systematic comparisons among the multiple wheel spinning criteria, features, and models that have been proposed, across multiple evaluation criteria (e.g., earliness, precision, and generalizability) and datasets. In our experiments, we constructed six wheel spinning detectors and compared their performance under two different wheel spinning criteria with three datasets. The results show that two prominent criteria for wheel spinning diverge substantially, and that a Random Forest model has the most consistent performance in early detection of wheel spinning across datasets and wheel spinning criteria. In addition, we found that a simple model overlooked by previous research (Logistic Regression trained on a single feature) is able to detect wheel spinning at an early stage with decent performance. This work brings us closer to unifying strands of prior work on wheel spinning (e.g., understanding how different criteria compare) and to early detection of wheel spinning in educational practice. [This paper was published in: Proceedings of the 12th International Conference on Educational Data Mining Montreal, QC, Canada, July 2-5, 2019.]

Learners' choices as to whether and how to use visual representations during learning are an important yet understudied aspect of self-regulated learning. To gain insight, we developed a "choice-based" intelligent tutor in which students can choose whether and when to use diagrams to aid their problem solving in algebra. In an exploratory classroom study with 26 students, we investigated how learners choose diagrams and how their choice behaviors relate to learning outcomes. Students who "proactively" chose to use diagrams achieved higher learning outcomes than those who "reactively" used diagrams when they made incorrect attempts. This study contributes to understanding of self-regulated use of visual representations during problem solving. [This paper was published in: "ICLS2022 Proceedings," International Society of the Learning Sciences, 2022, pp. 1649-52.]

Integrating visual representations in an interactive learning activity effectively scaffolds performance and learning. However, it is unclear whether and how "sustaining" or "interleaving" visual scaffolding helps learners solve problems efficiently and learn from problem solving. We conducted a classroom study with 63 middle-school students in which we tested whether sustaining or interleaving a particular form of visual scaffolding, called anticipatory diagrammatic self-explanation in an Intelligent Tutoring System, helps students' learning and performance in the domain of early algebra. Sustaining visual scaffolding during problem solving helped students solve problems efficiently with no negative effects on learning. However, in-depth log data analyses suggest that interleaving visual scaffolding allowed students to practice important skills that may help them in later phases of algebra learning. This paper extends scientific understanding that sustaining visual scaffold does not over-scaffold student learning in the early phase of skill acquisition in algebra. [This paper was published in: "Proceedings of the 44th Annual Conference of the Cognitive Science Society," edited by J. Culbertson et al., Cognitive Science Society, 2022, pp. 1751-58.]

One pedagogical technique that promotes conceptual understanding in mathematics learners is self-explanation integrated with worked examples (e.g., Rittle-Johnson et al., 2017). In this work, we implemented self-explanations with worked examples (correct and erroneous) in a software-based Intelligent Tutoring System (ITS) for learning algebra. We developed an approach to eliciting self-explanations in which the ITS guided students to select explanations that were conceptually rich in nature. Students who used the ITS with self-explanations scored higher on a posttest that included items tapping both conceptual and procedural knowledge than did students who used a version of the ITS that included only traditional problem-solving practice. This study replicates previous findings that self-explanation and worked examples in an ITS can foster algebra learning (Booth et al., 2013). Further, this study extends prior work to show that guiding students towards conceptual explanations is beneficial. [This paper was published in: "Proceedings of the 44th Annual Conference of the Cognitive Science Society," edited by J. Culbertson et al., Cognitive Science Society, 2022, pp. 3466-72.]

Social interdependence is a key concept in CSCL research. However, investigations of students' positive and negative interdependence during collaborative activities have often relied on self-report, rather than dialogue analysis. Bringing together politeness and social interdependence theory, we assessed "dialogue indicators" of positive and negative interdependence from behavioral data (namely, face-saving and face-threatening dialogue moves) and compared the results with those of self-report scales. We analyzed a data set of 30 elementary students learning fractions with an intelligent tutoring system (ITS). Our initial analyses focus on the link between use of language that is face-saving (e.g., marking identity with statements such as "we are great") or face-threatening (e.g., insulting), and students' preferences to collaborate and compete. We found only non-significant correlations between these two broad categories, but found significant correlations between dialogue indicators, such as the use of identity markers and joking, that suggest directions for subsequent studies.

The gold standard for identifying more effective pedagogical approaches is to perform an experiment. Unfortunately, frequently a hypothesized alternate way of teaching does not yield an improved effect. Given the expense and logistics of each experiment, and the enormous space of potential ways to improve teaching, it would be highly preferable if it were possible to estimate in advance of running a study whether an alternative teaching strategy would improve learning. This is true even in learning at scale situations, since even if it is logistically easier to recruit a large number of subjects, it remains a high stakes environment because the experiment is impacting many real students. For certain classes of alternate teaching approaches, such as new ways to sequence existing material, it is possible to build student models that can be used as simulators to estimate the performance of learners under new proposed teaching methods. However, existing methods for doing so can overestimate the performance of new teaching methods. We instead propose the Robust Evaluation Matrix (REM) method which explicitly considers model mismatch between the student model used to derive the teaching strategy and that used as a simulator to evaluate the teaching strategy effectiveness. We then present two case studies from a fractions intelligent tutoring system and from a concept learning task from prior work that show how REM could be used both to detect when a new instructional policy may not be effective on actual students and to detect when it may be effective in improving student learning. [This paper was published in: "L@S '17 Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale" Cambridge, Massachusetts, p3-12. ACM New York, NY. 2017.]

Collaborative and individual learning are both frequently used in classrooms to support learning. However, little research has investigated the benefits of combining individual and collaborative learning, as compared to learning only individually or only collaboratively. With our study, we address this research gap. We compared a combined condition to individual-only or collaborative-only learning conditions using intelligent tutoring systems for fractions. The study was conducted with 382 4th and 5th grade students. Students across all three conditions had significant learning gains. However, the combined condition had higher learning gains than the individual or collaborative condition. This difference was more pronounced for 4th grade students than for 5th grade students. In addition, we found that students in the combined condition expressed higher situational interest in the activity compared to those working individually and the same as students working only collaboratively. Through a combination, we may support better student learning.