Search

This paper develops a method for the construction and evaluation of cognitive models to support students in their problem-solving skills during robotics in school, aiming to build a basis for an implementation of a tutoring system in the future. Two Wizard-of-Oz studies were conducted, one in the classroom and one in the lab. Based on the cognitive model, the human wizards gave support to 20 students working in pairs. The studies were video recorded and a qualitative analysis was conducted. This qualitative research approach is described in detail. The evaluation of the studies showed that students reacted mostly positively to the wizards. We also uncovered ways in which students' problem-solving skills could be improved. Based on the evaluation and observations of the Wizard-of-Oz studies, the paper proposes a design for a future robotics skills tutoring system.

Despite considerable advances in knowledge tracing algorithms, educational technologies that use this technology typically continue to use older algorithms, such as Bayesian Knowledge Tracing. One key reason for this is that contemporary knowledge tracing algorithms primarily infer next-problem correctness in the learning system, but do not attempt to infer the knowledge the student can carry out of the system, information more useful for teachers. The ability of knowledge tracing algorithms to predict problem correctness using data from intelligent tutoring systems has been extensively researched, but data from outcomes other than next-problem correctness have received less attention. In addition, there has been limited use of knowledge tracing algorithms in games, because algorithms that do attempt to infer knowledge from answer correctness are often too simple to capture the more complex evidence of learning within games. In this study, data from a digital learning game, (anonymized), was used to compare ten knowledge tracing algorithms' ability to predict students' knowledge carried outside the learning system--measured here by posttest scores--given their game activity. All Opportunities Averaged (AOA), a method proposed by Authors (2020) was used to convert correctness predictions to knowledge estimates, which were also compared to the built-in estimates from algorithms that produced them. Although statistical testing was not feasible for these data, three algorithms tended to perform better than the others: Dynamic Key-Value Memory Networks, Logistic Knowledge Tracing, and a multivariate version of Elo. Algorithms' built-in estimates of student ability underperformed estimates produced by AOA, suggesting that some algorithms may be better at estimating performance than ability. Theoretical and methodological challenges related to comparing knowledge estimates with hypothesis testing are also discussed.

Involving stakeholders throughout the creation of new educational technologies can help ensure their usefulness and usability in real-world contexts. However, given the complexity of learning analytics (LA) systems, it can be challenging to meaningfully involve non-technical stakeholders throughout their design and development. This article presents a detailed case study of the iterative co-design of Lumilo, a wearable, real-time learning analytics tool for teachers working in AI-enhanced K-12 classrooms. In the process, we argue that the co-design of LA systems requires "new kinds of prototyping methods." We introduce one of our own prototyping methods, REs, to address unique challenges of co-prototyping data-driven algorithmic systems such as LA tools. This work presents the first end-to-end demonstration in the literature of how non-technical stakeholders can participate throughout the whole design process for a complex LA system -- from early generative phases to the selection and tuning of analytics to evaluation in real-world contexts. We conclude with a summary of methodological recommendations for future LA co-design efforts.

Digital learning games are designed to foster both student learning and enjoyment. Given this goal, an interesting research topic is whether game mechanics that promote learning and those that promote enjoyment have different effects on students' experience and learning performance. We explored these questions in "Decimal Point," a digital learning game that teaches decimal numbers and operations to 5th and 6th graders, through a classroom study with 159 students and two versions of the game. One version encouraged playing and learning through an "open learner model" (OLM, N = 55), while one encouraged playing for enjoyment through an analogous "open enjoyment model" (OEM, N = 54). We compared these versions to a control version that is neutral with respect to learning and enjoyment (N = 50). While students learned in all three conditions, our results indicated no significant condition differences in learning outcomes, enjoyment, or engagement. However, the learning-oriented group engaged more in re-practicing, while the enjoyment-oriented group demonstrated more exploration of different mini-games. Further analyses of students' interactions with the open learner and enjoyment models revealed that students who followed the learner model demonstrated better in-game learning and test performance, while following the enjoyment model did not impact learning outcomes. These findings indicate that emphasizing learning or enjoyment can lead to distinctive game play behaviors, and that open learner models can be helpful in a learning game context. In turn, our analyses have led to preliminary ideas about how to use AI to provide recommendations that are more aligned with students' dynamic learning and enjoyment states and preferences.

There is an established gender gap in middle school math education, where female students report higher anxiety and lower engagement, which negatively impact their performance and even long-term career choices. This work investigates the role of digital learning games in addressing this issue by studying Decimal Point, a math game that teaches decimal numbers and operations to 5th and 6th graders. Through data from four published studies of Decimal Point, involving 624 students in total, the authors identified a consistent gender difference that was replicated across all studies -- male students tended to do better at pretest, while female students tended to learn more from the game. In addition, female students were more careful in answering self-explanation questions, which significantly mediated the relationship between gender and learning gains in two out of four studies. These findings show that learning games can be an effective tool for bridging the gender gap in middle school math education, which in turn contributes to the development of more personalized and inclusive learning platforms.

Jim Greer and his colleagues argued that student modelling is essential to provide adaptive instruction in tutoring systems and showed that effective modelling is possible, despite being enormously challenging. Student modelling plays a prominent role in many intelligent tutoring systems (ITSs) that address problem-solving domains. However, considerably less attention has been paid to using a student model to personalize instruction in tutorial dialogue systems (TDSs)--ITSs that engage students in natural-language, conceptual discussions. This paper describes Rimac, a TDS that tightly couples student modelling with tutorial dialogues about conceptual physics. Rimac is distinct from other TDSs insofar as it dynamically builds a persistent student model that guides reactive and proactive decision making in order to provide adaptive instruction. An initial pilot study set in high school physics classrooms compared a control version of Rimac without a student model with an experimental version that implemented a "poor man's student model"; that is, the model was initialized using students' pretest scores but not updated further. Both low and high prior knowledge students showed significant pretest to posttest learning gains. However, high prior knowledge students who used the experimental version of Rimac learned more efficiently than high prior knowledge students who used the control version. Specifically, high prior knowledge students who used the student model driven tutor took less time to complete the intervention but learned a similar amount as students who used the control version. A subsequent study found that both high and low prior knowledge students learned more efficiently from a version of the tutor that dynamically updates its student model during dialogues than from a control version that included the static "poor man's student model." We discuss future work needed to improve the performance of Rimac's student model and to integrate TDSs in the classroom.

Background: Mindfulness practices enhance executive function skills and academic achievement, spurring interest in integrating mindfulness interventions into education. Embedding mindfulness practice into a digital math game may provide a low‐cost, scalable way to induce mindfulness and boost game‐based learning, yet this approach remains unexplored. Objectives: We investigated the learning benefits of integrating mindfulness exercises in a digital math learning game and examined how students' trait mindfulness might moderate the outcomes. Methods: Two classroom studies were conducted with 404 5th and 6th grade students from six public schools in the U.S. (nStudy 1 = 227, nStudy 2 = 177). The two randomized controlled experiments assigned students to one of the three conditions: passive control (playing the digital learning game Decimal Point), story‐enriched active control, or mindfulness‐enriched condition. Trait mindfulness, learning gains, and in‐game problem‐solving (including problem‐solving duration, error count and correctness after reminder) were assessed. Study 2 included a manipulation check to better understand the effects of the mindfulness intervention. Results: Findings showed no significant differences in learning gains, problem‐solving duration or error count among the conditions. Students' trait mindfulness did not moderate these outcomes. Mindfulness reminders in the mindfulness‐enriched game led to more correct answers after errors than jokes in the story‐enriched game. Study 2 revealed that we failed to induce higher state mindfulness through the mindfulness inductions. Conclusions: Mindfulness prompts could be especially beneficial for students experiencing frustration during gameplay, warranting more exploration for digital game‐based instruction. We highlight barriers and future directions for fostering mindfulness through computer‐based instruction in classrooms. Lay Description: What is currently known about this topic?: Researchers focus on digital games as productive learning environments for math, with potential for higher learning gains compared with traditional methods. However, results are mixed, and not all games lead to improved math outcomes.Executive function (EF) skills are crucial for math learning, and mindfulness‐based interventions show promise in enhancing EF skills in school‐aged students.Embedding mindfulness practice into a digital math game may provide a low‐cost, scalable way to induce mindfulness and, in turn, boost EF skills and game‐based learning, yet this approach remains unexplored What does this paper add?: This paper presents the results of two randomized control trials investigating the feasibility and benefits of incorporating mindfulness exercises into a digital math game designed for middle school students. Benefits are compared to those of a story‐enriched and regular version of the same digital math learning game.The paper also explores variations in the effects of the mindfulness‐enriched game based on students' initial trait mindfulness levels.We observe that listening to mindfulness inductions at the beginning of game sessions do not induce mindfulness, and therefore does not benefit math learning.We find that mindfulness prompts received after recurrent errors can be beneficial for students' problem solving. Implications of study findings for practitioners: Our study provides important information on how digital learning game designers should attempt to induce mindfulness in a digital game to promote learning.Digital learning game designers should consider incorporating mindfulness exercises into their games strategically. Presenting mindfulness inductions in audio format at the beginning of game sessions may not induce mindfulness or benefit math learning. Instead, designers should focus on integrating mindfulness prompts at moments when students encounter frustration within the learning game.Beyond embedding audio mindfulness exercises in the game, learning designers should also consider alternative technological and game features to induce mindfulness within a learning game. [ABSTRACT FROM AUTHOR]

Excitement about learning from computer-based games has been papable in recent years and has led to the development of many educational games. However, there are relatively few sound empirical studies in the scientific literature that have shown the benefits of learning mathematics from games as opposed to more traditional approaches. The empirical study reported in this paper provides evidence that a mathematics educational game can provide superior learning opportunities, as well as be more engaging. In a study involving 153 students from two middle schools, 70 students learned about decimals from playing an educational game--Decimal Point--whereas 83 students learned the same content by a more conventional, computer-based approach. The game led to significantly better gain scores in solving decimal problems, on both an immediate (d = 0.43) and delayed (d = 0.37) posttest and was rated as significantly more enjoyable (d = 0.95). Low prior knowledge students especially benefitted from the game. This paper also summarizes the game's design characteristics.

Help seeking is an important process in self-regulated learning (SRL). It may influence learning with intelligent tutoring systems (ITSs), because many ITSs provide help, often at the student's request. The Help Tutor was a tutor agent that gave in-context, real-time feedback on students' help-seeking behavior, as they were learning with an ITS. Key goals were to help students become better self-regulated learners and help them achieve better domain-level learning outcomes. In a classroom study, feedback on help seeking helped students to use on-demand help more deliberately, even after the feedback was no longer given, but not to achieve better learning outcomes. The work made a number of contributions, including the creation of a knowledge-engineered, rule-based, executable model of help seeking that can drive tutoring. We review these contributions from a contemporary perspective, with a theoretical analysis, a review of recent empirical literature on help seeking with ITSs, and methodological suggestions. Although we do not view on-demand, principle-based help during tutored problem solving as being as important as we once did, we still view it as helpful under certain circumstances, and recommend that it be included in ITSs. We view the goal of helping students become better self-regulated learners as one of the grand challenges in ITSs research today.

In 2009, we reported on a new Intelligent Tutoring Systems (ITS) technology, example-tracing tutors, that can be built without programming using the Cognitive Tutor Authoring Tools (CTAT). Creating example-tracing tutors was shown to be 4-8 times as cost-effective as estimates for ITS development from the literature. Since 2009, CTAT and its associated learning management system, the "Tutorshop," have been extended and have been used for both research and real-world instruction. As evidence that example-tracing tutors are an effective and mature ITS paradigm, CTAT-built tutors have been used by approximately 44,000 students and account for 40% of the data sets in "DataShop," a large open repository for educational technology data sets. We review 18 example-tracing tutors built since 2009, which have been shown to be effective in helping students learn in real educational settings, often with large pre/post effect sizes. These tutors support a variety of pedagogical approaches, beyond step-based problem solving, including collaborative learning, educational games, and guided invention activities. CTAT and other ITS authoring tools illustrate that non-programmer approaches to building ITS are viable and useful and will likely play a key role in making ITS widespread.

Erroneous examples--step-by-step problem solutions with one or more errors for students to find and fix--hold great potential to help students learn. In this study, which is a replication of a prior study (Adams et al. 2014), but with a much larger population (390 vs. 208), middle school students learned about decimals either by working with interactive, web-based erroneous examples or with more traditional supported problems to solve. The erroneous examples group was interactively prompted to find, explain, and fix errors in decimal problems, while the problem-solving group was prompted to solve the same decimal problems and explain their solutions. Both groups were given correctness feedback on their work by the web-based program. Although the two groups did not differ on an immediate post-test, the erroneous examples group performed significantly better on a delayed test, given a week after the initial post-test (d?=?0.33, for gain scores), replicating the pattern of the prior study. Interestingly, the problem solving group reported liking the intervention more than the erroneous examples group (d?=?0.21 for liking rating in a questionnaire) and found the user interface easier to interact with (d?=?0.37), suggesting that what students like does not always lead to the best learning outcomes. This result is consistent with that of desirable "difficulty" studies, in which a more cognitively challenging learning task results in deeper and longer-lasting learning.

During the past two decades a variety of approaches to support argumentation learning in computer-based learning environments have been investigated. We present an approach that combines argumentation diagramming and collaboration scripts, two methods successfully used in the past individually. The rationale for combining the methods is to capitalize on their complementary strengths: Argument diagramming has been shown to help students construct, reconstruct, and reflect on arguments. However, while diagrams can serve as valuable resources, or even guides, during conversations, they do not provide explicit support for the discussion itself. Collaboration scripts, on the other hand, can provide direct support for the discussion, e.g., through sentence openers that encourage high quality discussion moves. Yet, students often struggle to comply with the rules of a script, as evidenced by both the misuse and nonuse of sentence openers. To try to benefit from the advantages of both of these instructional techniques, while minimizing their disadvantages, we combined and experimented with them within a single instructional environment. In particular, we designed a collaboration script that guides student dyads through a process of analyzing, interrelating and evaluating opposing positions on a contentious topic with a goal to jointly generate a well-reasoned conclusion. We compare a baseline version of the script, one that only involves argument diagramming, with an enhanced version that employs an additional peer critique script, implemented with sentence openers, in which student pairs were assigned the roles of a proponent and a constructive critic. The enhanced version of the script led to positive effects: student discussions contained a higher number of elaborative moves and students assessed their argumentation learning more positively.

The present research investigated whether immediate metacognitive feedback on students' help-seeking errors can help students acquire better help-seeking skills. The Help Tutor, an intelligent tutor agent for help seeking, was integrated into a commercial tutoring system for geometry, the Geometry Cognitive Tutor. Study 1, with 58 students, found that the real-time assessment of students' help-seeking behavior correlated with other independent measures of help seeking, and that the Help Tutor improved students' help-seeking behavior while learning Geometry with the Geometry Cognitive Tutor. Study 2, with 67 students, evaluated more elaborated support that included, in addition to the Help Tutor, also help-seeking instruction and support for self-assessment. The study replicated the effect found in Study 1. It was also found that the improved help-seeking skills transferred to learning new domain-level content during the month following the intervention, while the help-seeking support was no longer in effect. Implications for metacognitive tutoring are discussed. (Contains 3 tables and 7 figures.)

Should an intelligent software tutor be polite, in an effort to motivate and cajole students to learn, or should it use more direct language? If it should be polite, under what conditions? In a series of studies in different contexts (e.g., lab versus classroom) with a variety of students (e.g., low prior knowledge versus high prior knowledge), the "politeness effect" was investigated in the context of web-based intelligent tutoring systems, software that runs on the Internet and employs artificial intelligence and learning science techniques to help students learn. The goal was to pinpoint the appropriate conditions for having the web-based tutors provide polite feedback and hints (e.g., "Let's convert the units of the first item") versus direct feedback and hints (e.g., "Convert the units of the first item now"). In the study presented in this paper, 132 high school students in a classroom setting, grouped as low and high prior knowledge learners according to a pre-intervention knowledge questionnaire, did not benefit more from polite feedback and hints than direct feedback and hints on either an immediate or delayed posttest, both of which contained near transfer and conceptual test items. Of particular interest and contrary to an earlier lab study, low prior knowledge students did not benefit more from using the polite version of a tutor. On the other hand, a politeness effect was observed for the students who made the most errors during the intervention, a different proxy for low prior knowledge, hinting that even in a classroom setting, politeness may be beneficial for more needy students. This article presents and discusses these results, as well as discussing the politeness effect more generally, its theoretical underpinnings, and future directions. (Contains 6 tables and 2 figures.)

This paper reports on an aspect of the EC funded Argunaut project which researched and developed awareness tools for moderators of online dialogues. In this study we report on an investigation into the nature of creative thinking in online dialogues and whether or not this creative thinking can be coded for and recognized automatically such that moderators can be alerted when creative thinking is occurring or when it has not occurred after a period of time. We outline a dialogic theory of creativity, as the emergence of new perspectives from the interplay of voices, and the testing of this theory using a range of methods including a coding scheme which combined coding for creative thinking with more established codes for critical thinking, artificial intelligence pattern-matching techniques to see if our codes could be read automatically from maps and "key event recall" interviews to explore the experience of participants. Our findings are that: (1) the emergence of new perspectives in a graphical dialogue map can be recognized by our coding scheme supported by a machine pattern-matching algorithm in a way that can be used to provide awareness indicators for moderators; (2) that the trigger events leading to the emergence of new perspectives in the online dialogues studied were most commonly disagreements and (3) the spatial representation of messages in a graphically mediated synchronous dialogue environment such as Digalo may offer more affordance for creativity than the much more common scrolling text chat environments. All these findings support the usefulness of our new account of creativity in online dialogues based on dialogic theory and demonstrate that this account can be operationalized through machine coding in a way that can be turned into alerts for moderators. (Contains 7 figures.)

Assessment of students' self-regulated learning (SRL) requires a method for evaluating whether observed actions are appropriate acts of self-regulation in theEv specific learning context in which they occur. We review research that has resulted in an automated method for context-sensitive assessment of a specific SRL strategy, help seeking while working with an intelligent tutoring system. The method relies on a computer-executable model of the targeted SRL strategy. The method was validated by showing that it converges with other measures of help seeking. Automated feedback on help seeking driven by this method led to a lasting improvement in students' help-seeking behavior, although not in domain-specific learning. The method is unobtrusive, is temporally fine-grained, and can be applied on a large scale and over extended periods. The approach could be applied to other SRL strategies besides help seeking. (Contains 3 figures.)

An emerging trend in classrooms is the use of networked visual argumentation tools that allow students to discuss, debate, and argue with one another in a synchronous fashion about topics presented by a teacher. These tools are aimed at teaching students how to discuss and argue, important skills not often taught in traditional classrooms. But how do teachers support students during these e-discussions, which happen at a rapid pace, with possibly many groups of students working simultaneously? Our approach is to pinpoint and summarize important aspects of the discussions (e.g., Are students staying on topic? Are students making reasoned claims and arguments that respond to the claims and arguments of their peers?) and alert the teachers who are moderating the discussions. The key research question raised in this work: Is it possible to automate the identification of salient contributions and patterns in student e-discussions? We present the systematic approach we have taken, based on artificial intelligence (AI) techniques and empirical evaluation, to grapple with this question. Our approach started with the generation of machine-learned classifiers of individual e-discussion contributions, moved to the creation of machine-learned classifiers of pairs of contributions, and, finally, led to the development of a novel AI-based graph-matching algorithm that classifies arbitrarily sized clusters of contributions. At each of these levels, we have run systematic empirical evaluations of the resultant classifiers using actual classroom data. Our evaluations have uncovered satisfactory or better results for many of the classifiers and have eliminated others. This work contributes to the fields of computer-supported collaborative learning and artificial intelligence in education by introducing sophisticated and empirically evaluated automated analysis techniques that combine structural, textual, and temporal data. (Contains 16 footnotes, 5 tables, and 13 figures.)

The Cognitive Tutor Authoring Tools (CTAT) support creation of a novel type of tutors called example-tracing tutors. Unlike other types of ITSs (e.g., model-tracing tutors, constraint-based tutors), example-tracing tutors evaluate student behavior by flexibly comparing it against generalized examples of problem-solving behavior. Example-tracing tutors are capable of sophisticated tutoring behaviors; they provide step-by-step guidance on complex problems while recognizing multiple student strategies and (where needed) maintaining multiple interpretations of student behavior. They therefore go well beyond VanLehn's (2006) minimum criterion for ITS status, namely, that the system has an inner loop (i.e., provides within-problem guidance, not just end-of-problem feedback). Using CTAT, example-tracing tutors can be created without programming. An author creates a tutor interface through drag-and-drop techniques, and then demonstrates the problem-solving behaviors to be tutored. These behaviors are recorded in a "behavior graph," which can be easily edited and generalized. Compared to other approaches to programming by demonstration for ITS development, CTAT implements a simpler method (no machine learning is used) that is currently more pragmatic and proven for widespread, real-world use by non-programmers. Development time estimates from a large number of real-world ITS projects that have used CTAT suggest that example-tracing tutors reduce development cost by a factor of 4 to 8, compared to "historical" estimates of ITS development time and cost. The main contributions of the work are a novel ITS technology, based on the use of generalized behavioral examples to guide students in problem-solving exercises, as well as a suite of mature and robust tools for efficiently building real-world ITSs without programming. (Contains 28 figures, 2 tables, and 5 footnotes.)