Search

In this entry, we define Intelligent Tutoring Systems (ITSs) and present a description of their core components. We outline a history of the development of ITSs with a focus on key issues that have driven change and innovation in ITSs from their inception to present day. We also present a brief case study on a specific ITS, Inq-ITS (Inquiry Intelligent Tutoring System; inqits.com), that is currently in use in 50 states in the US as well as many countries globally (12+) for the assessment of students' science competencies, and in turn, scaffolding students' science competencies. [This chapter was published in: "International Encyclopedia of Education, 4th ed., vol. 14," edited by R. J. Tierney et al., Elsevier, 2023, pp. 460-470.]

This chapter addresses students' data interpretation, a key NGSS inquiry practice, with which students have several different types of difficulties. In this work, we unpack the difficulties associated with data interpretation from those associated with warranting claims. We do this within the context of Inq-ITS (Inquiry Intelligent Tutoring System), a lightweight LMS, providing computer-based assessment and tutoring for science inquiry practices/skills. We conducted a systematic analysis of a subset of our data to address whether our scaffolding is supporting students in the acquisition and transfer of these inquiry skills. We also describe an additional study, which used Bayesian Knowledge Tracing (Corbett and Anderson. User Model User- Adapt Interact 4(4):253-278, 1995), a computational approach allowing for the analysis of the fine-grained sub-skills underlying our practices of data interpretation and warranting claims. [This is a chapter in: Auer, M. E. et al. (Eds), "Cyber-Physical Laboratories in Engineering and Science Education," (p191-217). Cham, Switzerland: Springer International.]

Researchers are trying to develop assessments for inquiry practices to elicit students' deep science learning, but few studies have examined the relationship between students' "doing," i.e. "performance assessment," and "writing," i.e. "open responses," during inquiry. Inquiry practices include generating hypotheses, collecting data, interpreting data, warranting claims, and communicating findings. The first four practices involve "doing" science, whereas the last involves writing scientific explanations, i.e. arguing using evidence. In this study, we explored whether what students wrote in their constructed responses reflected what they did during science inquiry in the Inq-ITS system. Results showed that more than half of the students' writing did not match what they did in the environment. Findings revealed multiple types of students in the "messy middle," which has implications for both teacher instruction and intelligent tutoring systems, such as Inq-ITS, in terms of providing real-time feedback for students to address the full complement of inquiry practices. [This paper was published in: "AIED 2017, LNAI 10331," edited by E. André et al., Springer International Publishing, 2017, pp. 175-187.]

In this chapter, we provide an overview of the design, data-collection, and data-analysis efforts for a digital learning and assessment environment for scientific inquiry / science practices called "Inq-ITS" ("I"nquiry "I"ntelligent "T"utoring "S"ystem; www.inqits.org). We first present a brief literature review on current science standards, learning sciences research on students' difficulties with scientific inquiry practices, and modern assessment design frameworks. We then describe how we used pilot data from four case studies with hands- on inquiry tasks for middle school students to better understand these difficulties and design various components of the Inq-ITS system to support students' inquiry accordingly. Lastly, we describe how we used key computational techniques from knowledge-engineering and educational data mining to analyze data from students' log files in this environment to (1) automatically score students' inquiry skills, (2) provide teachers with fine-grained, rich, classroom-based formative assessment data on these practices, and (3) react in real time to scaffold students as they engage in inquiry. [This paper was published in: Rupp, A.A. & Leighton, J.P (Eds.) "The Wiley Handbook of Cognition and Assessment: Frameworks, Methodologies, and Applications." West Sussex, UK (pp. 508-534).]

Many national policy documents underscore the importance of 21st century skills, including critical thinking. In parallel, recent American frameworks for K-12 Science education call for the development of critical thinking skills in science, also referred to as science inquiry skills/practices. Assessment of these skills is necessary, as indicated in policy documents; however, this has posed a great challenge for assessment researchers. Recently, some science learning environments seek to assess these science skills. These systems log all students' interactions within the given system, and if fully leveraged, these logs provide rich assessments of inquiry skills. Here we describe our environment Inq-ITS (Inquiry Intelligent Tutoring System), that uses Educational Data Mining to assess science inquiry skills, as described as 21st century skills. Additionally, here we describe how we measure students' skills at designing controlled experiments, a lynchpin skill of inquiry, in the context of complex systems. In doing so, our work addresses 21st century skill assessment in two ways, namely of inquiry (designing and conducting experiments), and in the context of complex systems, a key topic area of 21st century skills. We use educational data mining to develop our assessment of this skill for complex systems. [This paper was published in "Thinking Skills and Creativity" v18 p81-90 2015.]

There are well-acknowledged challenges to scaling computerized performance-based assessments. One such challenge is reliably and validly identifying ill-defined skills. We describe an approach that leverages a data mining framework to build and validate a detector that evaluates an ill-defined inquiry process skill, designing controlled experiments. The detector was originally built and validated for use with physical science simulations that have a simpler, linear causal structure. In this paper, we show that the detector can be used to identify demonstration of skill within a life science simulation on Eco- systems that has a complex underlying causal structure. The detector is evaluated in three ways: 1) identifying skill demonstration for a new student cohort, 2) handling the variability in how students conduct experiments, and 3) using it to determine when students are off-track before they finish collecting data. [This paper was published in: "Proceedings of the International Conference on Intelligent Tutoring Systems," 2014.]

Students conducted inquiry using simulations within a rich learning environment for 4 science topics. By applying educational data mining to students' log data, assessment metrics were generated for two key inquiry skills, testing stated hypotheses and designing controlled experiments. Three models were then developed to analyze the transfer of these inquiry skills between science topics. Model one, Classic Bayesian Knowledge Tracing, assumes that either complete transfer of skill occurs or no transfer occurs; model two (BKT-PST), an extension of BKT, assumes partial transfer and tests that assumption; and model three, a variant of BKT-PST, assumes no transfer and tests this assumption. An analysis of models one and two suggest that transfer of these inquiry skills across topics did occur. This work makes contributions to methodological approaches for measuring fine-grained skills using log files, as well as to the literature on the domain-specificity vs. domain-generality of inquiry skills. [This paper was published in: "Proceedings of the International Conference of Learning Sciences," 2014.]

When validating assessment models built with data mining, generalization is typically tested at the student-level, where models are tested on new students. This approach, though, may fail to find cases where model performance suffers if other aspects of those cases relevant to prediction are not well represented. We explore this here by testing if scientific inquiry skill models built and validated for one science topic can predict skill demonstration for new students and a new science topic. Test cases were chosen using two methods: student-level stratification, and stratification based on the amount of trials ran during students' experimentation. We found that predictive performance of the models was different on each test set, revealing limitations that would have been missed from student-level validation alone.

We present work toward automatically assessing and estimating science inquiry skills as middle school students engage in inquiry within a physical science microworld. Towards accomplishing this goal, we generated machine-learned models that can detect when students test their articulated hypotheses, design controlled experiments, and engage in planning behaviors using two inquiry support tools. Models were trained using labels generated through a new method of manually hand-coding log files, "text replay tagging". This approach led to detectors that can automatically and accurately identify these inquiry skills under student-level cross-validation. The resulting detectors can be applied at run-time to drive scaffolding intervention. They can also be leveraged to automatically score all practice attempts, rather than hand-classifying them, and build models of latent skill proficiency. As part of this work, we also compared two approaches for doing so, Bayesian Knowledge-Tracing and an averaging approach that assumes static inquiry skill level. These approaches were compared on their efficacy at predicting skill before a student engages in an inquiry activity, predicting performance on a paper-style multiple choice test of inquiry, and predicting performance on a transfer task requiring data collection skills. Overall, we found that both approaches were effective at estimating student skills within the environment. Additionally, the models' skill estimates were significant predictors of the two types of inquiry transfer tests. [This paper was published in "User Modeling and User-Adapted Interaction" v23 p1-39 2013.]

In recent years, an increasing number of analyses in Learning Analytics and Educational Data Mining (EDM) have adopted a "Discovery with Models" approach, where an existing model is used as a key component in a new EDM/analytics analysis. This article presents a theoretical discussion on the emergence of discovery with models, its potential to enhance research on learning and learners, and key lessons learned in how discovery with models can be conducted validly and effectively. We illustrate these issues through discussion of a case study where discovery with models was used to investigate a form of disengaged behavior, i.e., carelessness, in the context of middle school computer-based science inquiry. This behavior has been acknowledged as a problem in education as early as the 1920s. With the increasing use of high-stakes testing, the cost of student carelessness can be higher. For instance, within computer-based learning environments careless errors can result in reduced educational effectiveness, with students continuing to receive material they have already mastered. Despite the importance of this problem, it has received minimal research attention, in part due to difficulties in operationalizing carelessness as a construct. Building from theory on carelessness and a Bayesian framework for knowledge modeling, we use machine-learned detectors to predict carelessness within authentic use of a computer-based learning environment. We then use a discovery with models approach to link these validated carelessness measures to survey data, to study the correlations between the prevalence of carelessness and student goal orientation. [This paper was published in "American Behavioral Scientist" v57 n10 p1479-1498 2013.]

In this paper we explored whether engaging in two inquiry skills associated with data collection, designing controlled experiments and testing stated hypotheses, within microworlds for one physical science domain (density) impacted the acquisition of inquiry skills in another domain (phase change). To do so, we leveraged educational data mining techniques to both assess and estimate students' inquiry skills across domains. Analyses revealed that honing these skills in density activities provided benefits in terms of transfer and skill acquisition. More specifically, students who practiced in density activities first were more likely to show mastery of the designing controlled experiments skill than those who had no prior practice. These same students were also more likely to test their stated hypotheses during their first data collection in phase change. Thus, practice in one domain can positively impact acquisition and transfer of skill in a second domain, suggesting that inquiry skills also have a degree of domain generality.

The National frameworks for science emphasize inquiry skills (NRC, 1996), however, in typical classroom practice, science learning often focuses on rote learning in part because science process skills are difficult to assess (Fadel, Honey, & Pasnick, 2007) and rote knowledge is prioritized on high-stakes tests. Short answer assessments of inquiry have been used (cf., Alonzo & Aschbacher, 2004; Songer, 2006), however, these tend to not align well to current national frameworks (Quellmalz, Kreikemeier, DeBarger, & Haertel, 2007) and it is unclear whether they properly identify inquiry skills (Black, 1999; Pellegrino, 2001). Hands-on performance assessments are more authentic (Baxter and Shavelson 1994; Ruiz-Primo & Shavelson, 1996), however, these are seldom used in schools because of difficulty with reliable administration and the resulting high cost. The Science Assistments project (www.scienceassistments.org) has developed a rigorous, technology-based learning environment that assists and assesses hence, "assistments") middle school students in Earth, Life, and Physical Science so that teachers can assess their students' skills rigorously, frequently, and during instruction--in the context in which they are developing (Mislevy et al, 2002). The authors' program of work represents a significant advance over other programs that utilize pencil and paper assessments because theirs makes use of a state-of-the art logging infrastructure to do web-based tutoring (Razzaq et al, 2005). As a proof of concept for automated assessment of scientific inquiry skills, the authors used model-tracing (Corbett & Anderson, 1995; Koedinger & Corbett, 2006) to develop a cognitive model of science inquiry skills, particularly, the control for variables strategy (Chen & Klahr, 1999) and warranting claims with data. This model provides a rich qualitative, process-oriented scoring of students' inquiry "moves" within a guided scientific inquiry simulation for the domain of state change. They address the validity of this automated approach to performance assessment both quantitatively, in terms of reliability and predictive validity, and qualitatively, in terms of providing rich traces of student inquiry steps and "mis-steps" or haphazard inquiry (Buckley, Gobert et al, 2010). Participants were 78 eighth grade students, ranging in age from 12-14 years, from a public middle school in Central Massachusetts. In this paper the authors have shown that they can use model-tracing as a method of performance assessment for science inquiry skills, an ill-defined domain. This builds upon the extensive work that has been done to date for well-defined domains such as math (Corbett & Anderson, 1995; Koedinger & Corbett, 2006). Additionally: (1) the reliability of their machine-scored measures of inquiry are highly consistent across the 4 Assistment activities or "trials", suggesting that they can reliably capture students' inquiry performance on these rich inquiry tasks, and (2) their measures are moderately correlated with post-test measures of inquiry performance for analogous concepts. Lastly, their data show that model-tracing can detect interesting patterns of student inquiry such as confirmation bias and overcoming confirmation basis. These are important data with respect to demonstrating auto-scoring of rich inquiry behaviors, but are also important, particularly the former, in terms of its implications for adaptive scaffolding of student inquiry, such as that being done by the Science Assistments group (www.scienceassistments.org; Gobert et al, 2007, 2009). This work makes contribution to theoretical understanding of scientific inquiry, to its assessment, and to technical methods to auto-score inquiry. This represents an advance in this area since to date there has been difficulty in separating inquiry from the domain-specific context in which it was learned (Mislevy et al., 2002; Gobert, Pallant, & Daniels, 2010), and difficulty measuring inquiry skills due to their complexity and the amount of data required for reliable measurement (Shavelson et al, 1999). (Contains 1 table.)

The authors present work towards automatically assessing data collection behaviors as middle school students engage in inquiry within a physics microworld. In this study, the authors used machine learned models that can detect when students test their articulated hypotheses, design controlled experiments, and engage in planning behaviors using their inquiry support tools. They compared two approaches, an averaging-based method that assumes static skill level and Bayesian Knowledge Tracing, on their efficacy at predicting skill before a student engages in an inquiry activity and on predicting performance on a paper-style multiple choice test of inquiry and a transfer task requiring data collection skills. Their data were collected in a rural town in Central Massachusetts. Participants were 134 eighth grade students, ranging in age from 12-14 years, from a public middle school in Central Massachusetts. Their findings provide some evidence that the skills for successfully engaging in authentic inquiry and answering equivalent paper test-style questions are related (Black, 1999; Pellegrino, 2001). Furthermore, their findings support the notion that authentic skill learned in one context can be applied to other domains, as shown by the significant correlation between performance in designing controlled experiments in two domains. As such, these models have considerable potential to enable future "discovery with models" analyses (cf. Baker, 2010) that can shed light on the relationship between a student's mastery of systematic experimentation strategies and their domain learning. Additional research will be needed to determine if these findings are robust over different student populations and if the feature set and associated detectors are general enough (cf. Ghazarian & Noorhosseini, 2010) to be applicable to microworlds in other scientific domains. It will also be important to determine if these relationships will hold after incorporating scaffolding, thus giving students a better opportunity to both perform well despite incomplete knowledge, and to acquire these skills while using the microworld. (Contains 2 tables.)

Models and modeling are frequently used as instructional tools in science education to convey important information concerning both the explanatory and structural features of topic areas in science. The efficacy of models as such rests almost entirely upon students' ability to conceptualize them as abstracted "representations" of scientific phenomena. This investigation considers the relationship between students' epistemology of scientific models and their success at learning about a complex system, namely plate tectonics. Ninth-grade students were asked to draw diagrams at three specific points in a short text: (1) a static model; (2) a causal/dynamic model of the movement in the layers; and (3) an outcome in the world, i.e., volcanic eruption due to two plates moving away from one another. Students received a posttest to assess different types of knowledge, namely spatial knowledge, causal/dynamic knowledge, and knowledge through inference. Students were also administered an epistemology questionnaire to assess understanding of what a model is. Those who held more sophisticated epistemologies of science were better able to transfer what they had learned in order to reason about other plate tectonics phenomena and more difficult conceptual knowledge involved in the causal mechanisms of plate tectonics. Contains 25 references. (PVD)

Science assessments should evaluate the full complement of inquiry practices (NGSS, 2013). Our previous work has shown that a large proportion of students' open responses did not match their scientific investigations (Li et al., 2017a). The present study both unpacks and compares the sub-components underlying students' performance for experimenting to their written open responses. These findings have implications for the assessment of inquiry practices, design of real-time scaffolding, and teachers' instruction of science. [This paper was published in: "Rethinking Learning in the Digital Age: Making the Learning Sciences Count. 13th International Conference of the Learning Sciences," 2018.]