Search

This study investigated how the frequency and level of assistance provided to students interacted with prior knowledge to affect learning in the "Voyage to Galapagos" ("VTG") science inquiry-learning environment. "VTG" provides students with the opportunity to do simulated science field work in Galapagos as they investigate the key biology principles of variation, biological function, and natural selection. Thirteen teachers used the "VTG" module during their Natural Selection and Evolution curriculum unit. Students (N = 1728) were randomly assigned to one of four assistance conditions (Minimal-, Medium-, Medium-High, or High-Assistance). We predicted we would find an "Expertise Reversal Effect" (Kalyuga et al. in "Edu Psychol Rev" 194:509-539, 2007), whereby students with little prior knowledge benefit from assistance and students with higher prior knowledge benefit from minimal assistance. However, initial analyses revealed no interaction between prior knowledge and condition on student learning. To further explore results, we grouped students into 5 clusters based on student behaviors recorded during the use of "VTG." The effect of assistance conditions within these clusters showed that, in two of the five clusters, results were consistent with the Expertise Reversal Effect. However, in two other clusters, the effect was reversed such that students with low prior knowledge benefited from lower amounts of assistance and vice versa. Though this study has not identified which specific characteristics determine optimal assistance levels, it suggests that prior knowledge is not sufficient for determining when students will differentially benefit from assistance. We propose that other factors such as self-regulated learning should be investigated in future research.

This paper proposes that the field of AIED is now mature enough to break away from being delivered mainly through computers and pads so that it can engage with students in new ways and help teachers to teach more effectively. Mostly, the intelligent systems that AIED has delivered so far have used computers and other devices that were essentially designed for businesses or personal use, and not specifically for education. The future holds the promise of creating technologies designed specifically for learning and teaching by combining the power of AIED with advances in the field of robotics and in the increasing use of sensor devices to monitor our surroundings and actions. The paper assumes that "schools" (i.e., a place where children will gather to learn) will still exist in some shape or form in 25 years and that teachers will continue to oversee and promote learning among the students. It proposes that there will be educational cobots assisting teachers in the classrooms of tomorrow and provides examples from current work in robotics. It also envisions smart classrooms that make use of sensors to support learning and illustrates how they might be used in new ways if AIED applications are embedded into them.

Online testing holds much promise for assessing students' complex science knowledge and inquiry skills. In the current study, we examined the comparative effectiveness of assessment tasks and test items presented in online modules that used either a static, active, or interactive modality. A total of 1,836 students from the classrooms of 22 middle school science teachers in 12 states participated in the study as part of normal classroom activities. Students took assessments in the three different modalities on three consecutive days. The assessments tested key concepts about ecosystems and students' ability to use inquiry skills in an ecosystems context. Our in-depth analyses focused on how the different modalities elicited specific content knowledge of ecosystems (e.g., producers, consumers, predator-prey relationships) and specific inquiry skills (e.g., designing and interpreting experiments). We also investigated student use of technology supports, such as replaying animations or inspecting graphs. The results showed that the interactive modality enabled the testing of more complex reasoning and that additional experience working in the online environment improved student performance for all the modalities, especially for the interactive modality. Each of the three modalities provided useful information about students' understanding of ecosystems and related inquiry skills as well as their misconceptions. The study begins to build a knowledge base of what types of science knowledge and skills may be effectively measured in three different modalities of online assessment.

How can assessments measure complex science learning? Although traditional, multiple-choice items can effectively measure declarative knowledge such as scientific facts or definitions, they are considered less well suited for providing evidence of science inquiry practices such as making observations or designing and conducting investigations. Thus, students who perform very proficiently in "science" as measured by static, conventional tests may have strong factual knowledge but little ability to apply this knowledge to conduct meaningful investigations. As technology has advanced, interactive, simulation-based assessments have the promise of capturing information about these more complex science practice skills. In the current study, we test whether interactive assessments may be more effective than traditional, static assessments at discriminating student proficiency across 3 types of science practices: (a) identifying principles (e.g., recognizing principles), (b) using principles (e.g., applying knowledge to make predictions and generate explanations), and (c) conducting inquiry (e.g., designing experiments). We explore 3 modalities of assessment: "static," most similar to traditional items in which the system presents still images and does not respond to student actions, "active," in which the system presents dynamic portrayals, such as animations, which students can observe and review, and "interactive," in which the system depicts dynamic phenomena and responds to student actions. We use 3 analyses--a generalizability study, confirmatory factor analysis, and multidimensional item response theory--to evaluate how well each assessment modality can distinguish performance on these 3 types of science practices. The comparison of performance on static, active, and interactive items found that interactive assessments might be more effective than static assessments at discriminating student proficiencies for conducting inquiry.

This article reports on the collaboration of six states to study how simulation-based science assessments can become transformative components of multi-level, balanced state science assessment systems. The project studied the psychometric quality, feasibility, and utility of simulation-based science assessments designed to serve formative purposes during a unit and to provide summative evidence of end-of-unit proficiencies. The frameworks of evidence-centered assessment design and model-based learning shaped the specifications for the assessments. The simulations provided the three most common forms of accommodations in state testing programs: audio recording of text, screen magnification, and support for extended time. The SimScientists program at WestEd developed simulation-based, curriculum-embedded, and unit benchmark assessments for two middle school topics, Ecosystems and Force & Motion. These were field-tested in three states. Data included student characteristics, responses to the assessments, cognitive labs, classroom observations, and teacher surveys and interviews. UCLA CRESST conducted an evaluation of the implementation. Feasibility and utility were examined in classroom observations, teacher surveys and interviews, and by the six-state Design Panel. Technical quality data included AAAS reviews of the items' alignment with standards and quality of the science, cognitive labs, and assessment data. Student data were analyzed using multidimensional Item Response Theory (IRT) methods. IRT analyses demonstrated the high psychometric quality (reliability and validity) of the assessments and their discrimination between content knowledge and inquiry practices. Students performed better on the interactive, simulation-based assessments than on the static, conventional items in the posttest. Importantly, gaps between performance of the general population and English language learners and students with disabilities were considerably smaller on the simulation-based assessments than on the posttests. The Design Panel participated in development of two models for integrating science simulations into a balanced state science assessment system. (Contains 5 figures and 2 tables.)

Byline: Edys S. Quellmalz, Michael J. Timms, Barbara Buckley The Calipers project developed and studied a new generation of simulation-based assessment systems. The project aimed to demonstrate the potential of technology- and simulation-based assessments to provide high-quality evidence of complex performances for science tests that address accountability or formative goals. End-of-unit, benchmark assessments for the topics of ecosystems and for forces and motion were developed to test national science standards at the middle school and secondary levels. Technical quality evidence documented the alignment of the assessments with national science standards, expert reviews of content and item quality, cognitive analyses of students thinking-aloud, and analyses of teacher and student data gathered from classroom pilot testing. The project broke new ground in harnessing the affordances of technology to transform what, how, when and where science learning is assessed and to gather evidence of students' connected science knowledge and extended inquiry not well measured by traditional paper-based tests.