Search

This Postsecondary Teaching with Technology Collaborative (the Collaborative) report examines how faculty and staff at seven community colleges and two broad-access universities reconsidered students' online learning needs in the midst of the pandemic. The report focuses on a set of interconnected student mindsets and competencies that the authors call self-directed learning (SDL) skills; in their framing, SDL skills include motivational processes (e.g., building self-efficacy and sense of belonging), metacognitive processes (e.g., identifying and adjusting learning strategies and reflecting), and applied learning processes (e.g., managing time and seeking help). The authors explore whether and how the colleges offered supports for students to strengthen their SDL skills and the extent to which SDL skill supports were offered within online courses versus outside of class. Findings from interview data and secondary sources show that most institutions made changes to improve conditions for online teaching and learning, including improved use of learning management systems, increased professional development for instructors, improved physical technology infrastructure and expanded software licenses, and the development of student-facing online orientation materials. At the same time, increased use of online courses continued to present challenges, including poorer student performance in online course sections, lack of meaningful communication and engagement, lower likelihood for online students to seek help when they need it, and increased demands for students to apply SDL skills such as time and task management. Moreover, many SDL supports occurred at the institution level and therefore were not tailored to particular content areas or course types. These supports provided students with valuable and needed resources to help them succeed in college but also had some limitations. Finally, while most faculty and staff the authors interviewed expressed that SDL skills are important and should be supported by the institution, explicit instructional support for SDL skills inside classrooms appeared to be limited and uneven; faculty may not know how to teach these skills, may not think that they have the time or resources to do so, or may not see it as their responsibility. The report concludes with three recommendations for institutions to promote SDL skill development. "Teaching and Designing Online STEM Courses to Support Self-Directed Learning Skills" which focuses on teaching and designing online STEM courses to support SDL skills is a collaborative report.

The Postsecondary Teaching with Technology Collaborative (the Collaborative) is currently researching the obstacles students face in online STEM courses and the ways that institutions and instructors can better support these students. As part of this research program, the authors conducted qualitative research in partnership with six community colleges and two broad-access universities to learn if, when, and how instructors support self-directed learning (SDL) skill development in their online courses. SDL skills can be understood as motivational, metacognitive, and applied learning processes--in other words, the ways that students maintain motivation, reflect and think about their learning, plan and set goals, and adjust their study strategies. In spring 2022, the authors surveyed 141 STEM instructors online and conducted 12 virtual "course tours" with STEM instructors who shared their screens and walked researchers through their course structure and content in the learning management system. Instructors in the sample perceived online courses as learning environments where it is more difficult to engage students and help them stay on track with their coursework. According to faculty, students in online courses are less likely than those in in-person courses to experience the excitement and motivation of being part of a community of learners because of the lack of opportunities to engage with peers. Instructors also described fewer opportunities to learn about and build connections with their students through informal conversations, and they emphasized a number of barriers to incorporating SDL skills into instruction. Despite this, the authors provide a variety of examples of strategies and activities instructors indicated they use to support the processes that are theorized--according to the Collaborative's SDL framework--to lead to SDL skill development, as well as three case examples of SDL instruction that faculty incorporated into their STEM course teaching. The brief concludes with an overview of the types of instructional strategies that support SDL skill development and how they can be used throughout a course. A Collaborative report on teaching and designing online STEM courses to support SDL skills is also available.

In recent years, children's video viewing has shifted from television to online streaming platforms like YouTube. While such platforms include many educational videos for children, we know little about the educational value of the content in these videos. To understand the learning opportunities available to children when they watch videos online, this paper describes the quality, duration, and popularity of a sample of 1,198 YouTube videos focused on early literacy and math topics. We found that while most of these videos use concrete examples to illustrate learning content, only about half directly address the audience or involve characters, and very few integrate learning content into the narrative. Further, videos in our sample that directly addressed the audience were less popular, as measured by views and likes, as were videos that included a greater variety of math topics. We discuss important ways that user-generated educational videos may differ from educational television programs, including decreased reliance on narratives and characters as compared to educational television programs. Our findings also hint at novel influences on the content and quality of online videos, such as the likelihood that the algorithm will recommend a video or that a viewer will like it. Finally, we discuss implications for parents, content creators, and researchers.

The Mathematics, 3D Printing, and Computational Thinking Through Work-Based Learning (MPACT) program combines teacher professional development, specialized curriculum and materials and STEM industry mentors to provide grade 4-7 students with project-based experiences implemented across three learning modules. This technical report presents findings from SRI International's evaluation of MPACT implementationin the 2021-22 school year by MPACT Fellows--i.e., teachers who participated in the MPACT program--in four U.S. states. MPACT Fellows implemented the program in a year marked by ongoing difficulties due to the COVID-19 pandemic. Although MPACT professional development was delivered with fidelity, only 65% of MPACT Fellows implemented the full program (all three modules) with all of their classes. MPACT Fellows also provided fewer opportunities for students to meet with or learn about STEM industry mentors than intended. Despite this partial implementation, MPACT Fellows' perceptions of and efficacy in programmatic concepts increased meaningfully after participating in MPACT. Further, grade 4 and 5 MPACT students grew nearly a full standard deviation on a measure of geometry, computational thinking, and spatial reasoning over one school year. However, significant differences were not observed in students' socioemotional outcomes--specifically, behavioral engagement in math, behavioral disaffection in math, math self-efficacy, and math self-concept--between MPACT students and students in the comparison group. The considerable growth of MPACT students on the assessment and the documented program impacts on teachers' perceptions provide limited but suggestive evidence that the program could demonstrate improved student outcomes in ideal conditions, if examined over a longer time frame or using different impact measures.

Expulsion and suspension (or exclusionary discipline) in early care and education (ECE) are widely recognized as harmful to children. Numerous states have developed expulsion and suspension prevention policies through legislation or administrative rules. However, there is limited research on the implementation of state policies designed to reduce exclusionary discipline and on ECE providers' use of program supports that could help reduce exclusionary discipline practices. In this report, we use survey data to investigate Arkansas ECE program leaders' and teachers' understanding of the state's expulsion prevention policy, their use of suspension and expulsion and their participation in professional development (PD) and other supports to meet the needs of children with challenging behavior. We also explore how teachers' attributions of the reasons for challenging behavior predict their use of the available PD and supports. Key findings include: (1) Publicly funded programs in Arkansas (which are subject to state policy) engaged in exclusionary discipline at lower rates than non-publicly funded programs, despite reporting a higher percentage of children with challenging behavior; (2) Many Arkansas ECE program leaders were aware of the state's expulsion prevention policy, but fewer demonstrated a strong understanding of the requirements; (3) More than three quarters of Arkansas ECE teachers reported receiving some form of PD or training on topics such as encouraging positive teacher-child interactions, promoting children's positive behavior and helping children develop skills like managing their emotions; and (4) The use of exclusionary discipline practices was related to teachers' views of families as a reason for children's challenging behavior. At the end of the report, we offer recommendations based on these findings.

A recently launched professional development initiative in Arkansas, called Array, is designed to proactively identify early care and education (ECE) programs that could benefit from social-emotional-focused (SE-focused) professional development and support, and to offer assistance tailored to a program's particular needs. Because Array is designed to initiate contact with ECE programs, it has the potential to intervene early and prevent serious escalations of teachers' frustration in response to children's challenging behavior. This report examines results from case studies in three ECE programs that participated in Array between June and December 2022. The results focus on unique features of Array, how it operates, reported benefits to programs, and recommendations for continued improvement.

In 2022-23, SRI Education conducted an evaluation of a pilot implementation of the Maine Early Childhood Consultation Partnership® (ECCP®) for Maine's Department of Health and Human Services, Office of Child and Family Services. Created and developed by Advanced Behavioral Health, Inc., ECCP® is an evidence-based, manualized model of infant and early childhood mental health consultation (IECMHC) that builds capacity of the early care and education workforce to address young children's social-emotional and behavioral needs through intensive, time-limited consultation with a trained mental health consultant. SRI Education collected and analyzed data from state leaders, consultants, trainers, supervisors, teachers and parents/guardians in five counties to assess the pilot implementation and provide recommendations for statewide scale up and sustainability of the model. [This report was prepared for Office of Child & Family Services, Maine Department of Health and Human Services.]

The University of California, Irvine Writing Project's (UCI Writing Project's) Pathway to Academic Success project (Pathway) aims to improve students' academic writing through professional development for secondary English language arts (ELA) and advanced English language development (ELD) teachers. Based on evidence of Pathway's previous success in improving student achievement, in 2018 the UCI Writing Project received a federal Education Innovation and Research (EIR) grant to expand its capacity to deliver Pathway professional development to new sites and in new contexts. In support of this grant, SRI International (SRI) conducted a random assignment evaluation of Pathway in grades 7-11, over the course of three school years (2019/20, 2020/21, and 2021/22). Despite disruptions to program implementation during the COVID-19 pandemic, we found that Pathway led to positive, statistically significant impacts on students' writing. This study is the second independent evaluation of Pathway when implemented by expansion sites with UCI Writing Project support. Both studies have found positive and statistically significant effects on students' writing. The size, rigor, and independence of these studies provide a strong evidence base to support Pathway's effectiveness in improving secondary students' academic writing at scale and in diverse contexts.

This study reviewed applied research literature testing the efficacy of using instructional strategies to develop college students' self-directed learning skills. Researchers conducted a systematic database review of recent empirical studies from 2011 through 2021 and synthesized findings from four recent literature reviews into for two types of self-directed learning processes--motivational and metacognitive. The review sought to identify instructional strategies that could be adapted to online courses and showed promising evidence of increasing college students' reported use or awareness of self-directed learning skills or improving academic success in science, technology, engineering, or mathematics (STEM) courses, particularly those offered in the first year of higher education. The initial database search found 4,263 studies using terms related to "self-regulated learning," STEM domains, technology use, and both postsecondary and secondary students. Initial screening focused on studies using self-regulated learning as either an outcome or predictor variable in classroom interventions with a college or high school population reduced the sample to 209 studies. Subsequent screening identified 15 studies using treatment-comparison design with baseline equivalence and outcomes related to course grades, domain knowledge, and self-regulated learning skills. Our review found that several strategies showed significant positive impacts on academic outcomes or self-reported use of self-regulated learning skills and appeared adaptable to online learning application. We ultimately drew from literature beyond self-regulated learning, including motivational frameworks such as self-determination theory. Consultations with education partners favored aligning our framework with the adult-learning orientation of self-directed learning theory. We synthesized these findings with the findings from the four recent literature review that showed promising results for all students or historically marginalized populations. [This report was created by the Postsecondary Teaching with Technology Collaborative.]

Digital learning programs are a promising tool to support teachers in providing students with strong early literacy instruction, but more research is needed to understand their effectiveness. Sound Town is a digital early literacy program that provides children in prekindergarten with instruction and practice in foundational literacy skills, using video-based activities. A team at SRI Education completed an independent evaluation of Sound Town to learn more about the impact of Sound Town on students' phonological awareness skills and letter-sound correspondence skills. The findings indicate that Sound Town has the potential to support students in their development of foundational literacy skills and shows promise as a supplement to prekindergarten teachers' core literacy instruction. [The team at Hoogalit, Inc. supported this evaluation.]

The goal of the William and Flora Hewlett Foundation's Deeper Learning + Diffusion of Innovation and Scaled Impact Initiative, launched in 2018, was to generate knowledge about how fundamental shifts in teaching and learning could be scaled within public school systems efficiently, expediently, and equitably. The foundation funded 10 research-practice partnerships (RPPs) to accelerate educational improvement and facilitate learning. This report summarizes the progress the RPPs made and how the grant structures and supports facilitated or inhibited this progress. The report includes several implications for funders seeking to scale change in educational settings and optimize learning from these efforts.

Research consistently shows that among young children, poor social skills and problem behaviors can lead to poor relationships with teachers and peers (Boivin et al., 1995), higher risk of academic failure (Buhs et al., 2006), and higher school dropout (Gilliam, 2005). Moreover, prekindergarten (PreK) and kindergarten (K) teachers report feeling underprepared to address students' social-emotional needs (Alkon et al., 2003), and they often have few resources that have been developed with young children in mind. Consequently, there is an unmet need for supporting PreK and K teachers to implement developmentally appropriate, evidence-based, and effective strategies to support children's social-emotional learning (SEL). The field lacks information on how to scale what we know from smaller studies to district- or state-wide practice while maintaining the quality and impact of SEL initiatives. We present findings from an independent evaluation conducted by SRI examining the implementation and impact of the "Pyramid Model" (PM) in one urban and one rural district in the southeast United States. The PM consists of four components: materials and resources to support SEL in the classroom, training for teachers on high-quality SEL teaching practices, Practice-Based Coaching (PBC) supports for teachers that is ongoing and job-embedded, and additional supports for students in need of more intensive support. The model developers trained district coaches to deliver PBC to support teachers' implementation of the PM strategies. The evaluation team conducted two studies to examine impacts of the PM on teachers and students: a randomized controlled trial (RCT) using school-level assignment of PreK teachers, and an RCT using teacher-level assignment of K teachers. Four cohorts of teachers and students participated between 2018 and 2022. Comparison schools and classrooms continued to implement business-as-usual instructional and SEL supports. Participating schools primarily served ethnically and racially diverse student populations with approximately one fourth of the student populations identified as multilingual learners. Participating PreK teachers were primarily white and female with significant experience teaching young children and certifications/degrees in early childhood. Participating K teachers were similar to PreK teachers but were less likely to have degrees in early childhood. The study sample of students matched the population of students at the schools, with a third identified as Black or African American, a fourth identified as Hispanic or Latinx, and a third identified as white. The project met its primary goals of testing the innovative use of district-hired coaches to support teacher implementation of the PM in the classroom and expanding and testing the PM into K classrooms. The evaluation findings support the use of district-hired coaches to coach teachers to fidelity. The impact studies showed small but nonsignificant effects on teacher-reported math skills (PreK only) and students' social problem-solving skills (K only), suggesting the promise of the PM intervention for improving the social-emotional competence of the youngest learners and supporting academic learning and success. An important contextual factor is that teachers' implementation of the PM and data collection activities were interrupted by the COVID-19 pandemic.

Instructional coaching, informed by observation tools that measure teachers' practices, has been effective in improving teaching quality in early learning programs. However, existing measurement tools limit teachers' abilities to implement this type of instructional coaching at scale. To address this challenge, a team at SRI Education, along with partners at Substantial and the Bill and Melinda Gates Foundation, created a Target Product Profile (TPP) that describes the ideal features and capabilities of observation tools to support instructional coaching with prekindergarten teachers. This TPP is designed to spur investments, improvements, and innovations in existing and future observation tools for instructional coaches that measure important teaching practices and provide useful feedback on ways to improve classroom quality. [This resource was developed with Substantial.]

High-quality early care and education (ECE) provides a developmentally supportive environment for children in foster care that can help address early adversities, but recent studies have shown low levels of ECE participation among these children (Klein et al., 2016; Lee, 2020). To address this important issue, the Arkansas Office of Early Childhood (OEC) is collaborating with SRI International, the National Center for Children in Poverty, and the University of Arkansas for Medical Sciences. The partnership team is focused on learning about the supply, quality, and stability of ECE settings for children in foster care, as well as the facilitators and barriers to children in foster care accessing high-quality ECE. This brief presents key findings from focus groups and interviews with resource parents (that is, foster parents), ECE center-based program directors, and ECE home-based providers about their experiences caring for children ages birth through 5 in foster care. Key findings from this report include the following: (1) Working parents report that they cannot accept foster child placements if they lack child care and that children benefit from the developmental supports of high-quality ECE; (2) Resource parents struggle to find child care on their own, with many relying on informal networks of support (e.g., social media, faith-based communities) to find child care; (3) Even when a child can be enrolled in a quality program, their ECE placement is often disrupted during transitions (e.g., reunification); (4) Resource parents and ECE providers report challenges related to using and accepting child care vouchers (e.g., delayed voucher payments); and (5) ECE providers recognize that providing care to children in foster care requires deep commitment and specific skills, and they want specialized training and support to do this. We provide suggestions for how these findings can inform strategies to increase the participation of children in foster care in high-quality ECE. [This paper was created with the Arkansas Office of Early Childhood and the University of Arkansas for Medical Sciences.]

This report describes initial findings from a study conducted across three California school districts during the 2021-22 school year of science curriculum materials that were designed to promote learning as called for by the Next Generation Science Standards (NGSS). The study, led by WestEd, was an independent randomized controlled trial to evaluate the efficacy of the NGSS-designed Amplify Science 1st grade curriculum. The curriculum aims to support science language and literacy in day-to-day science lessons in ways meant to integrate science learning with students' literacy development. It is among the first randomized controlled trials of widely available curriculum materials for the NGSS in K-3 classrooms. Of high interest was studying the extent to which the curriculum supports young students' three-dimensional learning, their literacy learning related to reading and the use of science-related language, and the nature of teachers' implementation. The results show that science curriculum materials that use the high-leverage strategy of integrated science and literacy can improve students' proficiency with science, enrich their science vocabulary knowledge and usage, and simultaneously develop their reading skills. [This study was conducted in partnership with LFC Research.]

The Barr Foundation's Engage New England (ENE) initiative was an effort to catalyze high school innovation by developing exemplary schools that support the success of students who are off track to graduate. Grounded in the tenets of positive youth development, the ENE initiative provided grants and technical assistance to support new or redesigned schools in creating rigorous and purposeful educational programs for students who have not experienced success in traditional high schools. Through ENE, the foundation sought to demonstrate how student-centered schools can meet the varied needs of students, especially historically underserved students, and ensure their postsecondary success. This brief presents lessons learned about school transformation from the foundation's experience supporting three cohorts of grantees. The insights gained from the ENE initiative can inform reform efforts in all high schools.

The Mathematics, 3D Printing, and Computational Thinking Through Work-Based Learning (MPACT) program relies on teacher professional development, specialized curriculum and materials, and STEM industry mentors to provide grades 4-7 students with project-based experiences implemented across three learning modules. This technical report presents findings from a program evaluation of MPACT as implemented in the 2021-22 school year by MPACT Fellows--i.e., teachers who participated in the MPACT program--in four U.S. states. MPACT Fellows implemented MPACT in a year marked by ongoing difficulties brought on by the COVID-19 pandemic. Although MPACT professional development was delivered with fidelity, only 65 percent of MPACT Fellows implemented the full program with all of their classes. MPACT Fellows also provided fewer opportunities for students to meet with or learn about STEM industry mentors than intended. Despite this partial implementation, MPACT Fellows' perceptions of and efficacy in programmatic concepts increased meaningfully after participating in MPACT. Furthermore, grades 4 and 5 MPACT students grew nearly a full standard deviation on a measure of geometry, computational thinking, and spatial reasoning over one school year. However, significant differences were not observed in students' socioemotional outcomes--specifically, behavioral engagement in math, behavioral disaffection in math, math self-efficacy, and math self-concept--between MPACT students and comparison students. The considerable growth of MPACT students on the assessment and the documented program impacts on teachers' perceptions therefore provide limited, suggestive evidence that the program could demonstrate improved student outcomes in ideal conditions if examined over a longer time frame or using different impact measures.

Developed by the National Writing Project (NWP), the College, Career, and Community Writers Program (C3WP) seeks to improve students' argument writing by building teachers' understanding of and skill in teaching source-based argument writing. The program features intensive professional development, skill-based instructional resources, and formative assessment tools designed to help teachers bring C3WP into their classrooms. After two prior studies of C3WP showed positive and statistically significant effects on grade 7-10 students' source-based argument writing, NWP received a 2016 Investing in Innovation (i3) Scale-up grant from the US Department of Education. With this grant, NWP aimed to: (1) develop greater capacity to implement and sustain C3WP through scaling activities and (2) implement C3WP in grades 7-10 in 2018-19 and 2019-20 and grades 4-5 in 2019-20. SRI received funds through this grant to serve as external evaluator. This technical report provides SRI's independent evaluation findings in line with i3 reporting requirements. SRI employed a district-level cluster randomized controlled trial to assess the impact of C3WP on students' source-based argument-writing achievement. In spring 2018, SRI randomized 48 rural districts in 16 states to receive either C3WP or a business-as-usual comparison condition. One treatment district attrited shortly after randomization. SRI used various data sources, including administrative program data and teacher surveys, to examine scaling and program implementation. SRI administered a source-based argument writing performance task as a baseline measure in fall 2018 and outcome measure in spring 2019, after one year of program implementation in grades 7-9. SRI deidentified the writing and NWP scored it using the four attributes measured by the Analytic Writing Continuum for Source-based Writing (AWC-SBA): content, structure, stance, and conventions. The COVID-19 pandemic disrupted the second year of C3WP implementation in grades 7-10 and the only year of implementation in grades 4-5. Ultimately, SRI found that NWP's implementation of C3WP met all measured thresholds of implementation fidelity and scaling other than those impacted by the pandemic (number of hours of professional development in 2019-20 and regional conferences, intended to be held in 2020). SRI found that one year of C3WP had positive and statistically significant impacts on all four attributes of grade 7-9 students' writing measured by the AWC-SBA.

Beginning teachers enter a profession that places particularly challenging demands on novice practitioners. The New Teacher Center's (NTC) induction model provides intensive, instructionally focused coaching to teachers during their first two years in the classroom, in-depth training for induction mentors, a suite of tools to guide coaching cycles, and capacity-building for district leaders to sustain induction mentoring programs after NTC's direct involvement in the district ends. With funding from a U.S. Department of Education Investing in Innovation (i3) scale-up grant, NTC tested strategies for scaling its validated induction model to 301 schools in five school districts serving high proportions of students of color and students from low-income households. NTC adapted its model to support district adoption at scale, including an option for deploying part-time, school-based mentors, reduced requirements for mentor training, and online training and video-sharing tools. SRI's evaluation of the implementation and impact of NTC's i3 scale-up grant employed a cluster-randomized controlled trial design with schools as the unit of randomization. All first-year teachers in randomized schools were included in the study. Treatment teachers received induction supports from NTC-trained mentors, while control teachers received the supports provided by their districts under business-as-usual conditions. The evaluation examined fidelity of implementation to the model as designed, the contrast between the induction supports in the treatment and control conditions, and impacts on three key outcomes: 1) teachers' classroom practice as measured by the Danielson Framework for Teaching, 2) student achievement on state standardized assessments in mathematics and English Language Arts (ELA) in grades 4 through 8, and 3) teacher retention within district. NTC's induction model was not implemented with adequate fidelity in any of the five sites according to thresholds set by NTC, and the mentoring received by NTC treatment teachers was not substantially different in key respects from the mentoring received by control teachers. There were no statistically significant impacts of the model as implemented on overall teacher practice, student achievement, or teacher retention. Exploratory findings suggest conditions under which NTC might see a greater impact. There was a positive correlation between students' mathematics achievement and mentoring that met NTC's fidelity thresholds for frequency and duration, as well as between mathematics achievement and mentoring that met NTC's expectations for instructional focus. NTC induction supports also had a positive impact on student ELA achievement in schools with higher proportions of historically underserved students. These findings indicate the importance of ensuring high-quality implementation of a program. Under the i3 scale-up grant, NTC attempted to adapt its model for scaling, but the partner districts failed to fully implement key components and mediators as intended. There is evidence that the model has promise when fully implemented, particularly in schools with higher proportions of historically underserved students, but without further research this evidence is simply suggestive.

This case study report is for people interested in funding, managing, forming, or improving community partnerships. It provides an in-depth account of strategies associated with success in Community Collaboratives for Early Learning and Media, or CC-ELM. A national network of thirty CC-ELM--local partnerships among public media stations and organizations such as schools, libraries, afterschool program providers, parent and family advocacy groups, and housing authorities--were developed and implemented between 2015 and 2020 as part of the CPB-PBS Ready To Learn Initiative funded by the U.S. Department of Education. CPB-PBS Ready To Learn Research partners EDC and SRI International completed six case studies as part of their evaluation of CC-ELM implementation and impacts. The report opens with a cross-case analysis that places case study findings in the context of overall evaluation findings; the six case study reports follow, each focused on a single community. The Ready To Learn Initiative is a cooperative agreement funded and managed by the U.S. Department of Education's Office of Elementary and Secondary Education. It supports the development of innovative educational television and digital media targeted to preschool and early elementary school children and their families. Its goal is to promote early learning and school readiness, with a particular interest in reaching children from low-income families. In addition to creating television and other media content, the program supports activities intended to promote national distribution of the programming, effective educational uses of the programming, community-based outreach, and research on educational effectiveness.

This report is based on the discussion that emerged from a convening of a panel of 22 experts in artificial intelligence (AI) and in learning. It introduces three layers that can frame the meaning of AI for educators. First, AI can be seen as "computational intelligence" and capability can be brought to bear on educational challenges as an additional resource to an educator's abilities and strengths. Second, AI brings specific, exciting new capabilities to computing, including sensing, recognizing patterns, representing knowledge, making and acting on plans, and supporting naturalistic interactions with people. Third, AI can be used as a toolkit to enable us to imagine, study, and discuss futures for learning that don't exist today. Experts voiced the opinion that the most impactful uses of AI in education have not yet been invented. The report enumerates important strengths and weaknesses of AI, as well as the respective opportunities and barriers to applying AI to learning. Through discussions among experts about these layers, we observed new design concepts for using AI in learning. The panel also made seven recommendations for future research priorities. [The report was also created with the Center for Integrative Research in Computing and Learning Sciences (CIRCLS), the University of Pittsburgh, and NC State University.]

This report reflects on progress from over eight years of research projects in the cyberlearning community. The community involved computer scientists and learning scientists who received NSF awards to investigate the design of more equitable learning experiences with emerging technology--focusing on developing the learning theories and technologies that are likely to become important within 5-10 years. In early 2020, the Center for Innovative Research in Cyberlearning's team analyzed the portfolio of past and current projects in this community, and convened a panel of experts to reflect on important trends and issues, including artificial intelligence and learning; learning theories; research methods; out-of-school-time learning; and trends at NSF and beyond.

This project took place at four Montana middle and high schools in three rural settings located across the state. Study participants included students and teachers located in four middle and high schools. Each of the schools had large percentages of struggling readers, American Indian students, English learners, and low-income students. Comparison schools were selected from the 11 other Montana middle and high schools that had participated in the Montana Striving Readers Program, a previous literacy focused school improvement initiative. The study team worked with teams of 8-12 school administrators, instructional coaches, reading intervention teachers, and general education teachers to implement a set of quick turnaround interventions using a Plan Do Study Act (PDSA) process. These PDSA teams worked with SRI and MT-OPI to design and implement two 6-week PDSA cycles in each school year. During each PDSA cycle, team members implemented small alterations in their school practices or processes, referred to as "change ideas," which were designed to advance student literacy skills and focused on students who struggled with reading. PDSA team members were encouraged to select change ideas that would support struggling readers in particular, with the idea that these supports would help all learners access core classroom instruction and content. Teams met weekly to review data on relevant teacher practices and make changes to implementation. At the end of the cycle, the teams reviewed the data on student outcomes and developed a plan for the next PDSA cycle. School teams engaged in six PDSA cycles across the 4-year project. The study team used propensity scores to test the effect of participation in the PDSA cycles on student literacy outcomes. The propensity score is the predicted probability of participating in a treatment based on a set of potentially confounding covariates (i.e., student demographic and disability characteristics, baseline score) using logistic regression. Propensity score analysis attempts to equalize the mean values of potentially confounding observed covariates in the treatment and comparison groups, assuring that differences in outcomes between the treatment and covariate effect are not the result of differences in mean values of those covariates. Comparison schools were selected from the 11 other Montana middle and high schools that had participated in the Montana Striving Readers Program, a previous literacy focused school improvement initiative. Students in the comparison schools received literacy instruction through the Montana Striving Readers Program, without the additional supports and strategies provided through the partnership's PDSA approach. Student outcome data from the treatment schools and comparison schools were used to answer research questions regarding differences in student literacy/reading achievement. Data on the overall reading ability of middle school students were collected via fall and spring Istation Indicators of Progress Advanced Reading assessments as well as the spring administration of the Smarter Balanced Assessment Consortium English language arts assessment. The study team used data from the ACT Reading assessment conducted in the spring of grades 10 and 11 to measure the reading skills of high school students. Researchers used a two-level model with students at level one and classrooms at level two to compare outcomes in literacy and achievement of students in treatment and comparison schools. To match improvement and comparison schools, the study used nearest neighbor criteria based on school and average student characteristics, and propensity score matching to arrive at a comparison group of students from the comparison schools that was matched to students in the four improvement schools. The study did not find evidence of a statistically significant association between a school's participation in the intervention and students' literacy skills.

This study's goal was to understand the extent to which providing access to media resources focused on critical science and engineering concepts can help children living in low-income households learn. The resources examined were from the PBS KIDS multi-platform property "The Cat in the Hat Knows a Lot About That!" Researchers found that exposure to these resources had meaningful impacts on 4- to 5-year-old children's physical science knowledge and their ability to engage with science and engineering practices.

Education Development Center, Inc. (EDC) and SRI International conducted an independent study of the Public Broadcasting Service (PBS) KIDS "Play & Learn Science" app, which includes in-app and direct hands-on science explorations for children ages 3 to 6 and their parents. This study explored the impact of the PBS KIDS "Play & Learn Science" app, when used in a supportive context, on: (1) children's understanding of science concepts and use of science and engineering practices; children's use of science vocabulary; (2) child and parent-child engagement in science and engineering; and (3) parent confidence supporting their child's science learning. Findings suggest that when used in a supported context, the PBS KIDS "Play & Learn Science" app can benefit both children and their parents. [The PBS KIDS "Play & Learn Science" app was developed as part of the Corporation for Public Broadcasting (CPB) and Public Broadcasting Service (PBS) Ready To Learn Initiative, funded by the U.S. Department of Education.]

Purpose: The Bill & Melinda Gates Foundation launched the Next Generation Courseware Challenge (NGCC) in 2014 to address the limited availability of affordable, high-quality adaptive courseware to postsecondary students. SRI International was retained to evaluate the NGCC initiative as a whole and to conduct impact studies for the NGCC courseware. Methods: SRI International conducted 28 impact analyses to examine the extent to which NGCC courseware had a significant impact on postsecondary students' course achievement. Results: The average NGCC courseware effect size of +0.09 was equivalent to moving an average student at the 50th percentile in the course (as it was taught in the past) to the 54th percentile through the implementation of NGCC courseware. Implications: NGCC courseware may be an effective method of delivering postsecondary course content to postsecondary students.

This report presents the results of a first-of-its-kind national survey of parents, who were asked about the types of early science educational activities they do with their young children. It also includes a follow-up qualitative study, focused on low-income families, which sought to illuminate parents' survey responses. The purpose of the study was to provide new insights on the ways in which parents help their young children learn, as well as new information on parents' beliefs and practices related to early science learning and the use of learning media. Selected key findings: (1) Nearly all parents, regardless of income or education level, think it is important to help their young children learn, especially social skills, literacy, and mathematics; (2) Most parents say they are confident about their ability to teach their young children literacy, math, and social skills, but fewer parents are confident about science learning; and (3) Seven of 10 parents say that knowing what young children need to learn about science, and having ideas for doing science with everyday materials, would help them.

Cyberlearning researchers envision and investigate the future of learning with technology. As of summer 2017, the Cyberlearning and Future Learning Technologies (CFTL) program of the National Science Foundation (NSF) had made 279 research grant awards. In addition, several hundred other NSF research projects have cyberlearning themes. Many of these cyberlearning projects are in the exploratory stage or aim at capacity building, consistent with the goal of expanding frontiers. These projects typically do not aim to produce market-ready products or prove efficacy. Rather, the early results are often proof-of-concept designs, along with relevant theoretical insights and advances in methods. This report describes six design insights (community mapping, expressive construction, digital performance spaces, virtual peers and coaches, remote scientific labs, collaborative learning with touch interfaces) and 3 methodological advances (multimodal analysis, analytics for assessment, user- and community-centered design) emerging from cyberlearning research.

In a rigorous evaluation of ASSISTments as an online homework support conducted in the state of Maine, SRI International reported that "the intervention significantly increased student scores on an end-of-the-year standardized mathematics assessment as compared with a control group that continued with existing homework practices." (Roschelle, Feng, Murphy & Mason, 2016). Naturally, education stakeholders want to know how big the improvement was.To answer this type of question, researchers report an effect size as a simple way of quantifying the difference between two groups. We reported an effect size of g = 0.18 of a standard deviation (t(20) = 2.992, p = 0.007) based on a two level hierarchical linear model (Roschelle et al., 2016). An effect size is calculated by dividing the difference in scores between the two groups by the pooled standard deviation (Hedges, 1981). The underlying idea is that the strength of an effect depends both on the magnitude of the score difference and on how much the scores vary naturally. Consider this analogy to a commute: If it takes exactly 25 minutes to get to work every day, then a reduction to 22 minutes might mean a lot. Yet if the commute time varies between 10 minutes and 60 minutes, a reduction of the average time of 25 minutes to 22 minutes might not feel like much. Roschelle and colleagues (2016) also reported an improvement index corresponding to the effect size: "Students at the 50th percentile without the intervention would improve to the 58th percentile if they received the ASSISTments treatment." An improvement index is the expected percentile gain for the average student in the control group--the student who scored at the 50th percentile on the outcome measure--if that student had attended a school where the intervention was implemented. Reporting the effect size or an improvement index does not appear to answer educators'questions completely, however. To an educator, the implications of whether such numbers are high or low may not be unclear. In this technical report, we present alternatives for explaining the effect size, building on the guidance of Lipsey et al. (2012), a leading researcher who developed broad recommendations for effect size reporting. First, we provide additional detail on how we calculated the effect size and highlight the range of values that might be considered valid for this study. Second, we give comparisons with conventional benchmarks, a strategy that Lipsey and colleagues criticized but that still bears reporting. Third, we offer comparisons based on the recommendations of Lipsey et al. The report closes with a discussion of the challenges of interpreting effect sizes. The sidebar at the end of the report provides sample statements that educators may use to describe the study.

Technology has the potential to be a powerful tool for improving access to learning for adult learners. It can help create personalized pathways to support learning differences and open up a world of digital information and resources that form the gateway to today's job market. Despite enthusiasm from educators shown in recent studies, in general, adult education programs lag behind their K-12 and higher education counterparts in incorporating technology. This paper explores how digital opportunities to this underserved population can be accelerated; if technology efforts in K-12 and higher education can be used to create an environment that gives adult learners a more personalized learning experience; how the gap between enthusiasm and effective implementation can be bridged; and what evidence should inform adult education practices and policies. It presents four factors that contribute to effective implementation of technology. These factors are drawn from work that was undertaken to support fourteen program sites that participated in a study of digital learning technologies' effectiveness in adult basic education. [This report was produced in partnership with Mockingbird Education.]

This is the final external evaluation report prepared by SRI International for the Rural Math Excel Partnership (RMEP) project, an investing in innovation (i3) development project funded by the U.S. Department of Education. Operated by Virginia Advanced Study Strategies, Inc. (VASS), the RMEP project included six rural school districts (LEAs) in five Virginia counties as partners. The project goal was to develop and implement a model of shared responsibility among families, math teachers, and communities in rural areas to prepare students enrolled in Algebra I, Geometry, Algebra II, and Algebra Functions and Data Analysis (AFDA) courses for success in advanced high school and postsecondary STEM studies. The long term outcome was for students to leave school ready, at a minimum, to enroll in postsecondary programs focused on technician-level careers in STEM-related fields considered important to the regional rural economy. Due to low levels of implementation by some teachers in project years 1 and 2, in fall 2015 (year 3) the RMEP team focused their supports and services on a group of 24 high-implementation teachers in the seven middle and seven high schools. Key implementation and impact findings were the RMEP team completed five of the six core implementation activities meeting the standard of performance set by the evaluation team; student and family access to technology did not meet performance standards. It was difficult and time consuming to locate students and families in need of tablets and broadband access at the 14 schools and then to provide these individuals with the necessary services in their homes. Furthermore, district firewalls, teacher comfort level with technology, and registration requirements for the MARi online video platform created significant delays for the RMEP project throughout Year 2. By the end of the project's third year, however, RMEP provided technology access to all the students and their families of the 24 teachers. Willingness of individual teachers to perform their role in the model of shared responsibility varied, especially in the number of videos that teachers assigned to students and their efforts to hold Family Math Nights. Although full implementation of the model was restricted to a single semester, there was evidence that this higher level of support was beginning to have positive impacts on the teachers in terms of video assignments and student completion of these assignments. Evaluations from families and students showed that participating in RMEP-related events were useful and worth their time, though attendance was lower than expected for these events. Teachers and community members reported that organizing these events required a large time commitment and that they needed more help in identifying ways to increase attendance. SRI evaluators found that the RMEP project had no impact on students' achievement or attitudes by the end of 2015. Possible reasons included differences between the content knowledge that the Virginia Standards of Learning (SOL) exams assess and the content emphasized by the RMEP project. Evaluators were not able to limit the sample to only those students whose teachers implemented the intervention in the 2015-16 school year (the high-implementing teachers). The small sample size of students may not have been large enough to detect a very small effect. Key model components were not fully implemented until fall 2015, an insufficient time period for the intervention to have a significant impact on the targeted outcomes. Nevertheless, the RMEP project can serve as an illustrative example for other such initiatives, and suggests that similar projects should consider level of participant buy-in, anticipate and be able to troubleshoot technology access issues, and provide enough time as well as staff support for full implementation.

The study presented here addresses the question of how time spent viewing and playing with PBS KIDS educational, non-commercial media at home, in family settings, can foster positive outcomes for children and parents/caregivers. The study focused on PEG+CAT, a first-generation transmedia program designed to promote children's mathematics and approaches to learning (ATL) skills. Target mathematics skills included patterns, geometry (2-D and 3-D shapes), ordinal numbers and counting, and measurable attributes and spatial relationships. Target ATL skills included problem solving, perseverance, and self-regulation. The full report and highlights of the report's findings are also available. [For the full report "Supporting Parent-Child Experiences with PEG+CAT Early Math Concepts: Report to the CPB-PBS Ready to Learn Initiative," see ED594292.]

This report presents the findings from interviews with principals about their schools' participation in the ASSISTments efficacy study on seventh-grade mathematics. The purpose of the interviews was to explore the following areas: To what extent schools had policies or expectations regarding homework practice and completion; To what extent schools had concerns about homework completion rates; Whether the school had existing practices or norms for teachers to discuss student learning outcomes, and how often teachers met to focus on data; The extent to which principals met with teachers to discuss data; What other initiatives in the district or school could impact curriculum, instruction, assessment, or professional development and what the current priorities were in the school; Factors related to the community such as broad socio-economic status, or level of support for education, that might affect teaching and student achievement in the school.

Spurred by a demand to prepare more American workers for middle-class jobs in high technology industries, funders are investing aggressively in large-scale partnerships between community college workforce educators and employers. Despite such investments, however, few broadly usable insights have been gleaned about effective partnership and workforce program implementation. That gap limits the national capacity for identifying the best practices of these partnership efforts--identification that is needed for scaling the practices up for broader use to improve workforce education programming. To address this problem, from 2009 to 2015, researchers from SRI International's Education Division studied community college workforce educational implementation in five industries and geographic regions. That work has resulted in the Workforce Education Implementation Evaluation (WEIE), a framework for evaluating hard-to-measure aspects of the design, development, and delivery of workforce education partnerships and programs. Partnership strategies between employers and community colleges permit local regions to make adjustments in their workforce education system to ensure adequate supplies of skilled workers to meet employer demand.

This executive summary highlights findings from a larger study that is part of an interrelated series of studies included in the Ready To Learn summative evaluation being conducted by the Education Development Center, Inc.'s (EDC's) Center for Children and Technology and SRI Education's Center for Technology in Learning (EDC/SRI). The purpose of this CPB/PBS Ready to Learn Content Study was to explore, in a controlled environment, to what extent children can learn mathematics from "PEG+CAT" outside of instructional environments and relationships, how parents perceived the resources, and how well children are able to engage with these resources independently. "PEG+CAT" is a unique transmedia property that emphasizes early mathematics and problem solving. It includes video episodes, online games, and a tablet-based app, all of which allow children and their families to engage with the same characters, settings, and narratives on multiple devices, across various physical and social settings. Researchers identified and chose to include "PEG+CAT" assets focused on patterns and shapes for this study. Resources focused on counting and number relationships also were used to introduce the characters, narratives, and media experiences that would comprise the study experience. Parents reported positive impressions of the "PEG+CAT" resources, and also noted that their children engaged in "PEG+CAT" inspired play outside of the study. [For the full report, see ED594256.]

This report presents results from the "CPB-PBS Ready To Learn PEG+CAT" Content Study. "PEG+CAT" is a unique transmedia property that emphasizes early mathematics and problem solving. Each episode is structured around a specific mathematical problem the characters are trying to solve together. It includes video episodes, interstitial video, online games, and a tablet-based app, that allows children and their families to engage with the same characters, settings, and narratives on multiple devices, across various physical and social settings. The study examined how a sample of 59 four-year-old children engaged with and learned from selected sets of "PEG+CAT" videos and interactive games over five less-than-hour-long interactions. These five independent, unmediated interactions took place in a controlled learning laboratory setting. This study is one of an interrelated series of studies included in the Ready To Learn summative evaluation, being conducted by EDC's Center for Children and Technology and SRI Education's Center for Technology in Learning (EDC/SRI). EDC/SRI are serving as the external summative evaluators on behalf of the Ready To Learn partnership among the US Department of Education, the Corporation for Public Broadcasting, and PBS. [For the executive summary, see ED599706.]

Recruitment is a key challenge for researchers conducting any large school-based study. Control is needed not only over the condition participants receive, but also over how the intervention is implemented, and may include restrictions in other areas of school and classroom functioning. We report here on our experiences in recruiting participants for random assignment experiments in public primary and secondary schools, often across midsize and large school districts. Our perspective is based on over twenty current and completed randomized controlled trials (RCT) in K-12 school settings conducted by SRI International, including studies of educational technology, literacy, mathematics, science, instructional materials, teacher professional development, and student behavioral supports, following What Works Clearinghouse standards for RCTs. A list of such studies appears in the Appendix, with an abstract for each study. Considering our experience across these studies, we reflect on how we approached the recruitment problem and what worked during our efforts. As we examined our own projects as case studies of how project leaders manage the problems, patterns of successful projects emerged.

This 2-page brief presents 8 new directions drawn from the authors' report "Reflections on the Ready To Learn Initiative, 2010 to 2015" that educational media producers can take to better foster all children's school readiness and success. Since 2006, EDC and SRI have worked together on a series of efficacy studies and evaluations of Ready To Learn resources. Our research has found that digital media and technology can help families and teachers enhance the math and literacy learning of young children in low-income households. The "Reflections on the Ready To Learn Initiative" report and this 2-page brief are based on the authors' interviews with 26 children's media researchers, producers, and thought leaders. [For "Reflections on the Ready To Learn Initiative, 2010 to 2015," see ED567881.]