The District-Wide Effectiveness of the Achieve3000 Program: A Quasi-Experimental Study

Drawing on state reading achievement data provided by a California school district, we analyze the achievement impacts of Achieve3000, an intervention designed to offer K-12 students expanded opportunities to acquire literacy skills. Achieve3000 is a personalized, technology-adaptive program to supplement regular instruction in English language arts, science, and social studies. We apply propensity score matching methods to match 1,761 participating Achieve3000 students to 945 similar students to estimate the effects of the Achieve3000 intervention. Applying a doubly-robust regression model, our results suggest that Achieve3000 students outperformed comparison students by a mean effect size of 0.06. Further, two-stage least squares regression models reveal that treatment students' scores increased statistically significantly by 0.001 standard deviations for both each activity completed and for each additional hour of Achieve3000 participation. Despite some problems of implementation, our quasi-experimental evidence of the positive impacts of Achieve3000 suggests a possibly promising way to overcome COVID-19 learning losses through computer-based, one-on-one tutoring at scale.