Evaluation of the Rural Math Excel Partnership Project Final Report

This is the final external evaluation report prepared by SRI International for the Rural Math Excel Partnership (RMEP) project, an investing in innovation (i3) development project funded by the U.S. Department of Education. Operated by Virginia Advanced Study Strategies, Inc. (VASS), the RMEP project included six rural school districts (LEAs) in five Virginia counties as partners. The project goal was to develop and implement a model of shared responsibility among families, math teachers, and communities in rural areas to prepare students enrolled in Algebra I, Geometry, Algebra II, and Algebra Functions and Data Analysis (AFDA) courses for success in advanced high school and postsecondary STEM studies. The long term outcome was for students to leave school ready, at a minimum, to enroll in postsecondary programs focused on technician-level careers in STEM-related fields considered important to the regional rural economy. Due to low levels of implementation by some teachers in project years 1 and 2, in fall 2015 (year 3) the RMEP team focused their supports and services on a group of 24 high-implementation teachers in the seven middle and seven high schools. Key implementation and impact findings were the RMEP team completed five of the six core implementation activities meeting the standard of performance set by the evaluation team; student and family access to technology did not meet performance standards. It was difficult and time consuming to locate students and families in need of tablets and broadband access at the 14 schools and then to provide these individuals with the necessary services in their homes. Furthermore, district firewalls, teacher comfort level with technology, and registration requirements for the MARi online video platform created significant delays for the RMEP project throughout Year 2. By the end of the project's third year, however, RMEP provided technology access to all the students and their families of the 24 teachers. Willingness of individual teachers to perform their role in the model of shared responsibility varied, especially in the number of videos that teachers assigned to students and their efforts to hold Family Math Nights. Although full implementation of the model was restricted to a single semester, there was evidence that this higher level of support was beginning to have positive impacts on the teachers in terms of video assignments and student completion of these assignments. Evaluations from families and students showed that participating in RMEP-related events were useful and worth their time, though attendance was lower than expected for these events. Teachers and community members reported that organizing these events required a large time commitment and that they needed more help in identifying ways to increase attendance. SRI evaluators found that the RMEP project had no impact on students' achievement or attitudes by the end of 2015. Possible reasons included differences between the content knowledge that the Virginia Standards of Learning (SOL) exams assess and the content emphasized by the RMEP project. Evaluators were not able to limit the sample to only those students whose teachers implemented the intervention in the 2015-16 school year (the high-implementing teachers). The small sample size of students may not have been large enough to detect a very small effect. Key model components were not fully implemented until fall 2015, an insufficient time period for the intervention to have a significant impact on the targeted outcomes. Nevertheless, the RMEP project can serve as an illustrative example for other such initiatives, and suggests that similar projects should consider level of participant buy-in, anticipate and be able to troubleshoot technology access issues, and provide enough time as well as staff support for full implementation.