Search

Research-Practice Partnerships (RPPs) have emerged as a promising strategy for bridging the current gulf between research and practice by bringing together experts from both fields to address problems facing K-12 U.S. education. Education RPPs provide the organizational structure to facilitate sustained collaboration between researchers and practitioners to improve learning opportunities for students. However, there is limited literature about how to assess the effectiveness of these organizations. This white paper describes a framework that comprises five dimensions for assessing education RPPs. This framework includes a set of indicators for each dimension that describes where to look for evidence that an RPP is making progress on a particular dimension of effectiveness. The indicators are intended to guide the development of more specific protocols and measures that could contribute to a body of evidence related to partnership effectiveness. This framework can help RPP teams develop assessment plans that will enable them to monitor and improve their work on an ongoing basis. The authors begin by describing how they developed the dimensions and in doing so distinguish between three types of RPPs. They then describe the framework, which comprises five broad dimensions of effectiveness that represent objectives shared across the three types of RPPs. For each dimension, they articulate "indicators of progress" that partnerships might use to assess their progress in accomplishing each objective. They conclude by discussing how the framework might be used and then consider next steps for work in this area.

One purpose of education research is to develop and rigorously evaluate the effectiveness of programs for supporting students' learning and achievement. The Institute of Education Sciences has amplified that purpose (Shadish & Cook, 2009) and attempted to improve the methodological standards for conducting such work--primarily through the What Works Clearinghouse (WWC), which, since 2002, has supported an ongoing effort to synthesize research on the effectiveness of educational interventions, programs, and policies. According to its stringent, methodological standards, only "well-designed and well-implemented" randomized controlled trials and studies employing quasi-experimental designs with equating or matching are included in the WWC's 15 topical syntheses, one of which is mathematics. Despite repeated calls for increased attention to theory in program evaluation for more than 25 years, recent reviews of the literature suggest that the field has seen little change (Confrey & Stohl, 2004; Coryn, Noakes, Westine, & Schröter, 2010; Weiss, 1997). The requirement that evaluators link research designs to the theories underlying the programs they evaluate is not included in the WWC's standards for rigorous evaluations. Given the considerable differences in mathematical goals and theories of learning from which mathematics programs are designed, and the high stakes for students' mathematics learning and academic futures (as well as for the fortunes of program developers), it is important to ask whether the WWC's methodological specifications are sufficient, or whether program evaluation has once again lost (or perhaps never gained) sight of the role of theory in evaluation design and implementation (Bickman, 1987), resulting in overly-constrained and uninterpretable syntheses of otherwise methodologically strong evaluation research (Schoenfeld, 2006). The purpose of this paper is to determine whether calls for theory-based evaluation research have had an impact on the extent to which evaluators of mathematics programs attend to program theory in their design, implementation, and reporting of studies. For each report that met stringent methodological standards (i.e., WWC evidence standards), the authors asked: (1) What type of program theory was articulated (none, sub-theoretical, or theoretical; Lipsey et al., 1985); (2) What was the quality of evaluators' articulation of program theory (entirely implicit, drawn from a limited number of resources such as developer or publisher descriptions, or drawn from multiple resources and situated in the research literature); (3) How was the articulated program theory used in the evaluation; and (4) To what extent does attention to program theory vary by characteristics of the program and evaluator(s), such as the nature of the program's mathematics learning goals and instruction (i.e., back-to-basics, typical, inquiry-based, or blended); type of publication (e.g., peer-reviewed journal or technical report); funding source (e.g., Federal/state agency or publisher/developer); and timing of analysis (i.e., primary or secondary analysis)? Modeled after the WWC's process, every report was examined and coded independently by two individuals employing the same coding scheme, and research questions listed here. The findings of evaluation studies guide the decisions of policy makers at every level, including the adoption of both curriculum materials and intervention programs. These decisions are consequential for students' mathematics learning and academic futures. It is therefore crucial that evaluators "get it right" when assessing the effectiveness of such programs. This analysis indicates that WWC's methodological specifications are inadequate because they overlook understanding and using theory in evaluation design and implementation. In general, evaluation research of mathematics programs needs to improve in its attention to and use of program theory. Tables and figures are appended.

In this paper, the authors describe a case of measuring implementation fidelity within an evaluation study of Math Recovery (MR), a pullout tutoring program aimed at increasing the mathematics achievement of low-performing first graders, thereby closing the school-entry achievement gap by enabling them to achieve at the level of their higher-performing peers in the regular mathematics classroom. Two research questions guided the conduct and analysis of the larger study: 1) Does participation in MR raise the mathematics achievement of low performing first-grade students? 2) If so, do participating students maintain the gains made in first grade through the end of second grade? The analysis reported in this paper follows from a third question: 3) To what extent does fidelity of implementation influence the effectiveness of MR? The two-year evaluation of Math Recovery was conducted in 20 elementary schools (five urban, ten suburban and five rural), representing five districts in two states. The findings suggest it is possible to create a reliable instrument to measure implementation fidelity for differentiated interventions--an endeavor that has, heretofore, been largely avoided in evaluations of educational interventions. Many potentially high-quality interventions are unscripted, instead relying on teacher knowledge and professional development, requiring considerable differentiation by implementers. As individuals work to rigorously evaluate such programs, they need to develop reliable fidelity measures that are both feasible and true to program components, so that evaluators can adequately link measures of treatment integrity to outcomes, to more accurately determine the relative strength of interventions (Cordray & Pion, 2006). This paper outlines the development and use of one such measure as a case of how such fidelity instruments might be developed and used in the future. Critical aspects of the process included: (1) the identification of the core implementation components of the intervention (Fixsen et al., 2005); (2) close work with program developers to operationalize those components; (3) training of coders in both the program itself and the coding schemes/process; and (4) collaborating with the coding team to further refine operationalizations and coding decisions, to strike a balance of feasibility and adherence to program components.

The authors' goal was to evaluate the potential of Math Recovery (MR), a pullout, one-to-one tutoring program that has been designed to increase mathematics achievement among low-performing first graders, thereby closing the school-entry achievement gap and enabling participants to achieve at the level of their higher-performing peers in the regular mathematics classroom. Specifically, the research questions were as follows: (1) Does participation in MR raise the mathematics achievement of low performing first-grade students?; (2) If so, do participating students maintain the gains made in first grade through the end of second grade? The two-year evaluation of Math Recovery was conducted in 20 elementary schools (five urban, ten suburban and five rural), representing five districts in two states. Students were selected for participation at the start of first grade based on their performance on MR's screening interview and follow-up assessment interview. The authors recruited 18 teachers to receive training and participate as MR tutors from the participating districts--all of whom had at least two years of classroom teaching experience. The findings of this study have theoretical, practical, and policy significance. Practically, the positive causal effect of MR tutoring demonstrates that programs that are diagnostic rather than scripted in nature can overcome fidelity concerns and have an impact on student early mathematics performance. Theoretically, the findings indicate that investing in tutors' knowledge of student reasoning and pedagogical content knowledge can pay off in terms of improvement in student's mathematical learning, particularly if tutors use carefully designed tools such as the MR Learning and Instructional Frameworks that codify and schematize this knowledge. With regard to policy, the authors' finding that the MR program can reduce some of the pre-K mathematics achievement gap provides an initial indication that the cost of the program per student might be justified, although further work is needed to understand why initial gains made by participants appear to diminish after tutoring ends. It is possible that the forms of arithmetic reasoning that MR develops needs to be further supported in the regular classroom to see the full benefit of this form of tutoring. Longitudinal studies that track MR students and their initially higher performing peers until the end of elementary school are needed to address this question adequately.

In this paper we document the importance of institutional context in both constraining and enabling the work of mathematics teachers. We build from our current and ongoing collaborative efforts with middle-grades mathematics teachers to provide an analytic approach and resulting analysis that clarifies the critical role of institutional context in teacher development. The analysis delineates the communities of practice whose enterprises are concerned with how mathematics is taught and learned in the district and the importance of their interconnections (cf. Wenger, 1998). Our approach can best be viewed as a tool designed to support transformative educational change as iterative processes of continual improvement in mathematics education. [For complete proceedings, see ED489538.]

It is important to develop instructional sequences that build on students' current understandings and support shifts in their current ways of thinking. As part of the pilot work for a project on mathematics teaching, classroom performance assessments were conducted to obtain baseline data on students' current statistical understandings. The assessments were conducted in three sessions of a seventh-grade class. The assessment task was designed to provide information about students' current understandings of the mean and graphical representations of data because these ideas were the focus of a statistics chapter students previously studied. Students worked in small groups on the three performance tasks, each of which is described in detail. The analysis shows that students typically viewed the mean as a procedure that was to be used to summarize a group of numbers regardless of the task situation. Data analysis for these students meant "doing something with the numbers," an idea grounded in their previous mathematics experiences. Students' conversations about graphical representations highlight the procedures for constructing graphs with no attention to what the graphs signify and how that relates to the task situation. To help students develop a sense of data analysis as more than just "doing something with numbers," it is necessary to create tasks that are relevant to middle school students. An appendix contains a list of 69 sources for additional information. (Contains 8 figures and 11 references.) (SLD)