Search

Background: The federal government’s emphasis on supporting the implementation of evidence-based programs has fueled a need to conduct and assess rigorous evaluations of programs. Through partnerships with researchers, policy makers, and practitioners, evidence reviews—projects that identify, assess, and summarize existing research in a given area—play an important role in supporting the quality of these evaluations and how the findings are used. These reviews encourage the use of sound scientific principles to identify, select, and implement evidence-based programs. The goals and standards of each review determine its conclusions about whether a given evaluation is of high quality or a program is effective. It can be difficult for decision makers to synthesize the body of evidence when faced with results from multiple program evaluations. Sample: This study examined 14 federally funded evidence reviews to identify commonalities and differences in their assessments of evidence of effectiveness. Findings: There were both similarities and significant differences across the reviews. In general, the evidence reviews agreed on the broad critical elements to consider when assessing evaluation quality, such as research design, low attrition, and baseline equivalence. The similarities suggest that, despite differences in topic and the availability of existing research, reviews typically favor evaluations that limit potential bias in their estimates of program effects. However, the way in which some of the elements were assessed, such as what constituted acceptable amounts of attrition, differed. Further, and more substantially, the reviews showed greater variation in how they conceptualized “effectiveness.”

Background: In 2002, the U.S. Department of Education’s Institute of Education Sciences (IES) established the What Works Clearinghouse (WWC) at the confluence of a push to improve education research quality, a shift toward evidence-based decision-making, and an expansion of systematic reviews. In addition to providing decision makers with evidence to inform their choices, a systematic review sets expectations regarding study quality and execution for research on program efficacy. In this article, we examine education research through the filter of a long running systematic review to assess research quality over time and the role of the systematic review in producing evidence. Method: Using the WWC’s database of reviewed studies, we explored the relationships between study characteristics and dispositions as well as the differences by topic area and changes over time. Conclusion: Through its design standards, the WWC has defined its requirements for a study to be considered causal evidence, which may have been one of the factors contributing to observed improvement in the quality of education research over the past 15 years. The levels and rates of studies meeting standards have been increasing over the life of the WWC. Additionally, the number and proportion of studies excluded due to ineligible design are decreasing. Thus, less research is ineligible due to design issues, and more eligible studies are meeting standards. As IES continues to conduct and fund studies designed to meet standards, and more decisions are directly tied to evidence, the body of rigorous education research may continue to grow.

Background: Systematic reviews sponsored by federal departments or agencies play an increasingly important role in disseminating information about evidence-based programs and have become a trusted source of information for administrators and practitioners seeking evidence-based programs to implement. These users vary in their knowledge of evaluation methods and their ability to interpret systematic review findings. They must consider factors beyond program effectiveness when selecting an intervention, such as the relevance of the intervention to their target population, community context, and service delivery system; readiness for replication and scale-up; and the ability of their service delivery system or agency to implement the intervention. Objective: To support user decisions about adopting evidence-based practices, this article discusses current systematic review practices and alternative approaches to synthesizing and presenting findings and providing information. Method: We reviewed the publicly available information on review methodology and findings for eight federally funded systematic reviews in the labor, education, early childhood, mental health/substance abuse, family support, and criminal justice topic areas. Conclusion: The eight federally sponsored evidence reviews we examined all provide information that can help users to interpret findings on evidence of effectiveness and to make adoption decisions. However, they are uneven in the amount, accessibility, and consistency of information they report. For all eight reviews, there is room for improvement in supporting users’ adoption decisions through more detailed, accessible, and consistent information in these areas.

Read the final report from a large-scale, rigorous study examining how four math curricula affect achievement across two years—from first through second grades. The curricula were (1) Investigations in Number, Data, and Space; (2) Math Expressions; (3) Saxon Math; and (4) Scott Foresman-Addison Wesley Mathematics.

This appendix provides the details that underlie the analyses reported in the evaluation brief, “After Two Years, Three Elementary Math Curricula Outperform a Fourth.†The details are organized in six sections: Study Curricula and Design (Section A), Data Collection (Section B), Construction of the Analysis File (Section C), Curriculum Effects on Student Math Achievement (Section D), Curriculum Implementation (Section E), and Effects of Switching Curricula (Section F).

Read the final report from a large-scale, rigorous study examining how four math curricula affect achievement across two years—from first through second grades. The curricula were (1) Investigations in Number, Data, and Space; (2) Math Expressions; (3) Saxon Math; and (4) Scott Foresman-Addison Wesley Mathematics.

By offering $20,000 per teacher, seven school districts piloting a transfer-incentive strategy, known as the Talent Transfer Initiative (TTI), filled 90 percent of their targeted vacancies in hard-to-staff schools with some of the districts' highest-performing teachers. A new study highlights the implementation experience and intermediate impacts of TTI, which is intended to expand disadvantaged students' access to the most effective teachers. Previous research conducted by Mathematica shows that, on average, low-income middle school students are significantly less likely to have access to the highest-performing teachers.

This report provides an executive summary of the findings from the evaluation, including an outcomes evaluation and a process analysis, for which we gathered information from participant focus groups, program staff, other NIH staff, and administrative data to understand how the program has been implemented, especially in recent years.

For over a decade, the National Institutes of Health has been using repayment of education loans as an incentive to attract and retain first-rate biomedical scientists in its labs and clinics. This report from an evaluation of the NIH Intramural Loan Repayment Programs uses new survey data on recruitment of biomedical researchers and career paths of program participants and nonparticipants to assess the role of these policies in recruiting and retention outcomes. The findings suggest that eligibility for loan repayment is associated with higher recruitment success and that participation in loan repayment programs is associated with longer stays at the NIH and in research careers more generally.

Much of the research examining the question of how federal financial aid affects decisions to enroll in college has focused on the behavior of students in the relatively narrow range immediately following high school graduation, leaving unanswered the question of how changes in the availability of aid affect the behavior of older students. This analysis examines the question of how changes in the means-tested federal Pell grant program affects enrollment decisions of potential students in their twenties and thirties. Our results indicate sizable effects of the introduction of the Pell grant program on college enrollment decisions for older students.

Low-income students and students whose parents have not attended college typically are less likely than middle- and upper-income students to complete high school and attend college, and are thus less likely to reap the benefits of attending college. In 1965, Congress established the Talent Search Program to provide information on the high school courses students should take to prepare for college and financial aid, as well as to orient students to different types of colleges and the application process. This report presents findings from Mathematica’s study of the effectiveness of the Talent Search program in Florida, Indiana, and Texas. Based on administrative data and a quasi-experimental design to create matched comparison groups, researchers found that Talent Search participants were more likely than comparison students to apply for federal financial aid and enroll in public secondary institutions. The findings suggest that assisting low-income students to overcome information barriers may be effective in helping them achieve their aspirations to attend college. Practical information—such as guidance on how to complete financial aid and admission applications and what a college campus looks and feels like—may have been one of the key services that Talent Search projects delivered.

The National Institutes of Health (NIH) spends millions of dollars each year to provide scholarships to an elite group of highly talented undergraduate science students from disadvantaged backgrounds to encourage them to pursue careers in biomedical research. Mathematica’s evaluation found that the NIH Undergraduate Scholarship Program appears to be operating as intended and that scholars are achieving career milestones at high rates. They are making steady progress toward the goal of becoming biomedical researchers, and doing so at substantially higher rates than applicants selected as finalists but not awarded scholarships.

Upward Bound is one of the largest and longest running federal programs designed to help economically disadvantaged students go to college.

In recent years, researchers have argued that the shortage of highly qualified teachers in poor school districts may have less to do with attracting new teachers than with retaining them. To provide the scientific evidence that will support sound decisions about teacher induction, Mathematica is examining whether high-intensity teacher induction programs lead to higher retention rates and other positive teacher and student outcomes. This report lays out the study design in detail, documenting the process for building the sample of districts, schools, and students, and for conducting random assignment. In addition, it describes the data collection and analysis plans.