Search

Quasi-experimental studies are increasingly used to establish causal relationships in epidemiology and health systems research. Quasi-experimental studies offer important opportunities to increase and improve evidence on causal effects: (i) they can generate causal evidence when randomized controlled trials are impossible; (ii) they typically generate causal evidence with a high degree of external validity; (iii) they avoid the threats to internal validity that arise when participants in non-blinded experiments change their behavior in response to the experimental assignment to either intervention or control arm (such as compensatory rivalry or resentful demoralization); (iv) they are often well-suited to generate causal evidence on long-term health outcomes of an intervention, as well as non-health outcomes such as economic and social consequences; and (v) they can often generate evidence faster and at lower cost than experiments and other intervention studies.

This mixed methods study investigates why fewer than one in five impact evaluations integrates a value-for-money analysis of the development intervention being evaluated. This study distills four main insights from combined analysis of 33 semi-structured and unstructured interviews, surveys of 497 policy makers and 16 journal editors, and portfolio analyses of World Bank and worldwide impact evaluations. The study finds that low levels of training in cost data collection and analysis methods, together with a lack of standardization of the value-for-money assumptions (e.g., time horizons, discount rates, and economic or financial cost accounting) limit value-for-money integration into impact evaluations. Further eroding researchers' incentives, demand for cost evidence from the journals that publish impact evaluations is mixed. Ill-defined standards of rigor undermine editors' capacity to evaluate the quality of value-for-money analysis when it is integrated with impact evaluation evidence. Institutional funders of impact evaluations do not consistently demand that cost analysis be integrated into their funded evaluations. This study finds no evidence in support of the myth that policymakers do not demand cost evidence. Rather, it finds that researchers have few ways of knowing what kind of analysis policymakers need and when they need it. Improving the stock of impact evaluators who are cross trained in value-for-money methods, establishing standards in what constitutes rigor in costing, resolving methodological issues, and improving linkages between policymakers and researchers would lead to greater integration of value-for-money methods in impact evaluations.

Author(s): Dolan, Carrie B; BenYishay, Ariel; Grepin, Karen A; Tanner, Jeffery C; Kimmel, April D; Wheeler, David C; McCord, Gordon C | Abstract: ObjectiveTo test the impact of a nationwide Long-Lasting Insecticidal Nets [LLINs] distribution program in the Democratic Republic of Congo [DRC] on all-cause under-five child mortality exploiting subnational variation in malaria endemicity and the timing in the scale-up of the program across provinces.DesignGeospatial Impact Evaluation using a difference-in-differences approach.SettingDemocratic Republic of the Congo.Participants52,656 children sampled in the 2007 and 2013/2014 DRC Demographic and Health Surveys.InterventionsThe analysis provides plausibly causal estimates of both average treatment effects of the LLIN distribution campaign and geospatial heterogeneity in these effects based on malaria endemicity. It compares the under-five, all-cause mortality for children pre- and post-LLIN campaign relative to children in those areas that had not yet been exposed to the campaign using a difference-in-differences model and controlling for year- and province-fixed effects, and province-level trends in mortality.ResultsWe find that the campaign led to a 41% decline [3.7 percentage points, 95% CI 1.3 to 6.0] in under-5 mortality risk among children living in rural areas with malaria ecology above the sample median. Results were robust to controlling for household assets and the presence of other health aid programs. No effect was detected in children living in areas with malaria ecology below the median.ConclusionThe findings of this paper make important contributions to the evidence base for the effectiveness of large scale-national LLIN campaigns against malaria. We found that the program was effective in areas of the DRC with the highest underlying risk of malaria. Targeting bednets to areas with greatest underlying risk for malaria may help to increase the efficiency of increasingly limited malaria resources but should be balanced against other malaria control concerns.

Since 1945, over $4.9 trillion dollars of international aid has been allocated to developing countries. To date, there have been no estimates of the regional impact of this aid on the carbon cycle. We apply a geographically explicit matching method to estimate the relative impact of large-scale World Bank projects implemented between 2000 and 2010 on sequestered carbon, using a novel and publicly available data set of 61,243 World Bank project locations. Considering only carbon sequestered due to fluctuations in vegetative biomass caused by World Bank projects, we illustrate the relative impact of World Bank projects on carbon sequestration. We use this information to illustrate the geographic variation in the apparent effectiveness of environmental safeguards implemented by the World Bank. We argue that sub-national data can help to identify geographically heterogeneous impact effects, and highlight many remaining methodological challenges.