Search

This paper is a practical guide for researchers and practitioners who want to understand spillover effects in program evaluation. The paper defines spillover effects and discusses why it is important to measure them. It explains how to design a field experiment to measure the average effects of the treatment on eligible and ineligible subjects for the program in the presence of spillover effects. In addition, the paper discusses the use of nonexperimental methods for estimating spillover effects when the experimental design is not a viable option. Evaluations that account for spillover effects should be designed such that they explain the cause of these effects and whom they affect. Such an evaluation design is necessary to avoid inappropriate policy recommendations and neglecting important mechanisms through which the program operates.

Policy makers bemoan the lack of research findings to guide urgent decisions, whereas researchers' professional code puts rigor first. This article argues that provisional assessments, produced early in the research cycle, can bridge the gap. Numerous case studies point to the importance of early interaction with policy makers and the delivery of brief, policy-focused papers; but preliminary analyses may be flawed and so increase the chances of a wrong decision. This article demonstrates analytically that a preliminary assessment, supported by the offer of more refined research, provides an option that is superior, on average, to the current practice of submitting a final report at the end of the research cycle. Where practical implementation is concerned, it calls for donor-funded subsidies to promote the use of provisional assessments and for a rapid, independent, professional review process to ensure their quality. While the research-policy exchange in developing countries is a complex, context-specific phenomenon, the proposal offered here holds out some promise of improving decisions in the public sphere under a wide range of circumstances.

The number of countries that regularly participate in international large-scale assessments has increased sharply over the past 15 years, with the share of countries participating in the Programme for International Student Assessment growing from one-fifth of countries in 2000 to over one-third of countries in 2015. What accounts for this increase? This paper explores the evidence for three broad explanations: globalization of assessments, increasing technical capacity for conducting assessments, and increased demand for the microeconomic and macroeconomic data from these assessments. Data were compiled from more than 200 countries for this analysis, for six time periods between 2000 and 2015, yielding more than 1,200 observations. The data cover each country’s participation in each of six cycles of PISA as it relates to the country’s level of economic development, region, prior experience with assessment, and OECD membership. The results indicate that the odds of participation in PISA are markedly higher for OECD member countries, countries in the Europe and Central Asia region, high- and upper-middle-income countries, and countries with previous national and international assessment experience; the paper also finds that regional assessment experience is unrelated to PISA participation.

In order to maximize effectiveness, every national education system needs a comprehensive and coherent approach to student assessment. The systems approach for better education results (SABER) framework for student assessment identifies four key elements in this regard: classroom assessment, examinations, national large scale assessment, and international large scale assessment. Armenia has undoubtedly made significant progress in all of these areas since gaining independence from the Soviet Union in 1991. The purpose of this paper is to describe Armenia’s experiences, share the lessons learned along the way, and point out the challenges that countries face in trying to ensure that their assessment systems remain fit for purpose in a changing environment.

This paper develops a structural approach for modeling how respondents answer survey questions and uses it to estimate the proportion of respondents who are reticent in answering corruption questions, as well as the extent to which reticent behavior biases down conventional estimates of corruption. The context is a common two-step survey question, first inquiring whether a government official visited a business, and then asking about bribery if a visit was acknowledged. Reticence is a concern for both steps, since denying a visit sidesteps the bribe question. This paper considers two alternative models of how reticence affects responses to two-step questions, with differing assumptions on how reticence affects the first question about visits. Maximum-likelihood estimates are obtained for seven countries using data on interactions with tax officials. Different models work best in different countries, but cross-country comparisons are still valid because both models use the same structural parameters. On average, 40 percent of corruption questions are answered reticently, with much variation across countries. A statistic reflecting how much standard measures underestimate the proportion of all respondents who had a bribe interaction is developed. The downward bias in standard measures is highly statistically significant in all countries, varying from 12 percent in Nigeria to 90 percent in Turkey. The source of bias varies widely across countries, between denying a visit and denying a bribe after admitting a visit.

Despite the importance of trade facilitation as an area of trade and development policy, there have been very few impact evaluations of specific trade facilitation reforms. This paper offers an evaluation of in-house clearance, a reform that allows qualified firms in Serbia to clear customs from within their own warehouse rather than at the customs office. The pooled synthetic control method applied here offers a novel solution to many of the empirical challenges that frustrate efforts to evaluate trade facilitation reforms. The method is used to estimate causal impacts on trade outcomes for 21 firms that adopted in-house clearance for import shipments. The program compressed the distribution of clearance times for adopting firms, but the estimated effects on median clearance times, inspection rates, and import value were not statistically significant. Tests for heterogeneous program impact do not indicate that the program affected adopting firms differently. Overall, the results suggest that the most evident benefit of the program for participating firms is reduced uncertainty about clearance times.

A recent survey of rigorous impact evaluations of programs to help small and medium-size firms to formalize indicates that the programs do not seem to work for most informal firms. One of the few exceptions finds large effects of a tax simplification program in Brazil called SIMPLES on firms' formalization rates and performance indicators. Using the same data set but a different identification strategy, another study concludes that the program had limited effect on formalization rates. The aim of this paper is twofold. First, it revisits the two studies to reconcile their conflicting conclusions. Second, it investigates the validity of the identification strategy of both studies. The findings suggest that the conflicting results between the two studies are caused by the dates each used to identify when the program was put into effect. A robustness check indicates that data heaping and seasonality around November cast doubts on the identification strategy used in both studies to estimate the effect of this particular program.

The paper estimates the effect of product standards on firms’ export decisions using two novel datasets. The first covers all exporting firms in 42 developing countries. The second covers pesticide standards for 243 agricultural and food products in 63 importing countries over 2006–12. The analysis shows that product standards significantly affect foreign market access. More restrictive standards in the importing country, relative to the exporting country, lower firms’ probability of exporting as well as their export values and quantities. The relative restrictiveness of standards also deters exporting firms from entering new markets and leads to higher exit rates from those markets. Moreover, firm characteristics mediate the effect of product standards on firms’ export decisions. Smaller exporters are more negatively affected in their market entry and exit decisions by the relative stringency of standards than larger exporters. Positive network effects of exporters from the same country may help reduce the burden of importing countries’ standards on firms’ decisions to enter new markets.

For more than 20 years, the development community has claimed that monitoring and evaluation helps projects achieve their objectives. This study uses data from 1,300 World Bank projects evaluated between 2008 and 2014 to investigate this suggested link between the quality of monitoring and evaluation and project performance. The propensity score matching results indicate that the quality of monitoring and evaluation is significantly and positively associated with project outcome as institutionally measured at the World Bank. This positive relationship holds when controlling for project manager identity, and is robust to various specification choices. Through a systematic text analysis of the narrative produced by the Independent Evaluation Group to justify its monitoring and evaluation quality rating, the study shows that there are common markers of good quality monitoring and evaluation, such as: clear institutional setup and division of labor around monitoring and evaluation activities; simple monitoring and evaluation framework that is well aligned with clients' existing monitoring and evaluation systems; good integration with operational tasks; and a system that can generate regular and timely reporting, and that is used during and after lending.

We complement the empirical evidence on the sustainability of weight loss achieved through cash rewards and, for the first time, rigorously examine the potential of cash rewards to prevent weight cycling. In a three period randomized controlled trial, about 700 obese persons were first assigned to two treatment groups, which were promised cash contingent on the achievement of an individually assigned target weight, and to a control group. Successful participants were subsequently allocated to two treatment groups offered cash rewards for confirming the previously achieved target weight and to a control group. This is the first experiment of this kind that finds effects of weight loss rewards up to 18 months after they were removed. Additional rewards only significantly improve the sustainability of weight loss while they are in place. Die empirische Analyse untersucht zum einen die Nachhaltigkeit einer Gewichtsreduktion, die mittels Geldanreizen erzielt wurde, und zum anderen die Wirksamkeit von Geldanreizen zur Verhinderung des Jo-Jo-Effekts nach einer erfolgreichen Gewichtsreduktion. In einem drei-phasigen Feldexperiment wurden ca. 700 fettleibige Teilnehmer zunächst zufällig auf zwei Anreizgruppen und einer Kontrollgruppe verteilt. Unabhängig von der Gruppenzugehörigkeit sollten ein individuelles Gewichtsreduktionziel erreicht werden, wobei nur Mitglieder der Anreizgruppen eine Geldprämie für das Erreichen des Zieles erhalten konnten. Erfolgreiche Teilnehmer wurden danach erneut zufällig einer Kontrollgruppe und zwei Anreizgruppen zugewiesen, wobei das Halten des Zielgewichts nur für letztere durch eine Geldprämie inzentiviert wurde. Anders als in früheren Experimenten, können Effekt der Abnehmprämie selbst 18 Monate nach ihrem Auslaufen nachgewiesen werden. Monetäre Anreize zur Verhinderung des Jo-Jo-Effekt wirken hingegen nicht über ihr Auslaufen hinaus.

The World Bank Independent Evaluation Group (IEG) works to improve development results through excellence in evaluation. A key part of this mandate focuses on developing the Bank’s client countries’ capacities in monitoring and evaluation. To this end, IEG developed the International Program for Development Evaluation Training (IPDET) in 2001, and this executive training program has been implemented since then in partnership with Carleton University in Ottawa, Canada. IPDET is managed by Carleton University but has received substantial in-kind (through technical experts) and financial support over the years from IEG. IPDET was conceived to offer a one-of-a-kind learning program for filling a gap in development evaluation training. However, there is broad recognition that the landscape is changing, with increasing numbers of organizations providing monitoring and evaluation (ME) training in some form, an evolving mix of formal graduate degree and certificate programs preparing evaluators, innovations in learning supported by new technologies, and the growing engagement of local networks and evaluation associations in evaluation capacity development. In this context, IEG has commissioned a rapid review of the current landscape for ME training to develop an understanding of the current context in which IPDET operates. Finally, training programs focused specifically on development evaluation were of particular interest for this review. The distinction between ‘evaluation’ and ‘development evaluation’ is arguably an important one, with increasing attention focused on what kinds of peer groups and curriculum are needed to effectively build the ME skills and knowledge relevant for a developing country context.

This paper examines the spatial organization of jobs in Kampala, the capital city of Uganda, and applies the Lucas and Rossi-Hansberg (2002) model to explain the observed patterns in terms of the agglomeration forces and the commuting costs of workers. The paper suggests that: (i) Economic activities are concentrated in the downtown -- beyond which employment is spatially dispersed. (ii) Geographically weighted regressions identify five potential subcenters in 2011; however, none of these contribute significantly to employment. When explaining the variation in employment density across localities in Kampala, the research highlights that (i) density falls by 23.5 percent per kilometer increase in distance from the nearest potential subcenter; (ii) an increase in local production externalities of 10 percent increases density by 3.7 percent; and (iii) production externalities in Kampala's potential subcenters are extremely weak to have any significant impact even on nearby tracts.

This paper introduces a simple framework for understanding the dimensions and determinants of health worker performance based on the idea that there can be three different gaps affecting performance: a knowledge gap, the knowledge-capacity gap and the capacity-performance gap. The paper argues that performance is determined by a combination of competence, capacity and effort, and that any of these elements may lead to poor performance, and applies this framework to the measurement of health worker performance in Tanzania. Whilst discussing and highlighting key findings related to the assessment of health worker performance in Tanzania, the overarching objective of the paper is to offer a systematic way to analyze health worker performance through primary data collection and analysis to benefit researchers and countries beyond Tanzania.

This paper examines China’s rural minimum living standard guarantee (dibao) program, one of the largest minimum income cash transfer schemes in the world. Using household survey data matched with published administrative data, the paper describes the dibao program, estimates the program’s impact on poverty, and carries out targeting analysis. The analysis finds that the program provides sufficient income to poor beneficiaries but does not substantially reduce the overall level of poverty, in part because the number of beneficiaries is small relative to the number of poor. Conventional targeting analysis reveals rather large inclusionary and exclusionary targeting errors; propensity score targeting analysis yields smaller but still large targeting errors. Simulations of possible reforms to the dibao program indicate that expanding coverage can potentially yield greater poverty reduction than increasing transfer amounts. In addition, replacing locally diverse dibao lines with a nationally uniform dibao threshold could in theory reduce poverty. The potential gains in poverty reduction, however, depend on the effectiveness of targeting.

This paper evaluates the performance of alternative estimators of the gravity equation when zero trade flows result from economically-based data-generating processes with heteroscedastic residuals and potentially-omitted variables. In a standard Monte Carlo analysis, the paper finds that this combination can create seriously biased estimates in gravity models with frequencies of zero frequently observed in real-world data, and that Poisson Pseudo-Maximum-Likelihood models can be important in solving this problem. Standard threshold–Tobit estimators perform well in a Tobit-based data-generating process only if the analysis deals with the heteroscedasticity problem. When the data are generated by a Heckman sample selection model, the Zero-Inflated Poisson model appears to have the lowest bias. When the data are generated by a Helpman, Melitz, and Rubinstein-type model with heterogeneous firms, a Zero-Inflated Poisson estimator including firm numbers appears to provide the best results. Testing on real-world data for total trade throws up additional puzzles with truncated Poisson Pseudo-Maximum-Likelihood and Poisson Pseudo-Maximum-Likelihood estimators being very similar, and Zero-Inflated Poisson and truncated Poisson Pseudo-Maximum-Likelihood identical. Repeating the Monte Carlo analysis taking into account the high frequency of very small predicted trade flows in real-world data reconciles these findings and leads to specific recommendations for estimators.

In order to advance our understanding of why Results Based Financing (RBF) works or not, it is crucial that evaluations not only measure the impact of such an arrangement on final outcomes (population health), but also assess the changes in variables in the causal chain between intervention and final outcomes. Health worker performance is a key variable in this chain; it is only by changing health workers’ behaviors—their performance—that RBF can influence health outcomes. Careful assessment of impacts on health worker performance is therefore a natural and important element of any RBF impact evaluations. This paper discusses various approaches to evaluating the impact of RBF on health worker performance. The first part is a discussion of possible ways in which RBF may affect health worker behavior, based on economic theory and empirical evidence. The second part is a more practical discussion of how health worker performance and other relevant variables can be measured and how impacts can be estimated. This is followed by some practical steps that can be taken to ensure that the evaluation leads to actions that can be implemented; a brief conclusion completes the paper.

Participatory community development programs are designed to match government investments with local needs. In Morocco, where issues of inequality and poverty are high on the national agenda, a community development program, the National Initiative for Human Development, targeted high-poverty areas for additional investments. This paper examines whether, in addition to reducing poverty, such programs can also promote human development, specifically early childhood development. Early childhood development forms a critical foundation for later human development and plays a key role in the intergenerational transmission of socioeconomic status. Using panel data on communities just above and below the cutoff for National Initiative for Human Development inclusion in rural areas, regression discontinuity and fixed effect models are applied to identify the impact of the program on economic outcomes and early childhood development. Although the analysis finds some transitory impacts of the program on economic outcomes, it finds no impacts on early childhood development. Reducing inequality and promoting human development through early childhood development is likely to require specific, targeted, and sustained initiatives.

About a third of development projects fail to achieve satisfactory outcomes, according to agencies' independent evaluation units. To a large extent, these outcomes appear to be baked into projects at their inception due to inadequate project design or relevance. This prompts questions about the diffusion of project design concepts: To what extent are better-designed or better-performing projects more likely to be emulated? Do factors of bureaucratic or political attractiveness -- such as ease of set-up and rapidity of disbursement -- play a role? To address these questions, this paper explores the use of methods from network science. It constructs a network graph of the relationship among the components of all World Bank investment projects initiated from 1996 to 2014, based on the semantic similarity of the component descriptions. It uses the network to assess the characteristics of projects that are more 'prolific' in the sense of having closely related followers, and as tool for visualizing diffusion of design concepts. This illustrative exercise defines a measures of project 'influence' on subsequent projects and tests simple, nonexclusive hypotheses about the determinants of influence. It finds no significant impact of project outcome or quality of entry (as independently rated) on 'influence.' Nor does ease of project preparation (as proxied by time from concept note to effectiveness) have any significant effect. However, very small projects (less than $10 million) have markedly lower 'influence' on average. This finding may have implications for the usefulness of small projects as pilots for subsequent scale up.

This Discussion Paper presents the approach, findings, and recommendations from a desk review of the qualitative research conducted within Results-Based Financing programmes (RBF) under the Health Results Innovations Trust Fund (HRITF). The review included 17 studies conducted in Benin, Burundi, Cameroon, DRC, Ethiopia, Haiti, Kenya, Kyrgyzstan, Nigeria, Rwanda, Tajikistan, Tanzania, Zambia, and Zimbabwe. The studies reveal a body of high quality work that is consistent with the conceptual framework of RBF schemes, supported by political will, resources, and research capacity. Strengthening the added value of qualitative inquiry in on-going and future qualitative studies may be enabled by small shifts in thinking and practice, in line with a qualitative research paradigm. First, in order to better ground research in an existing country and system specific context, some interrogation of constructs and posited relationships in the existing conceptual framework for intervention/evaluation may be required. Second, to enable more in-depth and richer data that documents working practices and relations under RBF schemes, training of local researchers should place stronger emphasis on entry to the field, gaining trust, building rapport, and sustaining a dialogue with key informants. Third, smaller, more intensive and focused studies targeting fewer sites and smaller samples - but addressing a wider range of methods and informants within the health system - are likely to yield richer data that can support the understanding of how health workers and managers are responding to schemes, and what impact schemes have on service volumes and outputs.

The last few years have seen a growing commitment worldwide to universal health coverage (UHC). Yet there is a lack of clarity on how to measure progress towards UHC. This paper proposes a ‘mashup’ index that captures both aspects of UHC: that everyone—irrespective of their ability-to-pay—gets the health services they need; and that nobody suffers undue financial hardship as a result of receiving care. Service coverage is broken down into prevention and treatment, and financial protection into impoverishment and catastrophic spending; nationally representative household survey data are used to adjust population averages to capture inequalities between the poor and better off; nonlinear tradeoffs are allowed between and within the two dimensions of the UHC index; and all indicators are expressed such that scores run from 0 to 100, and higher scores are better. In a sample of 24 countries for which there are detailed information on UHC-inspired reforms, a cluster of high-performing countries emerges with UHC scores of between 79 and 84 (Brazil, Colombia, Costa Rica, Mexico and South Africa) and a cluster of low-performing countries emerges with UHC scores in the range 35–57 (Ethiopia, Guatemala, India, Indonesia and Vietnam). Countries have mostly improved their UHC scores between the earliest and latest years for which there are data—by about 5 points on average; however, the improvement has come from increases in receipt of key health interventions, not from reductions in the incidence of out-of-pocket payments on welfare.

Over the past two decades, community-based approaches to project delivery have become a popular means for governments and development agencies to improve the alignment of projects with the needs of rural communities and increase the participation of villagers in project design and implementation. This paper briefly summarizes the results of an impact evaluation of the National Solidarity Program, a community-driven development program in Afghanistan that created democratically elected community development councils and funded small-scale development projects. Using a randomized controlled trial across 500 villages, the evaluation finds that the National Solidarity Program had a positive effect on access to drinking water and electricity, acceptance of democratic processes, perceptions of economic wellbeing, and attitudes toward women. Effects on perceptions of local and national government performance and material economic outcomes were, however, more limited or short-lived.

The adoption of new clinical practice patterns by medical care providers is often challenging, even when the patterns are believed to be efficacious and profitable. This paper uses a randomized field experiment to examine the effects of temporary financial incentives paid to medical care clinics for the initiation of prenatal care in the first trimester of pregnancy. The rate of early initiation of prenatal care was 34 percent higher in the treatment group than in the control group while the incentives were being paid, and this effect persisted at least 15 months and likely 24 months or more after the incentives ended. These results are consistent with a model where the incentives enable providers to address the fixed costs of overcoming organizational inertia in innovation, and suggest that temporary incentives may be effective at motivating improvements in long-run provider performance at a substantially lower cost than permanent incentives.

This paper examines the micro and macro correlates of aid project outcomes in a sample of 3,821 World Bank projects and 1,342 Asian Development Bank projects. Project outcomes vary much more within countries than between countries: country-level characteristics explain only 10–25 percent of project outcomes. Among macro variables, country growth and the policy environment are significantly positively correlated with project outcomes. Among micro variables, shorter project duration and the presence of additional financing are significantly correlated with better project outcomes. In addition, the track record of the project manager in delivering successful projects is highly significantly correlated with project outcomes. There are few significant differences between the two institutions in the relationship between these variables and project outcomes.

This paper broadly examines the development process of Korea’s health care system toward the achievement of Universal Health Coverage. Korea implemented a series of health care reforms after a rapid expansion of population coverage to improve efficiency and equity in financing and delivery of health care. The authors also investigate changes in the governance structure of Korea’s national health Insurance, which is now represented by two agencies: National Health Insurance Service (NHIS) and Health Insurance Review and Assessment Service (HIRA). Health insurance agencies have improved the accountability and transparency of the health insurance system, thanks to the ICT-based centralized claim review and assessment. Lessons and challenges from Korea’s experiences and achievements on the road to UHC could provide valuable policy implications to low- and middle-income countries.