Search

Background: Given increasing concerns about the relevance of research to policy and practice, there is growing interest in assessing and enhancing the external validity of randomized trials: determining how useful a given randomized trial is for informing a policy question for a specific target population. Objectives: This article highlights recent advances in assessing and enhancing external validity, with a focus on the data needed to make ex post statistical adjustments to enhance the applicability of experimental findings to populations potentially different from their study sample. Research design: We use a case study to illustrate how to generalize treatment effect estimates from a randomized trial sample to a target population, in particular comparing the sample of children in a randomized trial of a supplemental program for Head Start centers (the Research-Based, Developmentally Informed study) to the national population of children eligible for Head Start, as represented in the Head Start Impact Study. Results: For this case study, common data elements between the trial sample and population were limited, making reliable generalization from the trial sample to the population challenging. Conclusions: To answer important questions about external validity, more publicly available data are needed. In addition, future studies should make an effort to collect measures similar to those in other data sets. Measure comparability between population data sets and randomized trials that use samples of convenience will greatly enhance the range of research and policy relevant questions that can be answered.

Background: Large-scale randomized experiments are important for determining how policy interventions change average outcomes. Researchers have begun developing methods to improve the external validity of these experiments. One new approach is a balanced sampling method for site selection, which does not require random sampling and takes into account the practicalities of site recruitment including high nonresponse. Method: The goal of balanced sampling is to develop a strategic sample selection plan that results in a sample that is compositionally similar to a well-defined inference population. To do so, a population frame is created and then divided into strata, which “focuses” recruiters on specific subpopulations. Units within these strata are then ranked, thus identifying “replacements” similar to sites that can be recruited when the ideal site refuses to participate in the experiment. Result: In this article, we consider how a balanced sample strategic site selection method might be implemented in a welfare policy evaluation. Conclusion: We find that simply developing a population frame can be challenging, with three possible and reasonable options arising in the welfare policy arena. Using relevant study-specific contextual variables, we craft a recruitment plan that considers nonresponse.

Background: Variations in local context bedevil the assessment of external validity: the ability to generalize about effects of treatments. For evaluation, the challenges of assessing external validity are intimately tied to the translation and spread of evidence-based interventions. This makes external validity a question for decision makers, who need to determine whether to endorse, fund, or adopt interventions that were found to be effective and how to ensure high quality once they spread. Objective: To present the rationale for using theory to assess external validity and the value of more systematic interaction of theory and practice. Methods: We review advances in external validity, program theory, practitioner expertise, and local adaptation. Examples are provided for program theory, its adaptation to diverse contexts, and generalizing to contexts that have not yet been studied. The often critical role of practitioner experience is illustrated in these examples. Work is described that the Robert Wood Johnson Foundation is supporting to study treatment variation and context more systematically. Results: Researchers and developers generally see a limited range of contexts in which the intervention is implemented. Individual practitioners see a different and often a wider range of contexts, albeit not a systematic sample. Organized and taken together, however, practitioner experiences can inform external validity by challenging the developers and researchers to consider a wider range of contexts. Researchers have developed a variety of ways to adapt interventions in light of such challenges. Conclusions: In systematic programs of inquiry, as opposed to individual studies, the problems of context can be better addressed. Evaluators have advocated an interaction of theory and practice for many years, but the process can be made more systematic and useful. Systematic interaction can set priorities for assessment of external validity by examining the prevalence and importance of context features and treatment variations. Practitioner interaction with researchers and developers can assist in sharpening program theory, reducing uncertainty about treatment variations that are consistent or inconsistent with the theory, inductively ruling out the ones that are harmful or irrelevant, and helping set priorities for more rigorous study of context and treatment variation.

Background: Government and private funders increasingly require social service providers to adopt program models deemed “evidence based,” particularly as defined by evidence-based program registries, such as What Works Clearinghouse and National Registry of Evidence-Based Programs and Practices. These registries summarize the evidence about programs’ effectiveness, giving near-exclusive priority to evidence from experimental-design evaluations. The registries’ goal is to aid decision making about program replication, but critics suspect the emphasis on evidence from experimental-design evaluations, while ensuring strong internal validity, may inadvertently undermine that goal, which requires strong external validity as well. Objective: The objective of this study is to determine the extent to which the registries’ reports provide information about context-specific program implementation factors that affect program outcomes and would thus support decision making about program replication and adaptation. Method: A research-derived rubric was used to rate the extent of context-specific reporting in the population of seven major registries’ evidence summaries ( N = 55) for youth development programs. Findings: Nearly all (91%) of the reports provide context-specific information about program participants, but far fewer provide context-specific information about implementation fidelity and other variations in program implementation (55%), the program’s environment (37%), costs (27%), quality assurance measures (22%), implementing agencies (19%), or staff (15%). Conclusion: Evidence-based program registries provide insufficient information to guide context-sensitive decision making about program replication and adaptation. Registries should supplement their evidence base with nonexperimental evaluations and revise their methodological screens and synthesis-writing protocols to prioritize reporting—by both evaluators and the registries themselves—of context-specific implementation factors that affect program outcomes.

Background: Systematic reviews sponsored by federal departments or agencies play an increasingly important role in disseminating information about evidence-based programs and have become a trusted source of information for administrators and practitioners seeking evidence-based programs to implement. These users vary in their knowledge of evaluation methods and their ability to interpret systematic review findings. They must consider factors beyond program effectiveness when selecting an intervention, such as the relevance of the intervention to their target population, community context, and service delivery system; readiness for replication and scale-up; and the ability of their service delivery system or agency to implement the intervention. Objective: To support user decisions about adopting evidence-based practices, this article discusses current systematic review practices and alternative approaches to synthesizing and presenting findings and providing information. Method: We reviewed the publicly available information on review methodology and findings for eight federally funded systematic reviews in the labor, education, early childhood, mental health/substance abuse, family support, and criminal justice topic areas. Conclusion: The eight federally sponsored evidence reviews we examined all provide information that can help users to interpret findings on evidence of effectiveness and to make adoption decisions. However, they are uneven in the amount, accessibility, and consistency of information they report. For all eight reviews, there is room for improvement in supporting users’ adoption decisions through more detailed, accessible, and consistent information in these areas.

Background: Systematic reviews—which identify, assess, and summarize existing research—are usually designed to determine whether research shows that an intervention has evidence of effectiveness, rather than whether an intervention will work under different circumstances. The reviews typically focus on the internal validity of the research and do not consistently incorporate information on external validity into their conclusions. Objectives: In this article, we focus on how systematic reviews address external validity. Methods: We conducted a brief scan of 19 systematic reviews and a more in-depth examination of information presented in a systematic review of home visiting research. Results: We found that many reviews do not provide information on generalizability, such as statistical representativeness, but focus on factors likely to increase heterogeneity (e.g., numbers of studies or settings) and report on context. The latter may help users decide whether the research characteristics—such as sample demographics or settings—are similar to their own. However, we found that differences in reporting, such as which variables are included and how they are measured, make it difficult to summarize across studies or make basic determinations of sample characteristics, such as whether the majority of a sample was unemployed or married. Conclusion: Evaluation research and systematic reviews would benefit from reporting guidelines for external validity to ensure that key information is reported across studies.

Background: For much of the last 40 years, the evaluation profession has been consumed in a battle over internal validity. Today, that battle has been decided. Random assignment, while still far from universal in practice, is almost universally acknowledged as the preferred method for impact evaluation. It is time for the profession to shift its attention to the remaining major flaws in the “standard model” of evaluation: (i) external validity and (ii) the high cost and low hit rate of experimental evaluations as currently practiced. Recommendations: To raise the profession’s attention to external validity, the author recommends some simple, easy steps to be taken in every evaluation. The author makes two recommendations to increase the number of interventions found to be effective within existing resources: First, a two-stage evaluation strategy in which a cheap, streamlined Stage 1 evaluation is followed by a more intensive Stage 2 evaluation only for those interventions found to be effective in a Stage 1 trial and, second, use of random assignment to guide the myriad program management decisions that must be made in the course of routine program operations. This article is not intended as a solution to these issues: It is intended to stimulate the evaluation community to take these issues more seriously and to develop innovative solutions.

This study investigates the relationship of panel accretion to panel attrition and examines its impact on threats to external validity in adolescent substance use research. Panel accretion involves the addition of new participants to a panel design after initial baseline data has been obtained. Accretion occurs in longitudinal studies in which data is routinely gathered on all participants, as is often the case in prevention or epidemiological research on adolescent substance use. Accretion, attrition, and panel findings for three, 2-year panel samples in grades 7 through 9 are examined. The results indicate that accretion and attrition samples have an isomorphic relationship to one another on a range of dependent measures when drawn from the same population. The results are discussed in terms of their implications for adolescent substance use research, the examination of validity threats due to panel bias, and understanding the ecological validity of research findings.

School-based prevention trials typically face large variations in school composition and levels ofpreintervention behavior. Such variations may inhibit efforts to maximize internal and external validity. This article presents a method for using school-level attributes in sample description and random assignment of schools to conditions. The utility of the archival data was examined using multivanate canonical techniques. The authors found that a small set of attributes could efficiently predict the observed school-level variations in smoking and other drug use. It is recommended that these attributes be used by other school-based studies to enhance experimen talfield studies.

Donald Campbell and his colleagues have elaborated an approach to research based on the elimination of threats to four validity types: internal, external, construct, and statistical conclusion. The central thesis of this article is that the process of eliminating validity threats depends fundamentally on no-difference findings, a fact that, unfor tunately, has not been made explicit by researchers. The implications of this neglect are explored using examples from a number of different substantive areas such as psychology, health, and medicine. Finally, the intrinsic role of no-difference findings is described in the context of all four validity types, and suggestions for improving the process are offered.

The estimate-and-subtract methodfor eliminating threats to validity is described. Yeaton and Sechrest (1986, 1987b) argue that "no-difference"findings are fundamental to the process of eliminating threats to validity. The estimate-and-subtract method, however, eliminates a threat to validity without reliance on no-difference findings. This article argues that eliminating a threat to validity using the estimate-and-subtract method often is preferable to eliminating a threat to validity using no-difference findings. The article also shows how to improve the use of no-difference findings in those instances where their use is appropriate.

Single-case experimental designs have received little attention in the program evaluation literature. This article provides an overview of single-case methodology, with a specific focus on the A-B-A and multiple baseline designs. The potential utility of these designs for evaluating social programs is discussed, and several examples are provided. Internal validity, statistical conclusion validity, external validity, construct validity, and cost-effec tiveness are considered. Single-case designs offer essentially equivalent protection against threats to experimental validity compared to traditional randomized group designs. It appears that these designs offer, at a minimum, viable methodological alternatives in situations where traditional randomized group designs are infeasible.

The article argues that the sctentific credibility and policymaking utility of policy evaluation research is threatened by inappropriate inference strategies. These inference strategies are a consequence of a system of incentives and a methodology that encourage vague, ambiguous, or nonspecification of primary inferences and extended or unwar ranted tangential inferences. The result is policy research designed more to promote the enterprise of policy research than to assist in the useful application of research in policymaking settings. These points are illustrated through an analysis of the inference strategy pursued in the Police Foundation's Kansas City Preventive Patrol Experiment. One solution to the problem of inappropriate inferencing is for policy researchers to adopt the practice ofspecifying an illustrative range of potential primary inferences at the initial stages of analysis.