Search

When a researcher tests an educational program, product, or policy in a randomized controlled trial and detects a significant effect on an outcome, the intervention is usually classified as something that “works.” When expected effects are not found, there is seldom an orderly and transparent analysis of plausible reasons why. Accumulating and learning from possible failure mechanisms is not standard practice in education research, and it is not common to design interventions with causes of failure in mind. This chapter develops Boruch and Ruby’s proposition that the education sciences would benefit from a systematic approach to the study of failure. We review and taxonomize recent reports of large-scale randomized controlled trials in K–12 schooling that yielded at least one null or negative major outcome, including the nature of the event and reasons (if provided) for why it occurred. Our purpose is to introduce a broad framework for thinking about educational interventions that do not produce expected effects and seed a cumulative knowledge base on when, how, and why interventions do not reach expectations. The reasons why an individual intervention fails to elicit an outcome are not straightforward, but themes emerge when researchers’ reports are synthesized.

This article summarizes key points made in a session at the American Society of Criminology meeting in Philadelphia in November 2017, entitled “The replication issue in science and its relevance for criminology”, organized by Friedrich Losel and Robert F. Boruch. In turn, this session was inspired by Friedrich Losel’s (2018) article in this journal, based on his 2015 Joan McCord Award Lecture of the Academy of Experimental Criminology. In the present article, Friedrich Losel introduces the topic of replication in criminology and summarizes his main arguments. Then, six leading criminologists present short papers on this topic. Robert F. Boruch points out the instability in social systems, David P. Farrington argues that systematic reviews are important, and Denise C. Gottfredson calls attention to the heterogeneity in conclusions across different studies. Lorraine Mazerolle reviews attempts to replicate experiments in procedural justice, Lawrence W. Sherman draws attention to enthusiasm bias in criminal justice experiments, and David Weisburd discusses the logic of null hypothesis significance testing and multi-center trials. Finally, some developments since November 2017 in research on replication in criminology are discussed.

This Campbell systematic review examines the impact of Teach For America (TFA) on learning outcomes. Four studies were included in the review. Studies had to be a quantitative evaluation of the effects of TFA on K–12 student academic outcomes. Studies also had to use a research design which: 1. allowed valid causal inferences about TFA's effects, 2. targeted participants K–12 students taught by TFA corps members or TFA alumni in the USA, 3. compared TFA corps members to novice teachers, or compared TFA alumni with veteran teachers, and 4. reported at least one academic student outcome in math, ELA, or science domains. A total of 919 citations were retrieved on TFA, of which 24 studies were eligible for review. However, when the research design and study quality along with types of TFA corps members and non‐TFA teachers compared were reviewed, the evidence base for estimating the effects of TFA on student academic outcomes was reduced to just four studies. There is no significant effect on reading from teaching by TFA corps members in their first or second year of teaching elementary‐grade students (PreK – grade 5) compared to non‐TFA teachers who are also in their first or second year of teaching elementary‐grade students. There is a small positive effect for early elementary‐grade students (PreK to grade 2) in reading but not in math. However, given the small evidence base, these findings should be treated with caution. Plain language summary There are too few well‐designed studies to know the effects of Teach For America on Math, English Language Arts, and Science outcomes of K–12 students in the USA Teach For America (TFA) is an alternate route teacher preparation program that aims to address the decades‐long shortage of effective teachers in many rural and urban public schools for kindergarten through 12th grade (K–12), that serve the highest proportions of high‐poverty students across the USA. This review finds that there are very few studies – just four – which reliably measure the effects of TFA on learning outcomes, so that no firm conclusions may be drawn. What is this review about? This systematic review evaluated the impact of TFA prepared teachers (corps members) relative to novice teachers and alumni relative to veteran teachers on K‐12 student outcomes in math, English Language Arts (ELA), and science. What is the aim of this review? This Campbell systematic review examines the impact of Teach For America on learning outcomes. Four studies were included in the review. What are the main findings of this review? Studies had to be a quantitative evaluation of the effects of TFA on K‐12 student academic outcomes. Studies also had to use a research design which: 1. allowed valid causal inferences about TFA's effects, 2. targeted participants K–12 students taught by TFA corps members or TFA alumni in the USA, 3. compared TFA corps members to novice teachers, or compared TFA alumni with veteran teachers, and 4. reported at least one academic student outcome in math, ELA, or science domains. A total of 919 citations were retrieved on TFA, of which 24 studies were eligible for review. However, when the research design and study quality along with types of TFA corps members and non‐TFA teachers compared were reviewed, the evidence base for estimating the effects of TFA on student academic outcomes was reduced to just four studies. There is no significant effect on reading from teaching by TFA corps members in their first or second year of teaching elementary‐grade students (PreK – grade 5) compared to non‐TFA teachers who are also in their first or second year of teaching elementary‐grade students. There is a small positive effect for early elementary‐grade students (PreK to grade 2) in reading but not in math. However, given the small evidence base, these findings should be treated with caution. What do the findings of this review mean? TFA is the most evaluated program of its kind. Multiple quasi‐experimental and experimental studies have been conducted on its effectiveness in improving student outcomes. However, this systematic review found that only a small number of these studies (1) met the evidence review standards and (2) compared the same type of TFA corps members and non‐TFA teachers. So it is not possible to draw firm policy conclusions. Future research can contribute to this evidence base by designing, implementing, and reporting experiments and quasi‐experiments to meet objective extant evidence standards and by comparing the same types of TFA and non‐TFA teachers so that effect sizes can be included in a future systematic review and meta‐analysis. How up‐to‐date is this review? The review authors searched for studies published up to January 2015. This Campbell systematic review was published in June 2018.

This article was written in celebration of the Tenth Anniversary issue of the Journal of Experimental Criminology. It focuses on efforts to mount high-quality randomized controlled trials, mainly in criminology and education. There are occasional excursions into medical, welfare, and manpower research. The aim is to uncover similarities and differences in the way controlled trials are handled, and is in the spirit of comparative sociology and history of science

Postmortems and autopsies, at the individual and hospital unit levels, are disciplined approaches to learning from medical failures. “Safety factors” that engineers use in designing structures and systems are based on past failures or trials and experiments to find points of failure. The applied social sciences, including education sciences, labor economics, and criminology, have less clarity about failure. While a bridge collapse is usually plain and spectacular, failures of education innovations or attempts at crime control are often quieter, not spectacular, and often occur for no transparent reasons. The applied social sciences lack disciplined, well-developed, and explicit approaches to anticipating the failure to meet expectations in testing the effectiveness of programs, analyzing the failures, and building a cumulative knowledge base on the phenomenon. Our fields can, for instance, identify “what works” pretty well from randomized controlled trials. However, little serious attention has been dedicated to understanding “why” and “how” a particular intervention failed to meet expectations in well-executed randomized controlled trials. This essay discusses a variety of research initiatives that are designed to better understand failure, especially in controlled trials. Keywords: failure; postmortems; social sciences; labor economics; criminology; research

This paper reports on the progress of users through 16 Coursera courses taught by University of Pennsylvania faculty for the first time between June 2012 and July 2013. Using descriptive analyses, this study advances knowledge by considering two definitions of massive open online course (MOOC) users (registrants and starters), comparing two approaches to measuring student progress through a MOOC course (sequential versus user driven), and examining several measures of MOOC outcomes and milestones. The patterns of user progression found in this study may not describe current or future patterns given the continued evolution of MOOCs. Nonetheless, the findings provide a baseline for future studies.

Due to evidence linking education and development, funding has been invested in interventions relevant to getting youth into school and keeping them there. This article reports on a systematic review of impact studies of these school enrollment interventions. Reports were identified through electronic searches of bibliographic databases and other methods. To be eligible, studies (1) assessed impact on primary or secondary school enrollment outcomes; (2) used a rigorous design; (3) were conducted in a low- or middle-income nation; (4) included at least one quantifiable measure of enrollment or related outcomes; (5) were available before December 2009; and (6) included data on participants post-1990. A coding instrument extracted data on study characteristics from each report. Standardized mean difference effect sizes were computed for the first effect reported. The sample includes 73 evaluations. The average effect size was positive across all outcomes. However, the results varied. Studies that focused on building new schools and other infrastructure interventions reported the largest average effects.

Background Informed consent procedures in cluster randomized trials (CRTs) are considerably more complicated than in individually randomized trials. In a CRT, the units of randomization, intervention, and observation may differ in a single trial; there can be multiple levels of participants (individual and cluster level); consent may be required separately for intervention and data collection; and there may be practical constraints to seeking informed consent, for example, due to cluster-level interventions or the sheer size of clusters. Purpose We aimed to document consent practices at individual and cluster levels, assess the adequacy of reporting consent in trial publications, and assess associations with two trial characteristics that may influence consent requirements in CRTs: presence or absence of study interventions and presence or absence of data collection procedures at individual and cluster levels. Methods We reviewed a random sample of 300 CRTs published during 2000–2008. We sent survey questionnaires to 285 unique authors of these trials to gather detailed information about consent procedures used in each trial. Results In all, 182 authors (64%) responded. Overall, 93% (95% confidence interval (CI): 88.8%−96.6%) indicated that participant consent had been sought for some aspects of the study. Consent was less frequently sought for a study intervention (70% of respondents) than for data collection (88%). More than half of the respondents (52%) indicated that consent had been sought at both cluster and individual levels. There was strong evidence for under-reporting of consent in trial publications: only 63% of all trial publications reported that informed consent had been sought for some aspect of the study. The odds ratios (ORs) summarizing the association of the two trial characteristics with cluster-level participant consent were weak (OR = 1.17, p = 0.70 for presence of cluster-level study intervention and OR = 1.54, p = 0.29 for data collection); on the other hand, the ORs summarizing the associations with individual-level consent were strong (OR = 6.2, p < 0.0001 for presence of individual-level intervention and OR = 14.7, p < 0.0001 for data collection). Limitations In all, 36% of authors did not respond to the survey; to the extent that consent practices in their trials were different than in respondents’ trials, our results may be biased. Conclusions There is a need for improvements in research practices in CRTs as well as their reporting. There may be a lack of clarity about consent requirements at the cluster level in particular. With the publication of the Ottawa Statement on the Ethical Design and Conduct of Cluster Randomized Trials, researchers and research ethics committees now have access to comprehensive ethics guidelines specific to CRTs.

Background Cluster randomized trials (CRTs) complicate the interpretation of standard research ethics guidelines for several reasons. For one, the units of allocation, intervention, and observation often may differ within a single trial. In the absence of tailored and internationally accepted ethics guidelines for CRTs, researchers and research ethics committees have no common standard by which to judge ethically appropriate practices in CRTs. Moreover, lack of familiarity with and consideration of the unique features of the CRT design by research ethics committees may cause difficulties in the research ethics review process, and amplify problems such as variability in the requirements and decisions reached by different research ethics committees. Purpose We aimed to characterize research ethics review of CRTs, examine investigator experiences with the ethics review process, and assess the need for ethics guidelines for CRTs. Methods An electronic search strategy implemented in MEDLINE was used to identify and randomly sample 300 CRTs published in English language journals from 2000 to 2008. A web-based survey with closed- and open-ended questions was administered to corresponding authors in a series of six contacts. Results The survey response rate was 64%. Among 182 of 285 eligible respondents, 91% indicated that they had sought research ethics approval for the identified CRT, although only 70% respondents reported research ethics approval in the published article. Nearly one-third (31%) indicated that they have had to meet with ethics committees to explain aspects of their trials, nearly half (46%) experienced variability in the ethics review process in multijurisdictional trials, and 38% experienced negative impacts of the ethics review process on their trials, including delays in trial initiation (28%), increased costs (10%), compromised ability to recruit participants (16%), and compromised methodological quality (9%). Most respondents (74%; 95% confidence interval (CI): 67%–80%) agreed or strongly agreed that there is a need to develop ethics guidelines for CRTs, and (70%; 95% CI: 63%–77%) that ethics committees could be better informed about distinct ethical issues surrounding CRTs. Limitations Thirty-six percent of authors did not respond to the survey. Due to the absence of comparable results from a representative sample of authors of individually randomized trials, it is unclear to what extent the reported challenges result from the CRT design. Conclusions CRT investigators are experiencing challenges in the research ethics review of their trials, including excessive delays, variability in process and outcome, and imposed requirements that can have negative consequences for study conduct. Investigators identified a clear need for ethics guidelines for CRTs and education of research ethics committees about distinct ethical issues in CRTs.

Svensk översättning av "Estimating the Effects of Interventions in Multiple Sites and Settings: Place-based Randomized Trials"

Objective Cluster randomised trial (CRT) investigators face challenges in seeking informed consent from individual patients (cluster members). This study examined associations between reporting of patient consent in healthcare CRTs and characteristics of these trials. Study design Consent practices and study characteristics were abstracted from a random sample of 160 CRTs performed in primary or hospital care settings that were published from 2000 to 2008. Multivariable logistic regression was used to examine associations between reporting of patient consent and methodological characteristics, as well as publication features such as date and journal of publication. Results 82 (53.8%) of 160 studies reported obtaining informed consent from individual patients. Reporting of patient consent was independently and positively associated with: smaller cluster size, the evaluation of experimental interventions targeted at patients, data collection from individual patients, publication later than 2004 and publication in higher-impact journals. Conclusions Reporting of consent practices in published CRTs should be improved. Consent practices in published CRTs appear to be related to the type of interventions under study, as well as journal impact and trends in research ethics practices. These findings will inform best practices in trial conduct and ethics review, remediation of errors in consent practices and ethics review and the development of regulatory guidance for CRTs.

Replication research is essential for the advancement of any scientific field. In this paper, we argue that prevention science will be better positioned to help improve public health if (a) more replications are conducted; (b) those replications are systematic, thoughtful, and conducted with full knowledge of the trials that have preceded them; and (c) state-of-the art techniques are used to summarize the body of evidence on the effects of the interventions. Under real-world demands it is often not feasible to wait for multiple replications to accumulate before making decisions about intervention adoption. To help individuals and agencies make better decisions about intervention utility, we outline strategies that can be used to help understand the likely direction, size, and range of intervention effects as suggested by the current knowledge base. We also suggest structural changes that could increase the amount and quality of replication research, such as the provision of incentives and a more vigorous pursuit of prospective research registers. Finally, we discuss methods for integrating replications into the roll-out of a program and suggest that strong partnerships with local decision makers are a key component of success in replication research. Our hope is that this paper can highlight the importance of replication and stimulate more discussion of the important elements of the replication process. We are confident that, armed with more and better replications and state-of-the-art review methods, prevention science will be in a better position to positively impact public health.

In the United States, the voting right of persons who live in long-term care facilities has not received adequate attention. Residents can lose one of their most cherished rights and electoral credibility can be corrupted. One solution to the problems of voting in long-term care is mobile polling, a process whereby election officials bring the ballot to residents of long-term care facilities, provide voters assistance when needed, and register voters as well. This study compared mobile polling to voting as usual in selected nursing homes in the State of Vermont during the U.S. 2008 general election. Results show that among election officials and nursing homes willing to try mobile polling, it is feasible and generally well accepted by long-term care staff, residents, and election officials; reduces concerns of voter fraud and manipulation; and enhances residents' dignity and rights. The challenges of mobile polling include motivating and training election officials to perform it, providing resour...

This article aims to update readers on different ways to arrange one's thinking about conventional null hypotheses in randomized trials. It covers basic criticism of conventional hypotheses and, beyond this, covers relevant developments in methodological, organizational, and science policy arenas. This article includes coverage of new ways to frame null hypotheses, new technical resources, standards for registering trials and reporting on them, cumulating results, common mistakes, and post-trial analysis of null results. The paper includes ideas for research and development on each topic.

Thomas Jefferson recognized the value of reason and scientific experimentation in the eighteenth century. This chapter extends the idea in contemporary ways to standards that may be used to judge the ethical propriety of randomized trials and the dependability of evidence on effects of social interventions.

Since “the year of the MOOC” in 2012, the effectiveness of massive open online courses (MOOCs) has been widely debated. Some argue that MOOCs are not an effective mode of instructional delivery because of low completion rates. In the interest of developing alternative indicators of performance this study draws from recent efforts to measure engagement in social media, as well as from research on indicators of student engagement in traditional college courses. Using data from 16 Coursera MOOCs offered by the University of Pennsylvania we calculate standardized access rates for lectures and assessments. While these indicators have clear limitations as measures of educational progress they offer a different, more nuanced understanding of the level and nature of users’ engagement with a MOOC. This paper shows that a very small share of users takes up available opportunities to access course content but notes that the standardized access rates compare favorably with those for social media sites and with response rates to large-scale direct mail marketing programs. For MOOC providers and platform managers, indicators like the ones developed in this study may be a useful first step in monitoring the extent to which different types and combinations of activities may be providing better opportunities for learning.

Randomized controlled experiments are designed so as to make fair comparisons about which of two or more interventions works better, and to produce the least equivocal evidence possible about the relative effects of the interventions. The studies of impact may also be called randomized trials or social experiments depending on the particular focus. This article covers these kinds of studies in diverse academic disciplines and service delivery areas, different countries, and various geopolitical jurisdictions.

This article is a revision of the previous edition article by R. Boruch, volume 18, pp. 12070–12072, © 2001, Elsevier Ltd.

% ROBERT BORUCH Phrases such as "evidence-based policy ' are seductive and are used promiscuously in some quarters. Here, we take the phrase and its intent seriously, as others have. In the international sector, for instance, the Campbell Collaboration was created to generate systematic reviews of dependable evidence on the effects of policies, programs, and practices. This is in the arenas of crime and justice, education, and social services. The Cochrane Collaboration, Campbells older sibling, was created to achieve a similar aim in health care. Their reviews of evidence are