Search

The practice of prospectively registering the details of intervention studies in a public database or registry is gaining momentum across disciplines as a strategy for increasing the transparency, credibility, and accessibility of study findings. In this article, we consider five registries that may be relevant for registration of intervention studies in the field of prevention science: ClinicalTrials.gov, the American Economic Association Registry of Randomized Controlled Trials (AEA RCT Registry), the Open Science Framework Preregistration (OSF Preregistration), the Registry for International Development Impact Evaluations (RIDIE), and the Registry of Efficacy and Effectiveness Studies (REES). We examine the five registries in terms of substantive focus, study designs, and contents of registry entries. We consider two paths forward for prospective registration of intervention studies in the field of prevention science: Path A: register all studies in ClinicalTrials.gov and Path B: allow individual researchers to select the registry with the "best fit." Lastly, we consider how the field might begin to establish norms around registration. [This article was published in "Prevention Science" (EJ1341830).]

As a result of the evidence-based decision-making movement, the number of randomized trials evaluating educational programs and curricula has increased dramatically over the past 20 years. Policy makers and practitioners are encouraged to use the results of these trials to inform their decision making in schools and school districts. At the same time, however, little is known about the schools taking part in these randomized trials, both regarding how and why they were recruited and how they compare to populations in need of research. In this article, we report on a study of 37 cluster randomized trials funded by the Institute of Education Sciences between 2011 and 2015. Principal investigators of these grants were interviewed regarding the recruitment process and practices. Additionally, data on the schools included in 34 of these studies were analyzed to determine the general demographics of schools included in funded research, as well as how these samples compare to important policy relevant populations. We show that the types of schools included in research differ in a variety of ways from these populations. Large schools from large school districts in urban areas were overrepresented, whereas schools from small school districts in rural areas and towns are underrepresented. The article concludes with a discussion of how recruitment practices might be improved in order to meet the goals of the evidence-based decision-making movement.

Cluster randomized trials (CRTs) are commonly used to evaluate educational interventions, particularly their effectiveness. Recently there has been greater emphasis on using these trials to explore cost-effectiveness. However, methods for establishing the power of cluster randomized cost-effectiveness trials (CRCETs) are limited. This study develops power computation formulas and statistical software to help researchers plan two- and three-level CRCETs. We illustrate the application of our formulas and software for the designs of CRCETs and discuss the influence of sample size, nesting effects, covariates, and the covariance between cost and effectiveness measures on the statistical power of cost-effectiveness estimates.

Over the past 15 years, we have seen an increase in the use of cluster randomized trials (CRTs) to test the efficacy of educational interventions. These studies are often designed with the goal of determining whether a program works, or answering the what works question. Recently, the goals of these studies expanded to include for whom and under what conditions an intervention is effective. In this study, we examine the capacity of a set of CRTs to provide rigorous evidence about for whom and under what conditions an intervention is effective. The findings suggest that studies are more likely to be designed with the capacity to detect potentially meaningful individual-level moderator effects, for example, gender, than cluster-level moderator effects, for example, school type.

With the increasing demand for evidence-based research on teacher effectiveness and improving student achievement, more impact studies are being conducted to examine the effectiveness of professional development (PD) interventions. Cluster randomized trials (CRTs) are often carried out to assess PD interventions that aim to improve both teacher and student outcomes. Due to the different design parameters (i.e., intraclass correlation and R[superscript 2]) and benchmark effect sizes associated with the student and teacher outcomes, two power analyses are necessary for planning CRTs that aim to detect both teacher and student effects in one study. These two power analyses are often conducted separately without considering how design choices to power the study to detect student effects may affect design choices to power the study to detect teacher effects and vice versa. In this study, we consider strategies to maximize the efficiency of the study design when both student and teacher effects are of primary interest.

The practice of prospectively registering the details of intervention studies in a public database or registry is gaining momentum across disciplines as a strategy for increasing the transparency, credibility, and accessibility of study findings. In this article, we consider five registries that may be relevant for registration of intervention studies in the field of prevention science: ClinicalTrials.gov, the American Economic Association Registry of Randomized Controlled Trials (AEA RCT Registry), the Open Science Framework Preregistration (OSF Preregistration), the Registry for International Development Impact Evaluations (RIDIE), and the Registry of Efficacy and Effectiveness Studies (REES). We examine the five registries in terms of substantive focus, study designs, and contents of registry entries. We consider two paths forward for prospective registration of intervention studies in the field of prevention science: Path A: register all studies in ClinicalTrials.gov and Path B: allow individual researchers to select the registry with the "best fit." Lastly, we consider how the field might begin to establish norms around registration.

This article describes the effects of a student-teacher-scientist partnership program on high school students' science outcomes. The design was a cluster-randomized trial with an analytic sample of 36 teachers randomly assigned to teach a photosynthesis- and respiration-focused module including student-designed investigations supported by online mentoring from scientists. Forty-nine teachers were randomly assigned to teach photosynthesis and respiration to students as they normally would. The core finding was that implementing the module in combination with the online mentoring resulted in significant effects on student achievement (effect size = 0.284) and attitudes toward scientists (effect size = 0.280). The study sample was highly representative, demographically, to the US population of high schools. Future research should replicate the current study design, with the current and other related interventions, while examining component-specific costs and efficacy. Related studies should also examine the relationship between the quality of scientist-student dialogue and student outcomes.

Calls for improving transparency across the social science are increasing. One strategy that is gaining momentum in the quest to increase transparency is the practice of preregistering a study and analysis plan prior to conducting a study. In this article, we examine the potential of preregistration as a strategy for increasing transparency in education studies. We review existing registries within the social sciences and provide a rationale for why we need a registry specifically geared towards education studies. Finally, we introduce the Registry of Efficacy and Effectiveness Studies (REES), developed by the Society for Research on Educational Effectiveness (SREE). The goal of REES is to increase transparency for studies seeking to draw causal conclusions within the education research and evaluation community.

Calls for improving transparency across the social science are increasing. One strategy that is gaining momentum in the quest to increase transparency is the practice of preregistering a study and analysis plan prior to conducting a study. In this article, we examine the potential of preregistration as a strategy for increasing transparency in education studies. We review existing registries within the social sciences and provide a rationale for why we need a registry specifically geared towards education studies. Finally, we introduce the Registry of Efficacy and Effectiveness Studies (REES), developed by the Society for Research on Educational Effectiveness (SREE). The goal of REES is to increase transparency for studies seeking to draw causal conclusions within the education research and evaluation community. [This article was published in "Educational Researcher" (EJ1204535).]

The purpose of this study was to examine the role of school background and school process in closing achievement gaps between White and non-White students in science. To answer the research questions, a series of two-level hierarchical linear models (HLM) was performed on the fourth-grade U.S. portion of the 2015 Trends in International Mathematics and Science Study (TIMSS) data. Results indicate that (a) the science achievement gap between White and non-White students is 0.21 standard deviation, holding student and school background constant; (b) the science achievement gap varies across schools; (c) none of the school background variables are associated with the achievement gap in a school; and (d) school emphasis on student academic learning is not only associated with higher school-level science achievement, but also with a narrower science achievement gap between White and non-White students. However, teacher collaboration is not associated with school-level science achievement but is associated with a larger achievement gap. Implications, limitations, and recommendations for further research are discussed.

The Common Guidelines for Education Research and Development were created as a joint effort between the Institute of Education Science and the National Science Foundation in an effort to streamline education research and contribute to an accumulation of knowledge that will lead to improved student outcomes. One type of research that emerged in the guidelines is impact research. In order to achieve the level of rigor expected for an impact study, it is common that a research team will employ a cluster randomized trial (CRT). This article provides empirical estimates of design parameters necessary for planning adequately powered CRTs focused on science achievement. Examples of how to use these parameters to improve the design of science impact studies are discussed.

Professional development for teachers is regarded as one of the principal pathways through which we can understand and cultivate effective teaching and improve student outcomes. A critical component of studies that seek to improve teaching through professional development is the detailed assessment of the intermediate teacher development processes that scaffold program content through three key types of outcomes--teacher knowledge, instruction, and student learning. Cross-level and sequential mediation strategies that probe and connect these processes and outcomes emerge as an important design consideration in these studies. We derive formulas that track the power with which school-randomized designs can detect professional development effects as they operate through a sequence of teacher-level mediators to affect student outcomes (e.g., school-randomized professional development studies). The results suggest that the sample sizes typically seen in well-planned experiments targeting total effects (e.g., 30-100 schools) can produce comparably high or disparately low levels of power for mediation effects--the similarity depends heavily on context and concomitant parameter values. The results are implemented in the "PowerUpR" package and in the "PowerUpR" Shiny application.

Experimental research in education and training programs typically involves administering treatment to whole groups of individuals. As such, researchers rely on the estimation of design parameter values to conduct power analyses to efficiently plan their studies to detect desired effects. In this study, we present design parameter estimates from a compilation of 11 studies involving teacher-level outcomes. These results expand upon what little is known about the clustering of variance among teacher-level outcomes. The findings inform the design of intervention studies which nest teachers within schools and aim to improve teacher mathematics and science content knowledge or instructional practices. The results show large differences in unconditional intraclass correlation coefficients across studies as well as within outcomes. They also quantify the relative importance of having a pretest measure to promote efficiency. This study highlights a need for improved reporting of this information in the literature to facilitate better experimental designs of interventions involving teacher-level outcomes.

The Society for Research on Educational Effectiveness, with funding from the Institute of Education Sciences (IES), has been working to develop and implement a registry for education studies, the Registry of Efficacy and Effectiveness Studies (REES) (https://www. sree.org/pages/registry.php). REES aims to increase transparency, rigor, and replicability of education evaluations. In this brief article, the authors describe how they have structured REES and the implications of this structure in meeting the goals of increasing transparency.

Researchers are often interested in whether the effects of an intervention differ conditional on individual- or group-moderator variables such as children's characteristics (e.g., gender), teacher's background (e.g., years of teaching), and school's characteristics (e.g., urbanity); that is, the researchers seek to examine for whom and under what circumstances an intervention works. Furthermore, the researchers are interested in understanding and interpreting variability in treatment effects through moderation analysis as an approach to exploring the sources of the treatment effect variability. This study develops formulas for power analyses to detect the moderator effects in designing three-level cluster randomized trials (CRTs). We develop the statistical formulas for calculating statistical power, minimum detectable effect size difference, and 95% confidence intervals for cluster or cross-level moderation, nonrandomly varying or random slopes, binary or continuous moderators, and designs with or without covariates. We demonstrate how the calculations can be used in the planning phase of three-level CRTs using the software PowerUp!-Moderator.

Designs that facilitate inferences concerning both the total and indirect effects of a treatment potentially offer a more holistic description of interventions because they can complement "what works" questions with the comprehensive study of the causal connections implied by substantive theories. Mapping the sensitivity of designs to detect these effects is of critical importance because it directly governs the types of evidence researchers can bring to bear on theories of action under realistic sample sizes. In this study, we develop closed-form expressions to estimate the variance of and the power to detect causally defined indirect effects in two-level group-randomized studies examining individual-level mediators (i.e., 2-1-1 mediation). We formulate our approach within the purview of typical multilevel mediation models and anchor their interpretation in the potential outcomes framework. The results provide power analysis formulas that reduce calculations to simple functions of the primary path coefficients (e.g., treatment-mediator and mediator-outcome relationships) and common summary statistics (e.g., intraclass correlation coefficients). Probing these formulas suggests that group-randomized designs can be well powered to detect indirect effects when carefully planned. The power formulas are implemented in the PowerUp software (causalevaluation.org).

Multisite trials, which are being used with increasing frequency in education and evaluation research, provide an exciting opportunity for learning about how the effects of interventions or programs are distributed across sites. In particular, these studies can produce rigorous estimates of a cross-site mean effect of program assignment (intent-to-treat), a cross-site standard deviation of the effects of program assignment, and a difference between the cross-site mean effects of program assignment for two subpopulations of sites. However, to capitalize on this opportunity will require adequately powering future trials to estimate these parameters. To help researchers do so, we present a simple approach for computing the minimum detectable values of these parameters for different sample designs. The article then uses this approach to illustrate for each parameter, the precision trade-off between increasing the number of study sites and increasing site sample size. Findings are presented for multisite trials that randomize individual sample members and for multisite trials that randomize intact groups or clusters of sample members.

We develop a theoretical and empirical basis for the design of teacher professional development studies. We build on previous work by (a) developing estimates of intraclass correlation coefficients for teacher outcomes using two- and three-level data structures, (b) developing estimates of the variance explained by covariates, and (c) modifying the conventional optimal design framework to include differential covariate costs so as to capture the point at which the cost of collecting a covariate overtakes the reduction in variance it supplies. We illustrate the use of these estimates to explore the absolute and relative sensitivity of multilevel designs in teacher professional development studies. The results from these analyses are intended to guide researchers in making more-informed decisions about the tradeoffs and considerations involved in selecting study designs for assessing the impacts of professional development programs.

Recently, there has been an increase in the number of cluster randomized trials (CRTs) to evaluate the impact of educational programs and interventions. These studies are often powered for the main effect of treatment to address the "what works" question. However, program effects may vary by individual characteristics or by context, making it important to also consider power to detect moderator effects. This article presents a framework for calculating statistical power for moderator effects at all levels for two- and three-level CRTs. Annotated R code is included to make the calculations accessible to researchers and increase the regularity in which a priori power analyses for moderator effects in CRTs are conducted.

Cluster randomized trials (CRTs) are commonly used to evaluate educational interventions, particularly their effectiveness. Recently there has been greater emphasis on using these trials to explore cost-effectiveness. However, methods for establishing the power of cluster randomized cost-effectiveness trials (CRCETs) are limited. This study develops power computation formulas and statistical software to help researchers plan two- and three-level CRCETs. We illustrate the application of our formulas and software for the designs of CRCETs and discuss the influence of sample size, nesting effects, covariates, and the covariance between cost and effectiveness measures on the statistical power of cost-effectiveness estimates. [ABSTRACT FROM AUTHOR]

This article examines the statistical precision of cluster randomized trials (CRTs) funded by the Institute of Education Sciences (IES). Specifically, it compares the total number of clusters randomized and the minimum detectable effect size (MDES) of two sets of studies, those funded in the early years of IES (2002-2004) and those funded in the recent years (2011-2013). Overall, the average precision in terms of MDES of studies in the recent cohort was more than double that of the early cohort (i.e. 0.48 compared to 0.23). The findings suggest a consistent and substantial increase in the precision of CRTs funded by IES in the past decade which is a critical step towards designing studies that have the potential to yield high-quality evidence about the effectiveness of educational interventions.

The Institute of Education Sciences has funded more than 100 experiments to evaluate educational interventions in an effort to generate scientific evidence of program effectiveness on which to base education policy and practice. In general, these studies are designed with the goal of having adequate statistical power to detect the average treatment effect. However, the average treatment effect may be less informative if the treatment effects vary substantially from site to site or if the intervention effects differ across context or subpopulations. This article considers the precision of studies to detect different types of treatment effect heterogeneity. Calculations are demonstrated using a set of Institute of Education Sciences funded cluster randomized trials. Strategies for planning future studies with adequate precision for estimating treatment effect heterogeneity are discussed.

Research designs in which clusters are the unit of randomization are quite common in the social sciences. Given the multilevel nature of these studies, the power analyses for these studies are more complex than in a simple individually randomized trial. Tools are now available to help researchers conduct power analyses for cluster randomized trials. A key challenge facing researchers in using the tools is that different tools use different language, notation, and design parameters. This makes it difficult for researchers to communicate about the steps for a power analysis or to evaluate the appropriateness of a power analysis. This article provides explicit connections between the language, notation, and design parameters of two widely used approaches and tools for conducting power analyses, "Optimal Design Plus" and "CRT Power." The formulas for translating design parameters between the two programs are provided and the comparability of the two programs is illustrated.

Research Findings: Examining the effects of professional development of the early childhood workforce that fit within the constraints of government policy is crucial for identifying types and amounts of effective training and informing child care policy. The present study used a cluster-randomized trial to evaluate the effects of a professional development program for child care providers designed to meet the criteria for 2 state-level policies: (a) that child care providers working in licensed centers engage in 10 hr of professional development annually and (b) that all licensed child care settings provide 30 min of developmentally appropriate literacy activity daily. Results indicated that 10 hr of professional development focused on literacy was effective for significantly improving the literacy practices and knowledge of child care providers. However, it was not effective in eliciting substantial growth in child literacy outcomes, at least in the short term. The lack of child outcomes illustrates the importance of measuring professional development effects at both the provider and child levels. Practice or Policy: This study illustrates the importance of critically questioning and analyzing state policy, particularly dosage. In practice, dosage is an influential factor in how professional development is selected by programs and providers, because most policies only specify a required number of hours to be completed. The design of policy, which can influence both provider practice and child outcomes, relies upon alignment between early childhood research and policy.