Search

Multisite trials can clarify the average impact of a new program and the heterogeneity of impacts across sites. Unfortunately, in many applications, compliance with treatment assignment is imperfect. For these applications, we propose an instrumental variable (IV) model with person-specific and site-specific random coefficients. Site-specific IV coefficients can be interpreted as site-average effects of program participation or as site-average effects of participation for the compliers. The validity of these interpretations depends on the analyst's assumptions. Within the framework of a two-level hierarchical linear model, we propose three ways to estimate the mean and variance of these site-specific effects: (a) estimate the impact of program participation and its standard error in each site, then combine these site-specific statistics to estimate the mean and variance of the true site effects; (b) estimate the mean and variance of the effect of treatment assignment on the outcome and the mean and variance of the effect of treatment assignment on program participation; then combine these results to obtain estimates of the mean and variance of the effect of program participation; and (c) use Site by Treatment interactions as multiple instruments. If we assume the IV coefficients to be homogenous across sites, the three approaches are equivalent to variants of familiar two-stage least squares estimates with site fixed effects. Estimates based on our model are valid under a weaker assumption: that site-average levels of compliance are independent of site-average effects of program participation. To illustrate our approach, we evaluate a district-wide policy intended to increase math instructional time in math for low-achieving students. Finally, we discuss how Method (c) can be extended to incorporate multiple mediators. [ABSTRACT FROM PUBLISHER]

The article presents views of the author in response to the comments regarding the quantitative methods for assessing causal mediation and the use of principal stratification. They emphasized on three types of assumptions including identification assumptions required in a randomized experiment for identifying causal mechanisms, required in a quasi-experimental study and model-based assumptions. They also discussed the analysis of controlled direct effects.

The conventional approaches to mediation analysis such as path analysis and structural equation modeling typically involve specifying two structural models, one for the mediator and the other for the outcome. We employ an alternative approach that avoids some strong identification assumptions invoked by the conventional approaches. By applying a new weighting procedure to the observed data, we estimate the average potential outcome if the entire population were treated, the average potential outcome if the entire population were untreated, and the average potential outcome if the entire population were treated and if every individual unit's mediator value would counterfactually remain at the same level as it would be when untreated. The estimated differences among these average potential outcomes provide estimates of the total effect, the natural direct effect, and the natural indirect effect. Applying this approach to multilevel educational data, we evaluate the total effect of the algebra-for-all policy in the Chicago Public Schools by comparing the math achievement of two ninth-grade cohorts. We further investigate whether the policy effect was mediated by the policy-induced change in class peer ability. Combining weighting with prognostic score-based difference-in-differences adjustment enables us to reduce both measured and unmeasured confounding. [ABSTRACT FROM PUBLISHER]

By incorporating two theoretical frameworks this study examines how school characteristics shape first-grade reading ability-grouping practices, and how this, in turn, affects students' reading achievement. The author uses the data from the Early Childhood Longitudinal Study and applies the propensity-score method to examine whether first-grade ability grouping improves student achievement, whether ability grouping increases achievement inequalities, and whether its effects vary by student initial abilities and/or school contexts. Findings support an argument that ability grouping is an organizational response to problems of diversity in the student body. Schools that use ability grouping are likely to have heterogeneous ability compositions. They are also public, low-performing, low socioeconomic status, and high-minority schools. In these schools, ability grouping has no effects or negative effects, particularly for low-ability students. In contrast, ability grouping may improve achievement for all students in schools with advantageous characteristics, mostly private schools, and may reduce achievement inequalities, because low-ability students benefit the most from this practice. [ABSTRACT FROM AUTHOR]

Expanded instructional time has become increasingly popular as a strategy to improve the academic outcomes of low-skilled students, particularly in the 9th grade. We evaluate the efficacy of a double-period algebra policy initiated in the Chicago Public Schools in 2003. This policy required all students with 8th-grade test scores below the national median to enroll in a support algebra course in addition to regular algebra in the 9th grade. Using regression discontinuity combined with interrupted time series designs, and instrumental variable models, we show the effects of the policy on students' grades, failure rates and test scores in 9th-grade algebra and 10th-grade geometry. Providing support courses improved algebra test scores for the target population but only modestly affected grades and failure rates. Students with very low initial abilities benefited less than students close to the national median. The policy also led schools to track algebra classes by students' entering math skills. As a result, it affected academic outcomes among students not targeted by the policy; test scores among high-ability students improved whereas their grades declined. [ABSTRACT FROM AUTHOR]

The article presents the authors views on the advantages and disadvantages of using variable options in multisite trials. He mentioned that a central concern in of the article is the interpretation one can make of the average effect of program participation in each site, the overall average effect of program participation, the site variance in site-specific effects.