Search

Clustered observational studies (COSs) are a critical analytic tool for educational effectiveness research. We present a design framework for the development and critique of COSs. The framework is built on the counterfactual model for causal inference and promotes the concept of designing COSs that emulate the targeted randomized trial that would have been conducted were it feasible. We emphasize the key role of understanding the assignment mechanism to study design. We review methods for statistical adjustment and highlight a recently developed form of matching designed specifically for COSs. We review how regression models can be profitably combined with matching and note best practices for estimates of statistical uncertainty. Finally, we review how sensitivity analyses can determine whether conclusions are sensitive to bias from potential unobserved confounders. We demonstrate concepts with an evaluation of a summer school reading intervention in a large U.S. school district.

Instrumental variable methods, subject to appropriate identification assumptions, enable consistent estimation of causal effects in the presence of unobserved confounding. Near–far matching has been proposed as one analytic method to improve inference by strengthening the effect of the instrument on the exposure and balancing observable characteristics between groups of subjects with low and high values of the instrument. However, in settings with hierarchical data (e.g. patients nested within hospitals), or where several covariate interactions must be balanced, conventional near–far matching algorithms may fail to achieve the requisite covariate balance. We develop a new matching algorithm, that combines near–far matching with refined covariate balance, to balance large numbers of nominal covariates while also strengthening the instrumental variable. This extension of near–far matching is motivated by a case-study that aims to identify the causal effect of prompt admission to an intensive care unit on 7-day and 28-day mortality.

Many interventions in education occur in settings where treatments are applied to groups. For example, a reading intervention may be implemented for all students in some schools and withheld from students in other schools. When such treatments are nonrandomly allocated, outcomes across the treated and control groups may differ due to the treatment or due to baseline differences between groups. When this is the case, researchers can use statistical adjustment to make treated and control groups similar in terms of observed characteristics. Recent work in statistics has developed matching methods designed for contexts where treatments are clustered. This form of matching, known as multilevel matching, may be well suited to many education applications where treatments are assigned to schools. In this article, we provide an extensive evaluation of multilevel matching and compare it to multilevel regression modeling. We evaluate multilevel matching methods in two ways. "First, we use these matching methods in a within-study comparison design and attempt to recover treatment effect estimates from three clustered randomized trials." Second, we conduct a simulation study. We find evidence that generally favors an analytic approach to statistical adjustment that combines multilevel matching with regression adjustment. We conclude with an empirical application.

Many observational studies of causal effects occur in settings with clustered treatment assignment. In studies of this type, treatment is applied to entire clusters of units. For example, an educational intervention might be administered to all the students in a school. We develop a matching algorithm for multilevel data based on a network flow algorithm. Earlier work on multilevel matching relied on integer programming, which allows for balance targeting on specific covariates but can be slow with larger data sets. Although we cannot directly specify minimal levels of balance for individual covariates, our algorithm is fast and scales easily to larger data sets. We apply this algorithm to assess a school-based intervention through which students in treated schools were exposed to a new reading program during summer school. In one variant of the algorithm, where we match both schools and students, we change the causal estimand through optimal subset matching to better maintain common support. In a second variant, we relax the common support assumption to preserve the causal estimand by only matching on schools. We find that the summer intervention does not appear to increase reading test scores. In a sensitivity analysis, however, we determine that an unobserved confounder could easily mask a larger treatment effect.

ObjectiveDelayed antimicrobial therapy in sepsis is associated with increased hospital mortality, but the impact of antimicrobial timing on long-term outcomes is unknown. We tested the hypothesis that hourly delays to antimicrobial therapy are associated with 1-year mortality in pediatric severe sepsis.DesignRetrospective observational study.SettingQuaternary academic pediatric intensive care unit (PICU) from February 1, 2012 to June 30, 2013.PatientsOne hundred sixty patients aged ≤21 years treated for severe sepsis.InterventionsNone.Measurements and main resultsWe tested the association of hourly delays from sepsis recognition to antimicrobial administration with 1-year mortality using multivariable Cox and logistic regression. Overall 1-year mortality was 24% (39 patients), of whom 46% died after index PICU discharge. Median time from sepsis recognition to antimicrobial therapy was 137 min (IQR 65-287). After adjusting for severity of illness and comorbid conditions, hourly delays up to 3 h were not associated with 1-year mortality. However, increased 1-year mortality was evident in patients who received antimicrobials ≤1 h (aOR 3.8, 95% CI 1.2, 11.7) or >3 h (aOR 3.5, 95% CI 1.3, 9.8) compared with patients who received antimicrobials within 1 to 3 h from sepsis recognition. For the subset of patients who survived index PICU admission, antimicrobial therapy ≤1 h was also associated with increased 1-year mortality (aOR 5.5, 95% CI 1.1, 27.4), while antimicrobial therapy >3 h was not associated with 1-year mortality (aOR 2.2, 95% CI 0.5, 11.0).ConclusionsHourly delays to antimicrobial therapy, up to 3 h, were not associated with 1-year mortality in pediatric severe sepsis in this study. The finding that antimicrobial therapy ≤1 h from sepsis recognition was associated with increased 1-year mortality should be regarded as hypothesis-generating for future studies.

In a clustered observational study, a treatment is assigned to groups and all units within the group are exposed to the treatment. We develop a new method for statistical adjustment in clustered observational studies using approximate balancing weights, a generalization of inverse propensity score weights that solve a convex optimization problem to find a set of weights that directly minimize a measure of covariate imbalance, subject to an additional penalty on the variance of the weights. We tailor the approximate balancing weights optimization problem to the clustered observational study setting by deriving an upper bound on the mean square error and finding weights that minimize this upper bound, linking the level of covariate balance to a bound on the bias. We implement the procedure by specializing the bound to a random cluster‐level effects model, leading to a variance penalty that incorporates the signal‐to‐noise ratio and penalizes the weight on individuals and the total weight on groups differently according to the the intra‐class correlation. [ABSTRACT FROM AUTHOR]

In education settings, treatments are often non-randomly assigned to clusters, such as schools or classrooms, while outcomes are measured for students. This research design is called the clustered observational study (COS). We examine the consequences of common support violations in the COS context. Common support violations occur when the covariate distributions of treated and control units have substantial areas of non-overlap. Such violations are likely to occur in a COS, especially with a small number of treated clusters. We provide a comprehensive review of methods for overlap violations in the context of COS designs. We provide an overview of diagnostic tests and trimming methods to ensure overlap holds for the distributions of treated and control covariates. We then outline how trimming changes the estimand and how profiling can be used to understand the causal quantity for which overlap holds. Finally, we demonstrate how steps to achieve adequate overlap can result in very narrowly defined causal effects that may have little policy relevance. We use data on Catholic schools to illustrate concepts throughout. [ABSTRACT FROM AUTHOR]