Search

In this issue of the Journal, Mooney et al. (Am J Epidemiol. 2021;190(8):1476–1482) discuss machine learning as a tool for causal research in the style of Internet headlines. Here we comment by adapting famous literary quotations, including the one in our title (from “Sonnet 43” by Elizabeth Barrett Browning (Sonnets From the Portuguese, Adelaide Hanscom Leeson, 1850)). We emphasize that any use of machine learning to answer causal questions must be founded on a formal framework for both causal and statistical inference. We illustrate the pitfalls that can occur without such a foundation. We conclude with some practical recommendations for integrating machine learning into causal analyses in a principled way and highlight important areas of ongoing work.

The consistency of propensity score (PS) estimators relies on correct specification of the PS model. The PS is frequently estimated using main-effects logistic regression. However, the underlying model assumptions may not hold. Machine learning methods provide an alternative nonparametric approach to PS estimation. In this simulation study, we evaluated the benefit of using Super Learner (SL) for PS estimation. We created 1,000 simulated data sets (n = 500) under 4 different scenarios characterized by various degrees of deviance from the usual main-term logistic regression model for the true PS. We estimated the average treatment effect using PS matching and inverse probability of treatment weighting. The estimators' performance was evaluated in terms of PS prediction accuracy, covariate balance achieved, bias, standard error, coverage, and mean squared error. All methods exhibited adequate overall balancing properties, but in the case of model misspecification, SL performed better for highly unbalanced variables. The SL-based estimators were associated with the smallest bias in cases of severe model misspecification. Our results suggest that use of SL to estimate the PS can improve covariate balance and reduce bias in a meaningful manner in cases of serious model misspecification for treatment assignment.

Although clinic-based cohorts are most representative of the “real world,” they are susceptible to loss to follow-up. Strategies for managing the impact of loss to follow-up are therefore needed to maximize the value of studies conducted in these cohorts. The authors evaluated adult patients starting antiretroviral therapy at an HIV/AIDS clinic in Uganda, where 29% of patients were lost to follow-up after 2 years (January 1, 2004–September 30, 2007). Unweighted, inverse probability of censoring weighted (IPCW), and sampling-based approaches (using supplemental data from a sample of lost patients subsequently tracked in the community) were used to identify the predictive value of sex on mortality. Directed acyclic graphs (DAGs) were used to explore the structural basis for bias in each approach. Among 3,628 patients, unweighted and IPCW analyses found men to have higher mortality than women, whereas the sampling-based approach did not. DAGs encoding knowledge about the data-generating process, including the fact that death is a cause of being classified as lost to follow-up in this setting, revealed “collider” bias in the unweighted and IPCW approaches. In a clinic-based cohort in Africa, unweighted and IPCW approaches—which rely on the “missing at random” assumption—yielded biased estimates. A sampling-based approach can in general strengthen epidemiologic analyses conducted in many clinic-based cohorts, including those examining other diseases.

Much of epidemiology and clinical medicine is focused on estimating the effects of treatments or interventions administered over time. In such settings of longitudinal treatment, time-dependent confounding is often an important source of bias. Marginal structural models (MSMs) are a powerful tool for estimating the causal effect of a treatment using observational data, particularly when time-dependent confounding is present. In recent statistical work, van der Laan et al. presented a generalized form of MSMs called "history-adjusted" MSMs (Int J Biostat 2005;1:article 4). Unlike standard MSMs, history-adjusted MSMs can be used to estimate modification of treatment effects by time-varying covariates. Estimation of time-dependent causal effect modification is frequently of great practical relevance. For example, clinical researchers are often interested in how the prognostic significance of a biomarker for treatment response can change over time. This article provides a practical introduction to the implementation and interpretation of history-adjusted MSMs. The method is illustrated using a clinical question drawn from the treatment of human immunodeficiency virus infection. Observational cohort data from San Francisco, California, collected between 2000 and 2004, are used to estimate the effect of time until switching antiretroviral therapy regimens among patients receiving a non suppressive regimen and how this effect differs depending on CD4-positive T-lymphocyte count.

We thank Robins et al. (1) for their thoughtful commentary on our article (2). Before responding, we wish to point out that they altered our notation slightly, using m rather than j to refer to the baseline time point. Robins et al. distinguish between two types of history-adjusted marginal structural models (HA-MSMs), those that consider an outcome at a fixed time point (“group 1,” or those with multiple baseline time points for a single time-point-specific outcome) and those that consider a moving outcome (“group 2,” or those that still use multiple baseline time points but use no more than a single baseline for each time-point-specific outcome). They raise the issue of model incompatibility and make the point that “only MSMs in group 1 can be incompatible and thus lead to logical inconsistencies” (1, p. 996). We concur with their discussion of the pitfalls that can be involved in modeling group 1 HA-MSMs and appreciate their emphasis on this issue. We make two major points in response. 1) As Robins et al. state explicitly (1), the pitfalls regarding model incompatibility do not apply to group 2 HA-MSMs. However, we discuss only group 2 HA-MSMs in our article (2). Therefore, we would like to clarify that the issue of model incompatibility raised in the commentary does not apply to the method outlined in our paper. 2) We feel that the method outlined in our paper for estimation of group 2 HA-MSMs constitutes a significant practical contribution, for the reasons stated below. First, as Robins et al. mention (1), group 2 HA-MSMs allow pooling of data across time and can thereby enhance the efficiency of the resulting estimators. Second, this approach provides insight into how effect estimates differ across strata defined by covariates measured at different time points. Our data example illustrates this application: Effect estimates at later time points following loss of viral suppression differed less between subjects with different CD4 T-cell counts than such estimates at earlier time points. We agree that this type of effect modification by time-dependent covariates is different from the effect modification in a group 1 HA-MSM, but we suggest that it is nonetheless both interesting and interpretable. Third, although the issue was beyond the scope of our article, group 2 HA-MSMs can be used to identify interesting dynamic treatment rules, in which treatment assignment is updated over time in response to changes in patient covariates (3). We appreciate Robins et al.’s focus on the confounding/ selection bias that can result from the way in which treatment prior to baseline was assigned. We agree that this is an important issue in the interpretation of HA-MSMs and their application to dynamic treatment rules, and we discuss it in more detail elsewhere (3, 4). Inclusion in V(j) (or V(m) in Robins et al.’s notation) of covariates sufficient to control confounding of past treatment guarantees that the parameter estimated is only a function of the distribution of the counterfactuals, whereas if this condition is not met, one estimates a causal effect for a selected subpopulation in which the selection was ruled by the treatment mechanism. For example, in our paper (2), we point out the potential role played by changes over time in the composition of the population remaining on nonsuppressive therapy. This issue can also arise in standard baseline-adjusted MSMs if the baseline history includes time-dependent covariates. Finally, Robins et al. raise an interesting philosophical question. They reason that the group 1 HA-MSM parameter, in spite of the fact that it is well-defined and identifiable, is not well-suited for modeling. One might argue in response that difficulty in modeling a specific parameter should be regarded as a welcome challenge by statisticians, rather than a motivation to avoid the parameter altogether. In the case of group 1 HA-MSMs, however, we wonder whether the issue is less that the parameter is difficult to model and more that in many instances it is not in fact the causal effect of interest. We look forward to future work exploring these issues.