Search

Directed acyclic graphs (DAGs) are a useful tool to represent, in a graphical format, researchers’ assumptions about the causal structure among variables while providing a rationale for the choice of confounding variables to adjust for. With origins in the field of probabilistic graphical modelling, DAGs are yet to be widely adopted in applied health research, where causal assumptions are frequently made for the purpose of evaluating health services initiatives. In this context, there is still limited practical guidance on how to construct and use DAGs. Some progress has recently been made in terms of building DAGs based on studies from the literature, but an area that has received less attention is how to create DAGs from information provided by domain experts, an approach of particular importance when there is limited published information about the intervention under study. This approach offers the opportunity for findings to be more robust and relevant to patients, carers and the public, and more likely to inform policy and clinical practice. This article draws lessons from a stakeholder workshop involving patients, health care professionals, researchers, commissioners and representatives from industry, whose objective was to draw DAGs for a complex intervention—online consultation, i.e. written exchange between the patient and health care professional using an online system—in the context of the English National Health Service. We provide some initial, practical guidance to those interested in engaging with domain experts to develop DAGs.

In studies based on electronic health records (EHR), the frequency of covariate monitoring can vary by covariate type, across patients, and over time, which can limit the generalizability of inferences about the effects of adaptive treatment strategies. In addition, monitoring is a health intervention in itself with costs and benefits, and stakeholders may be interested in the effect of monitoring when adopting adaptive treatment strategies. This paper demonstrates how to exploit non‐systematic covariate monitoring in EHR‐based studies to both improve the generalizability of causal inferences and to evaluate the health impact of monitoring when evaluating adaptive treatment strategies. Using a real world, EHR‐based, comparative effectiveness research (CER) study of patients with type II diabetes mellitus, we illustrate how the evaluation of joint dynamic treatment and static monitoring interventions can improve CER evidence and describe two alternate estimation approaches based on inverse probability weighting (IPW). First, we demonstrate the poor performance of the standard estimator of the effects of joint treatment‐monitoring interventions, due to a large decrease in data support and concerns over finite‐sample bias from near‐violations of the positivity assumption (PA) for the monitoring process. Second, we detail an alternate IPW estimator using a no direct effect (NDE) assumption. We demonstrate that this estimator can improve efficiency but at the potential cost of increase in bias from violations of the PA for the treatment process.

Policymakers seeking to target health policies efficiently towards specific population groups need to know which individuals stand to benefit the most from each of these policies. While traditional approaches for subgroup analyses are constrained to only consider a small number of pre-defined subgroups, recently proposed causal machine learning (CML) approaches help explore treatment-effect heterogeneity in a more flexible yet principled way. Causal forests use a generalisation of the random forest algorithm to estimate heterogenous treatment effects both at the individual and the subgroup level. Our paper aims to explore this approach in the setting of health policy evaluation with strong observed confounding, applied specifically to the context of mothers’ health insurance enrolment in Indonesia. Comparing two health insurance schemes (subsidised and contributory) against no insurance, we find beneficial average impacts of enrolment in contributory health insurance on maternal health care utilisation and infant mortality, but no impact of subsidised health insurance. The causal forest algorithm identified significant heterogeneity in the impacts of contributory insurance, not just along socioeconomic variables that we pre-specified (indicating higher benefits for poorer, less educated, and rural women), but also according to some other characteristics not foreseen prior to the analysis, suggesting in particular important geographical impact heterogeneity. Our study demonstrates the power of CML approaches to uncover unexpected heterogeneity in policy impacts. The findings from our evaluation of past health insurance expansions can potentially guide the re-design of the eligibility criteria for subsidised health insurance in Indonesia.

Background The present study seeks to evaluate the change in mental health inequalities in the department of Meta after the signing of Colombia’s Peace Agreement in 2016 with the FARC guerrilla group. Using a validated survey instrument composed of 20 questions (‘SRQ-20’), we measure changes in mental health inequalities from 2014, before the signing of the agreement, to 2018, after the signing. We then decompose the changes in inequalities to establish which socioeconomic factors explain differences in mental health inequalities over time. Methods Our study uses information from the Conflicto, Salud y Paz (CONPAS) survey conducted in the department of Meta, Colombia, in 1309 households in 2018, with retrospective information for 2014. To measure inequalities, we calculate the concentration indices for both years. Through the Oaxaca change decomposition method, we disaggregate changes in mental health inequalities into its underlying factors. This method allows us to explain the relationship between changes in mental health inequalities and changes in inequalities in several sociodemographic factors. It also identifies the extent to which these factors help explain the changes in mental health inequalities. Results Mental health inequalities in Meta were reduced almost by half from 2014 to 2018. In 2018, the population at the lower and middle socioeconomic levels had fewer chances of experiencing mental health disorders in comparison to 2014. The reduction in mental health differences is mostly attributed to reductions in the influence of certain sociodemographic variables, such as residence in rural zones and conflict-affected territories, working in the informal sector, or experiencing internal displacement. However, even though mental health inequalities have diminished, overall mental health outcomes have worsened in these years. Conclusions The reduction in the contribution of conflict-related variables for explaining mental health inequalities could mean that the negative consequences of conflict on mental health have started to diminish in the short run after the peace agreement. Nevertheless, conflict and the presence of other socioeconomic inequalities still contribute to persistent adverse mental health outcomes in the overall population. Thus, public policy should be oriented towards improving mental health care services in these territories, given the post-accord context.

Public payers around the world are increasingly using cost-effectiveness thresholds (CETs) to assess the value-for-money of an intervention and make coverage decisions. However, there is still much confusion about the meaning and uses of the CET, how it should be calculated, and what constitutes an adequate evidence base for its formulation. One widely referenced and used threshold in the last decade has been the 1-3 GDP per capita, which is often attributed to the Commission on Macroeconomics and WHO guidelines on Choosing Interventions that are Cost Effective (WHO-CHOICE). For many reasons, however, this threshold has been widely criticised; which has led experts across the world, including the WHO, to discourage its use. This has left a vacuum for policy-makers and technical staff at a time when countries are wanting to move towards Universal Health Coverage. This article seeks to address this gap by offering five practical options for decision-makers in low- and middle-income countries that can be used instead of the 1-3 GDP rule, to combine existing evidence with fair decision-rules or develop locally relevant CETs. It builds on existing literature as well as an engagement with a group of experts and decision-makers working in low, middle and high income countries.

Background: The present study seeks to evaluate the evolution of mental health inequalities in the department of Meta after the signing of Colombia's Peace Agreement in 2016 with the FARC guerrilla group. Using a validated survey instrument composed of 20 questions (‘SRQ-20’), we measure changes in mental health inequalities from 2014, before the signing of the agreement, to 2018, after the signing of the agreement. We then decompose the changes in inequalities to establish which socioeconomic factors explain differences over time.Methods: Our study uses information from the Conflicto, Salud y Paz (CONPAS) survey conducted in the department of Meta, Colombia, in 1,309 households in 2018, with retrospective information for 2014. To measure inequalities, we calculate the concentration indices for both years. Through the Oaxaca change decomposition method, we disaggregate changes in mental health inequalities into its underlying factors. This method allows us to explain the relationship between changes in mental health inequalities and reduced inequality in several sociodemographic factors. It also identifies the extent to which these factors help explain the changes in mental health inequalities. Results: Mental health inequalities in Meta were reduced almost by half from 2014 to 2018. In 2018, the population at the lower and middle socioeconomic levels had fewer chances of experiencing mental health disorders in comparison to 2014. The reduction in mental health differences is mostly attributed to reductions in the influence of certain sociodemographic variables, such as residence in rural zones and conflict-affected territories, working in the informal sector, or experiencing internal displacement. However, even though mental health inequalities have diminished, overall mental health outcomes have worsened in these years. Conclusions: The reduction in the contribution of conflict-related variables for explaining mental health inequalities could mean that the negative consequences of conflict on mental health have started to diminish in the short run after the peace agreement. Nevertheless, conflict and the presence of other socioeconomic inequalities still contribute to persistent adverse mental health outcomes in the overall population. Thus, public policy should be oriented towards improving mental health care services in these territories, given the post-accord context.

Objective: To compare interactive fixed effects (IFE) and generalised synthetic control (GSC) methods to methods prevalent in health policy evaluation and re-evaluate the impact of the hip fracture Best Practice Tariffs introduced for hospitals in England in 2010. Data sources: Simulations and Hospital Episode Statistics.Study design: Best Practice Tariffs aimed to incentivise providers to deliver care in line with guidelines. Under the scheme, 62 providers received an additional payment for each hip fracture admission, while 49 providers did not. We estimate the impact using Difference-in-Differences (DiD), Synthetic Controls (SC), IFE and GSC methods. We contrast the estimation methods’ performance in a Monte Carlo simulation study.Principal findings: Unlike DiD, SC and IFE methods, the GSC method provided reliable estimates across a range of simulation scenarios and was preferred for this case study. The introduction of Best Practice Tariffs led to a 5.9 percentage point increase in the proportion of patients having surgery within 48 hours, and a statistically insignificant 0.6 percentage point reduction in 30 day mortality. Conclusions: The GSC approach is an attractive method for health policy evaluation. We cannot be confident that Best Practice Tariffs were effective.Objective: To compare interactive fixed effects (IFE) and generalised synthetic control (GSC) methods to methods prevalent in health policy evaluation and re-evaluate the impact of the hip fracture Best Practice Tariffs introduced for hospitals in England in 2010. Data sources: Simulations and Hospital Episode Statistics.Study design: Best Practice Tariffs aimed to incentivise providers to deliver care in line with guidelines. Under the scheme, 62 providers received an additional payment for each hip fracture admission, while 49 providers did not. We estimate the impact using Difference-in-Differences (DiD), Synthetic Controls (SC), IFE and GSC methods. We contrast the estimation methods’ performance in a Monte Carlo simulation study.Principal findings: Unlike DiD, SC and IFE methods, the GSC method provided reliable estimates across a range of simulation scenarios and was preferred for this case study. The introduction of Best Practice Tariffs led to a 5.9 percentage point increase in the proportion of patients having surgery within 48 hours, and a statistically insignificant 0.6 percentage point reduction in 30 day mortality. Conclusions: The GSC approach is an attractive method for health policy evaluation. We cannot be confident that Best Practice Tariffs were effective.Objective: To compare interactive fixed effects (IFE) and generalised synthetic control (GSC) methods to methods prevalent in health policy evaluation and re-evaluate the impact of the hip fracture Best Practice Tariffs introduced for hospitals in England in 2010. Data sources: Simulations and Hospital Episode Statistics.Study design: Best Practice Tariffs aimed to incentivise providers to deliver care in line with guidelines. Under the scheme, 62 providers received an additional payment for each hip fracture admission, while 49 providers did not. We estimate the impact using Difference-in-Differences (DiD), Synthetic Controls (SC), IFE and GSC methods. We contrast the estimation methods’ performance in a Monte Carlo simulation study.Principal findings: Unlike DiD, SC and IFE methods, the GSC method provided reliable estimates across a range of simulation scenarios and was preferred for this case study. The introduction of Best Practice Tariffs led to a 5.9 percentage point increase in the proportion of patients having surgery within 48 hours, and a statistically insignificant 0.6 percentage point reduction in 30 day mortality. Conclusions: The GSC approach is an attractive method for health policy evaluation. We cannot be confident that Best Practice Tariffs were effective.Objective: To compare interactive fixed effects (IFE) and generalised synthetic control (GSC) methods to methods prevalent in health policy evaluation and re-evaluate the impact of the hip fracture Best Practice Tariffs introduced for hospitals in England in 2010. Data sources: Simulations and Hospital Episode Statistics.Study design: Best Practice Tariffs aimed to incentivise providers to deliver care in line with guidelines. Under the scheme, 62 providers received an additional payment for each hip fracture admission, while 49 providers did not. We estimate the impact using Difference-in-Differences (DiD), Synthetic Controls (SC), IFE and GSC methods. We contrast the estimation methods’ performance in a Monte Carlo simulation study.Principal findings: Unlike DiD, SC and IFE methods, the GSC method provided reliable estimates across a range of simulation scenarios and was preferred for this case study. The introduction of Best Practice Tariffs led to a 5.9 percentage point increase in the proportion of patients having surgery within 48 hours, and a statistically insignificant 0.6 percentage point reduction in 30 day mortality. Conclusions: The GSC approach is an attractive method for health policy evaluation. We cannot be confident that Best Practice Tariffs were effective.Objective: To compare interactive fixed effects (IFE) and generalised synthetic control (GSC) methods to methods prevalent in health policy evaluation and re-evaluate the impact of the hip fracture Best Practice Tariffs introduced for hospitals in England in 2010. Data sources: Simulations and Hospital Episode Statistics.Study design: Best Practice Tariffs aimed to incentivise providers to deliver care in line with guidelines. Under the scheme, 62 providers received an additional payment for each hip fracture admission, while 49 providers did not. We estimate the impact using Difference-in-Differences (DiD), Synthetic Controls (SC), IFE and GSC methods. We contrast the estimation methods’ performance in a Monte Carlo simulation study.Principal findings: Unlike DiD, SC and IFE methods, the GSC method provided reliable estimates across a range of simulation scenarios and was preferred for this case study. The introduction of Best Practice Tariffs led to a 5.9 percentage point increase in the proportion of patients having surgery within 48 hours, and a statistically insignificant 0.6 percentage point reduction in 30 day mortality. Conclusions: The GSC approach is an attractive method for health policy evaluation. We cannot be confident that Best Practice Tariffs were effective.

To identify the pattern of the risk of death over long-term in unresectable hepatocellular carcinoma by determining the appropriate distribution to extrapolate overall survival and to assess the role of the Weibull distribution as the standard survival model in oncology. To select the appropriate distribution, three types of data sources have been analysed. Patient level data from two randomized controlled trials and published Kaplan–Meier curves from a systematic literature review provided short term follow-up data. They were supplemented with patient level data, with long-term follow-up from the Cancer Institute New South Wales, Australia. Published Kaplan–Meier curves were read in and a time-to-event dataset was created. Distributions were fitted to the data from the different sources separately. Their fit was assessed visually and compared using statistical criteria based on log-likelihood, the Akaike information criterion (AIC), and the Bayesian information criterion (BIC). Based on both published and patient-level, and both short- and long-term follow-up data, the Weibull distribution, used very often in cost-effectiveness models in oncology, does not seem to offer a good fit in hepatocellular carcinoma among the different survival models. The best fitting distribution appears to be the lognormal, with loglogistic as the second-best fitting function. Results were consistent between the different sources of data. In unresectable hepatocellular carcinoma, the Weibull model, which is often treated at the gold standard, does not appear to be appropriate based on different sources of data (two clinical trials, a retrospective database and published Kaplan–Meier curves). Lognormal distribution seems to be the most appropriate distribution for extrapolating overall survival.

Internal armed conflicts have become more common and more physically destructive since the mid-20th century, with devastating consequences for health and development in low- and middle-income countries. This paper investigates the causal impacts of the long-term internal conflict on child health in Colombia, following an identification strategy based on the temporal and geographic variation of conflict intensity. We estimate the effect of different levels of conflict intensity on height-for-age (HAZ), weight-for-age (WAZ), and weight-for-height z-scores among children under five years old, and explore the underlying potential mechanisms, through maternal health behavior and health care utilization. We find a harmful effect of exposure to conflict violence in utero and in early childhood for HAZ and WAZ, in the full sample and even more strongly in the rural sample, yet these estimates are smaller than those found for shorter term conflicts. The underlying pathways appear to operate around the time of the pregnancy and birth (in the form of maternal alcohol use, use of antenatal care and skilled birth attendance), rather than during the post-birth period (via breastfeeding or vaccination), and the impacts accumulate over the childhood. The most adverse impacts of conflict violence on child health and utilization of maternal healthcare were observed in municipalities which suffered from intermittent presence of armed groups.

While machine learning (ML) methods have received a lot of attention in recent years, these methods are primarily for prediction. Empirical researchers conducting policy evaluations are, on the other hand, preoccupied with causal problems, trying to answer counterfactual questions: what would have happened in the absence of a policy? Because these counterfactuals can never be directly observed (described as the “fundamental problem of causal inference”) prediction tools from the ML literature cannot be readily used for causal inference. In the last decade, major innovations have taken place incorporating supervised ML tools into estimators for causal parameters such as the average treatment effect (ATE). This holds the promise of attenuating model misspecification issues, and increasing of transparency in model selection. One particularly mature strand of the literature include approaches that incorporate supervised ML approaches in the estimation of the ATE of a binary treatment, under the unconfoundedness and positivity assumptions (also known as exchangeability and overlap assumptions). This article begins by reviewing popular supervised machine learning algorithms, including trees-based methods and the lasso, as well as ensembles, with a focus on the Super Learner. Then, some specific uses of machine learning for treatment effect estimation are introduced and illustrated, namely (1) to create balance among treated and control groups, (2) to estimate so-called nuisance models (e.g., the propensity score, or conditional expectations of the outcome) in semi-parametric estimators that target causal parameters (e.g., targeted maximum likelihood estimation or the double ML estimator), and (3) the use of machine learning for variable selection in situations with a high number of covariates. Since there is no universal best estimator, whether parametric or data-adaptive, it is best practice to incorporate a semi-automated approach than can select the models best supported by the observed data, thus attenuating the reliance on subjective choices.

Background: Health impact evaluations (HIEs) are currently the main way of assessing policy changes in low-and middle-income countries (LMICs). However, evidence on effectiveness alone cannot reliably inform decisions over the allocation of limited resources. Health economic evaluation provides a suitable framework for ‘value for money’ assessments. Methods: In this article we explore to what extent economic evaluations have been conducted alongside published health impact evaluations, then we assess the quality of these, using criteria from an economic evaluation reference case developed for use in LMICs. Results: Among the 2419 HIEs stored in the International Initiative for Impact Evaluations (3ie) database, and among the 8155 studies identified by the Ovid Medline database search, only 70 studies included an economic evaluation. When measured against the quality assessment criteria, study quality showed great variation. Many studies did not fulfil the basic requirements for economic evaluation, such as stating the perspective of the budget holder, using generic health measures that can be compared across diseases, or suitably reflecting uncertainty. Conclusions: Greater effort should be directed towards bringing the fields of impact evaluation and economic evaluation together to better inform resource allocation decisions in global health.

We study a pay-for-efficiency scheme that encourages hospitals to admit and discharge patients on the same calendar day when clinically appropriate. Since 2010, hospitals in the English NHS are incentivised by a higher price for patients treated as same-day discharge than for overnight stays, despite the former being less costly. We analyse administrative data for patients treated during 2006–2014 for 191 conditions for which same-day discharge is clinically appropriate – of which 32 are incentivised. Using difference-in-difference and synthetic control methods, we find that the policy had generally a positive impact with a statistically significant effect in 14 out of the 32 conditions. The median elasticity is 0.24 for planned and 0.01 for emergency conditions. Condition-specific design features explain some, but not all, of the differential responses.

Summary For a continuous treatment, the generalised propensity score (GPS) is defined as the conditional density of the treatment, given covariates. GPS adjustment may be implemented by including it as a covariate in an outcome regression. Here, the unbiased estimation of the dose–response function assumes correct specification of both the GPS and the outcome‐treatment relationship. This paper introduces a machine learning method, the ‘Super Learner’, to address model selection in this context. In the two‐stage estimation approach proposed, the Super Learner selects a GPS and then a dose–response function conditional on the GPS, as the convex combination of candidate prediction algorithms. We compare this approach with parametric implementations of the GPS and to regression methods. We contrast the methods in the Risk Adjustment in Neurocritical care cohort study, in which we estimate the marginal effects of increasing transfer time from emergency departments to specialised neuroscience centres, for patients with acute traumatic brain injury. With parametric models for the outcome, we find that dose–response curves differ according to choice of specification. With the Super Learner approach to both regression and the GPS, we find that transfer time does not have a statistically significant marginal effect on the outcomes. © 2015 The Authors. Health Economics Published by John Wiley & Sons Ltd.

Internationally, policy makers are increasingly focussed on reducing the detrimental consequences and rising costs associated with unhealthy diets, inactivity, smoking, alcohol and other risk factors on the health of their populations. This has led to an increase in the demand for evidence-based, cost-effective Population Health Interventions (PHIs) to reverse this trend. Given that research designs such as randomised controlled trials (RCTs) are often not suited to the evaluation of PHIs, Natural Experiments (NEs) are now frequently being used as a design to evaluate such complex, preventive PHIs. However, current guidance for economic evaluation focusses on RCT designs and therefore does not address the specific challenges of NE designs. Using such guidance can lead to sub-optimal design, data collection and analysis for NEs, leading to bias in the estimated effectiveness and cost-effectiveness of the PHI. As a consequence, there is a growing recognition of the need to identify a robust methodological framework for the design and conducting of economic evaluations alongside such NEs. This paper outlines the challenges inherent to the design and conduct of economic evaluations of PHIs alongside NEs, providing a comprehensive framework and outlining a research agenda in this area.

This paper examines the synthetic control method in contrast to commonly used difference‐in‐differences (DiD) estimation, in the context of a re‐evaluation of a pay‐for‐performance (P4P) initiative, the Advancing Quality scheme. The synthetic control method aims to estimate treatment effects by constructing a weighted combination of control units, which represents what the treated group would have experienced in the absence of receiving the treatment. While DiD estimation assumes that the effects of unobserved confounders are constant over time, the synthetic control method allows for these effects to change over time, by re‐weighting the control group so that it has similar pre‐intervention characteristics to the treated group. We extend the synthetic control approach to a setting of evaluation of a health policy where there are multiple treated units. We re‐analyse a recent study evaluating the effects of a hospital P4P scheme on risk‐adjusted hospital mortality. In contrast to the original DiD analysis, the synthetic control method reports that, for the incentivised conditions, the P4P scheme did not significantly reduce mortality and that there is a statistically significant increase in mortality for non‐incentivised conditions. This result was robust to alternative specifications of the synthetic control method. © 2015 The Authors. Health Economics published by John Wiley & Sons Ltd.