Search

Objective. To evaluate the effects of specification choices on the accuracy of estimates in difference-in-differences (DID) models. Data Sources. Process-of-care quality data from Hospital Compare between 2003 and 2009. Study Design. We performed a Monte Carlo simulation experiment to estimate the effect of an imaginary policy on quality. The experiment was performed for three different scenarios in which the probability of treatment was (1) unrelated to pre-intervention performance; (2) positively correlated with pre-intervention levels of performance; and (3) positively correlated with pre-intervention trends in performance. We estimated alternative DID models that varied with respect to the choice of data intervals, the comparison group, and the method of obtaining inference. We assessed estimator bias as the mean absolute deviation between estimated program effects and their true value. We evaluated the accuracy of inferences through statistical power and rates of false rejection of the null hypothesis. Principal Findings. Performance of alternative specifications varied dramatically when the probability of treatment was correlated with pre-intervention levels or trends. In these cases, propensity score matching resulted in much more accurate point estimates. The use of permutation tests resulted in lower false rejection rates for the highly biased estimators, but the use of clustered standard errors resulted in slightly lower false rejection rates for the matching estimators. Conclusions. When treatment and comparison groups differed on pre-intervention levels or trends, our results supported specifications for DID models that include matching for more accurate point estimates and models using clustered standard errors or permutation tests for better inference. Based on our findings, we propose a checklist for DID analysis. Key Words. Hospitals, econometrics, health economics, quality of care, health policy, Health care delivery in the United States is changing at a dramatic pace. Millions of uninsured citizens, unsustainable cost growth, and uneven quality of care have prompted numerous policy responses [...]

Objective. To evaluate the impact of hospital value-based purchasing (HVBP) on clinical quality and patient experience during its initial implementation period (July 2011-March 2012). Data Sources. Hospital-level clinical quality and patient experience data from Hospital Compare from up to 5 years before and three quarters after HVBP was initiated. Study Design. Acute care hospitals were exposed to HVBP by mandate while critical access hospitals and hospitals located in Maryland were not exposed. We performed a difference-in-differences analysis, comparing performance on 12 incentivized clinical process and 8 incentivized patient experience measures between hospitals exposed to the program and a matched comparison group of nonexposed hospitals. We also evaluated whether hospitals that were ultimately exposed to HVBP may have anticipated the program by improving quality in advance of its introduction. Principal Findings. Difference-in-differences estimates indicated that hospitals that were exposed to HVBP did not show greater improvement for either the clinical process or patient experience measures during the program's first implementation period. Estimates from our preferred specification showed that HVBP was associated with a 0.51 percentage point reduction in composite quality for the clinical process measures (p > .10, 95 percent CI: -1.37, 0.34) and a 0.30 percentage point reduction in composite quality for the patient experience measures (p > .10, 95 percent CI: -0.79, 0.19). We found some evidence that hospitals improved performance on clinical process measures prior to the start of HVBP, but no evidence of this phenomenon for the patient experience measures. Conclusions. The timing of the financial incentives in HVBP was not associated with improved quality of care. It is unclear whether improvement for the clinical process measures prior to the start of HVBP was driven by the expectation of the program or was the result of other factors. Key Words. Pay-for-performance, hospitals, econometrics, health services research, Hospital value-based purchasing (HVBP) is the next step in the evolution of pay-for-performance from an appealing concept to a standard element of the U.S. health care system. The idea of [...]

Objective. The objective of this research was to apply a new methodology (population-level cost-effectiveness analysis) to determine the value of implementing an evidence-based practice in routine care. Data Sources/Study Setting. Data are from sequentially conducted studies: a randomized controlled trial and an implementation trial of collaborative care for depression. Both trials were conducted in the same practice setting and population (primary care patients prescribed antidepressants). Study Design. The study combined results from a randomized controlled trial and a pre-post-quasi-experimental implementation trial. Data Collection/Extraction Methods. The randomized controlled trial collected quality-adjusted life years (QALYs) from survey and medication possession ratios (MPRs) from administrative data. The implementation trial collected MPRs and intervention costs from administrative data and implementation costs from survey. Principal Findings. In the randomized controlled trial, MPRs were significantly correlated with QALYs (p = .03). In the implementation trial, patients at implementation sites had significantly higher MPRs (p = .01) than patients at control sites, and by extrapolation higher QALYs (0.00188). Total costs (implementation, intervention) were nonsignificantly higher ($63.76) at implementation sites. The incremental population-level cost-effectiveness ratio was $33,905.92/QALY (bootstrap interquartile range -$45,343.10/QALY to $99,260.90/QALY). Conclusions. The methodology was feasible to operationalize and gave reasonable estimates of implementation value. Key Words. Cost-effectiveness, implementation, depression, Determining the value of deploying evidence-based practices (EBPs) into routine clinical care is critical to policy makers, but cost-effectiveness is rarely assessed in implementation trials. An intervention demonstrated to be [...]

Objective. To illustrate how the analysis of bimodal U-shaped distributed utilization can be modeled with beta-binomial regression, which is rarely used in health services research. Data Sources/Study Setting. Veterans Affairs (VA) administrative data and Medicare claims in 2001-2004 for 11,123 Medicare-eligible VA primary care users in 2000. Study Design. We compared means and distributions of VA reliance (the proportion of all VA/Medicare primary care visits occurring in VA) predicted from beta-binomial, binomial, and ordinary least-squares (OLS) models. Principal Findings. Beta-binomial model fits the bimodal distribution of VA reliance better than binomial and OLS models due to the nondependence on normality and the greater flexibility in shape parameters. Conclusions. Increased awareness of beta-binomial regression may help analysts apply appropriate methods to outcomes with bimodal or U-shaped distributions. Key Words. Beta-binomial, bimodal, veterans, primary care, Medicare, utilization, There are a number of situations in which health services researchers can encounter data with a bimodal (U-shaped) distribution. When such a situation arises, it may be beneficial to adopt [...]

Objective. To demonstrate the value of shrinkage estimators when calculating a composite quality measure as the weighted average of a set of individual quality indicators. Data Sources. Rates of 28 quality indicators (QIs) calculated from the minimum dataset from residents of 112 Veterans Health Administration nursing homes in fiscal years 2005-2008. Study Design. We compared composite scores calculated from the 28 QIs using both observed rates and shrunken rates derived from a Bayesian multivariate normal-binomial model. Principal Findings. Shrunken-rate composite scores, because they take into account unreliability of estimates from small samples and the correlation among QIs, have more intuitive appeal than observed-rate composite scores. Facilities can be profiled based on more policy-relevant measures than point estimates of composite scores, and interval estimates can be calculated without assuming the QIs are independent. Usually, shrunken-rate composite scores in 1 year are better able to predict the observed total number of QI events or the observed-rate composite scores in the following year than the initial year observed-rate composite scores. Conclusion. Shrinkage estimators can be useful when a composite measure is conceptualized as a formative construct. Key Words. Composite measures, Bayesian models, quality indicators, The number of indicators developed to measure quality of patient care has expanded rapidly as pressures to improve quality have increased. Some of these indicators are different measures of the [...]