Search

Background: Incorporating the patient's perspective to evaluate national surgical quality has yet to be achieved in the United States and represents a tremendous unrealized opportunity for continuous quality improvement. The first phase of a multiphase pilot to measure patient-reported outcomes (PROs) using the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) is described., Methods: After conducting a baseline knowledge assessment and stakeholder engagement activities, a pilot to capture PROs was implemented using the ACS NSQIP platform. Hospital participation was voluntary. A 45-item survey was compiled from three validated PRO measures and administered electronically to patients accrued into the ACS NSQIP registry. Constructs measured were overall health-related quality of life, pain interference, and experiences of care. Lessons learned during implementation were compiled, and preliminary data analyses were conducted., Results: Between October 2017 and March 2018, 1,324 patients from 17 hospitals in the United States and Canada responded to the survey. Responding patients underwent 348 different types of operations, varying in complexity and riskiness. The median age of respondents was 63 years (interquartile range: 52-70), and the survey took on average 6.4 minutes to complete. Hospital response rates ranged from 10% to 50% with a 20% median., Conclusion: The implementation of this first phase demonstrates that the measurement of PROs is possible using the ACS NSQIP and that the barriers-perceived or real-can be reasonably overcome., (Copyright © 2018 The Joint Commission. Published by Elsevier Inc. All rights reserved.)

Background: Efforts to improve healthcare quality involve profiling hospitals and providers. Whether cancer-specific measures can be used reliably for profiling purposes has not been reported., Study Design: Hospitals and surgeons were profiled with 3 measures assessing the adequacy of lymphadenectomy for colon (ie at least 12 regional lymph nodes [12RLN] are removed and pathologically examined for resected colon cancer), gastric (ie at least 15 regional lymph nodes [G15RLN] are removed and pathologically examined for resected gastric cancer), and non-small cell lung (ie at least 10 regional lymph nodes [10RLN] are removed and pathologically examined for American Joint Committee on Cancer stage IA, IB, IIA, and IIB resected non-small cell lung cancer) cancers using hierarchical models. National Cancer Data Base cases spanning 2010 to 2013 were included if they met measure eligibility. Reliability estimates for hospital and surgeon performance across cumulative years of data (2013, 2012 to 2013, 2011 to 2013, and 2010 to 2013) were calculated with and without risk adjustment. Surgeon caseload minimums were projected to achieve reliabilities of 0.40 and 0.70., Results: Reliability estimates tended to increase with longer periods of data collection but at different rates, depending on measure, level of aggregation, and performance outlier status. Profiling hospitals using 12RLN with 2 years of data yielded a median reliability of 0.72 (interquartile range [IQR] 0.55 to 0.83); however, 4 years of data yielded a median reliability of only 0.31 (IQR 0.14 to 0.54) for surgeons. The G15RLN performance was poor overall; 10RLN had high reliability at both hospital (0.74; IQR 0.50 to 0.86) and surgeon (0.61; IQR 0.34 to 0.80) levels using 1 year of data, but the literature questions this measure's validity. Few surgeons could achieve appropriate levels of reliability regardless of increased data collection duration., Conclusions: Profiling hospitals based on measures such as these can achieve acceptable reliability in reasonable timeframes, but does not always. Either lower levels of reliability should be accepted to profile surgeons with these measures or longer timeframes should be used., (Copyright © 2016 American College of Surgeons. Published by Elsevier Inc. All rights reserved.)

Background: The American College of Surgeons, National Surgical Quality Improvement Program (ACS NSQIP) surgical quality feedback models are recalibrated every 6 months, and each hospital is given risk-adjusted, hierarchical model, odds ratios that permit comparison to an estimated average NSQIP hospital at a particular point in time. This approach is appropriate for "relative" benchmarking, and for targeting quality improvement efforts, but does not permit evaluation of hospital or program-wide changes in quality over time. We report on long-term improvement in surgical outcomes associated with participation in ACS NSQIP., Study Design: ACS NSQIP data (2006-2013) were used to create prediction models for mortality, morbidity (any of several distinct adverse outcomes), and surgical site infection (SSI). For each model, for each hospital, and for year of first participation (hospital cohort), hierarchical model observed/expected (O/E) ratios were computed. The primary performance metric was the within-hospital trend in logged O/E ratios over time (slope) for mortality, morbidity, and SSI., Results: Hospital-averaged log O/E ratio slopes were generally negative, indicating improving performance over time. For all hospitals, 62%, 70%, and 65% of hospitals had negative slopes for mortality, morbidity, and any SSI, respectively. For hospitals currently in the program for at least 3 years, 69%, 79%, and 71% showed improvement in mortality, morbidity, and SSI, respectively. For these hospitals, we estimate 0.8%, 3.1%, and 2.6% annual reductions (with respect to prior year's rates) for mortality, morbidity, and SSI, respectively., Conclusions: Participation in ACS NSQIP is associated with reductions in adverse events after surgery. The magnitude of quality improvement increases with time in the program.

Objective: To assess statistical reliability of hospital profiling models in ACS NSQIP (American College of Surgeons' National Surgical Quality Improvement Program)., Background: The ACS NSQIP January 2013 Semiannual Report provided risk-adjusted hospital quality assessments for 137 models., Methods: Median reliability and percentage of hospitals achieving acceptable reliability were computed for each model. Average median reliability was computed across models with common outcomes., Results: Median reliability varied across the 137 models, from a high of 0.91 for "All Cases Morbidity" to a low of 0.005 for "Procedure-Targeted Total Hip Arthroplasty Surgical Site Infection." Generally, reliability was greatest for models with larger sample sizes and higher outcome event rates. Among "Essentials" models, 72% attained a median reliability of 0.40 or more, and 24% of 0.70 or more. Among "Procedure-Targeted" models, 29% attained a median reliability of 0.40 or more, and 3% of 0.70 or more. Percentage of hospitals achieving an acceptable reliability of 0.40 ranged from 98% for "All Cases Morbidity" to 0% for "Procedure-Targeted Pancreatectomy Mortality." For Essentials models, average median reliability for each outcome, except mortality, was more than 0.40. However, for Procedure-Targeted models the average median was less than 0.40., Conclusions: For a large proportion of ACS NSQIP Essentials models, statistical reliability is adequate for assessing surgical quality and differentiating hospital performance. The Procedure-Targeted program is evolving in terms of statistical reliability, with promising results to date. These results also argue for broader discussions of statistical reliability in performance assessments for the profession.

Objective: To compare the classification of hospital statistical outlier status as better or worse performance than expected for postoperative complications using Medicare claims versus clinical registry data., Background: Controversy remains as to the most favorable data source for measuring postoperative complications for pay-for-performance and public reporting polices., Methods: Patient-level records (2005-2008) were linked between the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) and Medicare inpatient claims. Hospital statistical outlier status for better or worse performance than expected was assessed using each data source for superficial surgical site infection (SSI), deep/organ-space SSI, any SSI, urinary tract infection, pneumonia, sepsis, deep venous thrombosis, pulmonary embolism, venous thromboembolism, and myocardial infarction by developing hierarchical multivariable logistic regression models. Kappa statistics and correlation coefficients assessed agreement between the data sources., Results: A total of 192 hospitals with 110,987 surgical patients were included. Agreement on hospital rank for complication rates between Medicare claims and ACS-NSQIP was poor-to-moderate (weighted κ: 0.18-0.48). Of hospitals identified as statistical outliers for better or worse performance by Medicare claims, 26% were also identified as outliers by ACS-NSQIP. Of outliers identified by ACS-NSQIP, 16% were also identified as outliers by Medicare claims. Agreement between the data sources on hospital outlier status classification was uniformly poor (weighted κ: -0.02-0.34)., Conclusions: Despite using the same statistical methodology with each data source, classification of hospital outlier status as better or worse performance than expected for postoperative complications differed substantially between ACS-NSQIP and Medicare claims.

Importance: Surgical site infection (SSI) has emerged as the leading publicly reported surgical outcome and is tied to payment determinations. Many hospitals monitor SSIs using the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP), in addition to mandatory participation (for most states) in the Centers for Disease Control and Prevention's National Healthcare Safety Network (NHSN), which has resulted in duplication of effort and incongruent data., Objective: To identify discrepancies in the implementation of the NHSN and the ACS NSQIP at hospitals that may be affecting the respective SSI rates., Design, Setting, and Participants: A pilot sample of hospitals that participate in both the NHSN and the ACS NSQIP., Interventions: For each hospital, observed rates and risk-adjusted observed to expected ratios for year 2012 colon SSIs were collected from both programs. The implementation methods of both programs were identified, including telephone interviews with infection preventionists who collect data for the NHSN at each hospital., Main Outcomes and Measures: Collection methods and colon SSI rates for the NHSN at each hospital were compared with those of the ACS NSQIP., Results: Of 16 hospitals, 11 were teaching hospitals with at least 500 beds. The mean observed colon SSI rates were dissimilar between the 2 programs, 5.7% (range, 2.0%-14.5%) for the NHSN vs 13.5% (range, 4.6%-26.7%) for the ACS NSQIP. The mean difference between the NHSN and the ACS NSQIP was 8.3% (range, 1.6%-18.8%), with the ACS NSQIP rate always higher. The correlation between the observed to expected ratios for the 2 programs was nonsignificant (Pearson product moment correlation, ρ = 0.4465; P = .08). The NHSN collection methods were dissimilar among interviewed hospitals. An SSI managed as an outpatient case would usually be missed under the current NHSN practices., Conclusions and Relevance: Colon SSI rates from the NHSN and the ACS NSQIP cannot be used interchangeably to evaluate hospital performance and determine reimbursement. Hospitals should not use the ACS NSQIP colon SSI rates for the NHSN reports because that would likely result in the hospital being an outlier for performance. It is imperative to reconcile SSI monitoring, develop consistent definitions, and establish one reliable method. The current state hinders hospital improvement efforts by adding unnecessary confusion to the already complex arena of perioperative improvement.

Objective: To determine whether risk-adjusted colorectal SSI rates are statistically reliable as hospital quality measures., Background: Policymakers use surgical site infections (SSI) for public reporting of hospital quality and pay-for-performance because they are a relatively common and costly cause of patient morbidity., Methods: Patients who underwent a colorectal procedure in 2009 were identified from the American College of Surgeons National Surgical Quality Improvement Program. We developed hierarchical multivariate logistic models for (1) superficial SSI, (2) deep/organ-space SSI, and (3) "any SSI" and compared how each model ranked hospital-level risk-adjusted performance. Statistical reliability of hospital quality measurements was estimated on a scale from 0 to 1; with 0 indicating that apparent variation between a hospital's quality measurement and the average hospital is statistically unreliable, and 1 indicating that any observed variation is due to a real difference in performance., Results: Mean reliability of hospital-level quality measurements was 0.650 for superficial, 0.404 for deep/organ-space, and 0.586 for "any SSI." Lower reliability was accounted for by relatively little variation in risk-adjusted SSI rates between hospitals and insufficient numbers of colorectal cases submitted by individual hospitals. In 2009, we estimate that 22.1% of all US hospitals performed a sufficient number of colorectal cases to report superficial SSI rates at a high standard of statistical reliability and 1.0% did for deep/organ-space SSI., Conclusions: As currently constructed, colorectal SSI quality measures might not meet a high standard of statistical reliability for most hospitals, limiting their ability to confidently differentiate high and low performance. Despite an expectation of improving statistical power, combining superficial and deep/organ-space SSI into an "any SSI" measure worsens reliability.

Objective: To develop a reliable, robust, parsimonious, risk-adjusted 30-day composite colectomy outcome measure., Background: A fundamental aspect in the pursuit of high-quality care is the development of valid and reliable performance measures in surgery. Colon resection is associated with appreciable morbidity and mortality and therefore is an ideal quality improvement target., Methods: From 2010 American College of Surgeons National Surgical Quality Improvement Program data, patients were identified who underwent colon resection for any indication. A composite outcome of death or any serious morbidity within 30 days of the index operation was established. A 6-predictor, parsimonious model was developed and compared with a more complex model with more variables. National caseload requirements were calculated on the basis of increasing reliability thresholds., Results: From 255 hospitals, 22,346 patients were accrued who underwent a colon resection in 2010, most commonly for neoplasm (46.7%). A mortality or serious morbidity event occurred in 4461 patients (20.0%). At the hospital level, the median composite event rate was 20.7% (interquartile range: 15.8%-26.3%). The parsimonious model performed similarly to the full model (Akaike information criterion: 19,411 vs 18,988), and hospital-level performance comparisons were highly correlated (R = 0.97). At a reliability threshold of 0.4, 56 annual colon resections would be required and achievable at an estimated 42% of US and 69% of American College of Surgeons National Surgical Quality Improvement Program hospitals. This 42% of US hospitals performed approximately 84% of all colon resections in the country in 2008., Conclusions: It is feasible to design a measure with a composite outcome of death or serious morbidity after colon surgery that has a low burden for data collection, has substantial clinical importance, and has acceptable reliability.

Control charts are used in industry to monitor performance and are being used increasingly in hospitals as a quality improvement tool. The authors' objective was to determine if control charts using surgical site infection (SSI) rates predict changes in outlier status for risk-adjusted SSI rates using data from a surgical registry, the American College of Surgeons National Surgical Quality Improvement Program. Control charts of monthly SSI rates for 100 hospitals were analyzed for indicators of a performance change in 2009 (vs 2008) using standard rules. Hospitals also were classified as having better, worse, or no change in outlier status for risk-adjusted SSI rates in 2009 (vs 2008). There was moderate agreement between these methods (weighted κ = 0.401). Control charts predicted nonworsening performance well (specificity = 92.9%) and identified changes in SSI performance sooner; however, they failed to identify 31.2% of hospitals with worsened outlier status. This study demonstrates that these quality measurement tools have unique strengths and weaknesses and are complementary uses of the same clinical data source.

Background: The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) empowers surgeons and medical centers to reliably collect, analyze, and act on clinically collected outcomes data. How individual ACS NSQIP leaders designated as Surgeon Champions (SC) utilize the ACS NSQIP at the hospital level and the obstacles they encounter are not well studied., Materials and Methods: All SC representing the 236 hospitals participating in the ACS NSIQP were invited to complete a survey designed to assess the role of the SC, data use, continuous quality improvement (CQI) efforts, CQI culture, and financial implications., Results: We received responses from 109 (46.2%) SC. The majority (72.5%) of SC were not compensated for their CQI efforts. Factors associated with demonstrable CQI efforts included longer duration of participation in the program, frequent meetings with clinical reviewers, frequent presentation of data to administration, compensation for Surgical Champion efforts and providing individual surgeons with feedback (all P < 0.05). Almost all SC stated ACS NSQIP data improved the quality of care that patients received at the hospital level (92.4%) and that the ACS NSQIP provided data that could not be obtained by other sources (95.2%). All SCs considered future funding for participation in the ACS NSQIP secure., Conclusions: Active use of ACS NSQIP data provide SC with demonstrable CQI by regularly reviewing data, having frequent interaction with clinical reviewers, and frequently sharing data with hospital administration and colleagues. SC thus play a key role in successful quality improvement at the hospital level., (Copyright © 2011 Elsevier Inc. All rights reserved.)

Objective: We aimed to determine whether hospital-level surgical performance was similar across outpatient and inpatient settings., Background: The majority of surgical procedures in the United States are performed in an outpatient setting but most quality improvement focuses on inpatient care., Methods: Using data from the 2006 to 2008 American College of Surgeons- National Surgical Quality Improvement Program, risk-adjusted hospital observed to expected ratios for morbidity and mortality were compared for inpatient and outpatient cases. In addition, hospital outpatient performance in each year was compared with performances in subsequent years., Results: Hospitals demonstrated variation in outcomes for outpatient morbidity with both good and poor outliers in each year. Outpatient mortality was so rare as to not support robust modeling. There was a lack of congruence between hospital performance for outpatient morbidity and either inpatient morbidity or inpatient mortality in each year, indicating that inpatient performance is not interchangeable with outpatient performance. Outpatient morbidity performance correlation between years was only moderate (correlations 0.449-0.534, all P < 0.001) indicating that although outcomes from 1 year mildly predict subsequent years, substitution of data would likely lead to missed opportunities for improvement., Conclusions: Assessments of risk-adjusted hospital-level outpatient morbidity performance demonstrate (1) variability across American College of Surgeons- National Surgical Quality Improvement Program sites; (2) a lack of congruence between outpatient morbidity performance and either inpatient morbidity or mortality performance; (3) year-to-year variation of outpatient morbidity performance at individual institutions. Continuing evaluation of both outpatient and inpatient outcomes is supported. Given the substantial volume of outpatient care delivered, outpatient assessments are likely to be an important component of ongoing quality improvement efforts.

Background: There has been a long-standing desire to implement a multi-institutional, multispecialty program to address surgical quality improvement for children. This report documents results of the initial phase of the American College of Surgeons National Surgical Quality Improvement Program Pediatric., Study Design: From October 2008 to December 2009, patients from 4 pediatric referral centers were sampled using American College of Surgeons National Surgical Quality Improvement Program methodology tailored to children., Results: A total of 7,287 patients were sampled, representing general/thoracic surgery (n = 2,237; 30.7%), otolaryngology (n = 1,687; 23.2%), orthopaedic surgery (n = 1,367; 18.8%), urology (n = 893; 12.3%), neurosurgery (n = 697; 9.6%), and plastic surgery (n = 406; 5.6%). Overall mortality rate detected was 0.3% and 287 (3.9%) patients had postoperative occurrences. After accounting for demographic, preoperative, and operative factors, occurrences were 4 times more likely in those undergoing inpatient versus outpatient procedures (odds ratio [OR] = 4.71; 95% CI, 3.01-7.35). Other factors associated with higher likelihood of postoperative occurrences included nutritional/immune history, such as preoperative weight loss/chronic steroid use (OR = 1.49; 95% CI, 1.03-2.15), as well as physiologic compromise, such as sepsis/inotrope use before surgery (OR = 1.68; 95% CI, 1.10-1.95). Operative factors associated with occurrences included multiple procedures under the same anesthetic (OR = 1.58; 95% CI, 1.21-2.06) and American Society of Anesthesiologists classification category 4/5 versus 1 (OR = 5.74; 95% CI, 2.94-11.24). Specialty complication rates varied from 1.5% for otolaryngology to 9.0% for neurosurgery (p < 0.001), with specific procedural groupings within each specialty accounting for the majority of complications. Although infectious complications were the predominant outcomes identified across all specialties, distribution of complications varied by specialty., Conclusions: Based on this initial phase of development, the highly anticipated American College of Surgeons National Surgical Quality Improvement Program Pediatric has the potential to identify outcomes of children's surgical care that can be targeted for quality improvement efforts., (Copyright © 2010 American College of Surgeons. Published by Elsevier Inc. All rights reserved.)

Background: Risk-adjusted evaluation is a key component of the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP). The purpose of this study was to improve standard ACS NSQIP risk adjustment using a novel procedure risk score., Study Design: Current Procedural Terminology codes (CPTs) represented in ACS NSQIP data were assigned to 136 procedure groups. Log odds predicted risk from preliminary logistic regression modeling generated a continuous risk score for each procedure group, used in subsequent modeling. Appropriate subsets of 271,368 patients in the 2008 ACS NSQIP were evaluated using logistic models for overall 30-day morbidity, 30-day mortality, and surgical site infection (SSI). Models were compared when including either work Relative Value Unit (RVU), RVU and the standard ACS NSQIP CPT range variable (CPT range), or RVU and the newly constructed CPT risk score (CPT risk), plus routine ACS NSQIP predictors., Results: When comparing the CPT risk models with the CPT range models for morbidity in the overall general and vascular surgery dataset, CPT risk models provided better discrimination through higher c statistics at earlier steps (0.81 by step 3 vs 0.81 by step 46), more information through lower Akaike's information criterion (127,139 vs 130,019), and improved calibration through a smaller Hosmer-Lemeshow chi-square statistic (48.76 vs 116.79). Improved model characteristics of CPT risk over CPT range were most apparent for broader patient populations and outcomes. The CPT risk and standard CPT range models were moderately consistent in identification of outliers as well as assignment of hospitals to quality deciles (weighted kappa ≥ 0.870)., Conclusions: Information from focused, clinically meaningful CPT procedure groups improves the risk estimation of ACS NSQIP models., (Copyright © 2010 American College of Surgeons. Published by Elsevier Inc. All rights reserved.)

Background: Facility-level process measure adherence is being publicly reported. However, the association between measure adherence and surgical outcomes is not well-established. Our objective was to determine the degree to which Surgical Care Improvement Project (SCIP) process measures are associated with American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) risk-adjusted outcomes., Study Design: This cross-sectional study included hospitals participating in the ACS NSQIP and SCIP (n = 200). ACS NSQIP outcomes (30-day overall morbidity, serious morbidity, surgical site infections [SSI], and mortality) and adherence to SCIP SSI-related process measures (from the Hospital Compare database) were collected from January 1, 2008, through December 31, 2008. Hospital-level correlation coefficients between compliance with 4 process measures (ie, antibiotic administration within 1 hour before incision [SCIP-1]; appropriate antibiotic prophylaxis [SCIP-2]; antibiotic discontinuation within 24 hours after surgery [SCIP-3]; and appropriate hair removal [SCIP 6]) and 4 risk-adjusted outcomes were calculated. Regression analyses estimated the contribution of process measure adherence to risk-adjusted outcomes., Results: Of 211 ACS NSQIP hospitals, 95% had data reported by Hospital Compare. Depending on the measure, hospital-level compliance ranged from 60% to 100%. Of the 16 correlations, 15 demonstrated nonsignificant associations with risk-adjusted outcomes. The exception was the relationship between SCIP-2 and SSI (p = 0.004). SCIP-1 demonstrated an intriguing but nonsignificant relationship with SSI (p = 0.08) and overall morbidity (p = 0.08). Although adherence to SCIP-2 was a significant predictor of risk-adjusted SSI (p < 0.0001) and overall morbidity (p < 0.0001), inclusion of compliance for SCIP-1 and SCIP-2 caused only slight improvement in model quality., Conclusions: Better adherence to infection-related process measures over the observed range was not significantly associated with better outcomes with one exception. Different measures of quality might be needed for surgical infection., (Copyright © 2010 American College of Surgeons. Published by Elsevier Inc. All rights reserved.)

Background: Cholecystectomy is among the most common surgical procedures performed in the United States. The current state of cholecystectomy outcomes, including variations in hospital performance, is unclear. The objective of this study is to compare the risk factors, indications, and 30-day outcomes, as well as variations in hospital performance associated with laparoscopic (LC) versus open cholecystectomy (OC) at 221 hospitals during a 4-year period., Study Design: Using the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) database (2005-2008), patients were identified who underwent cholecystectomy and related procedures (cholangiogram and/or common bile duct exploration). Four outcomes were studied, ie, 30-day overall morbidity, serious morbidity, surgical site infections, and mortality. Forward stepwise logistic regressions yielded patient-level predicted probabilities, and hospital-level observed-to-expected ratios were determined., Results: Of 65,511 patients, 58,659 (89.5%) underwent LC; 6,852 (10.5%) underwent OC. OC patients were considerably older with a higher comorbidity burden. LC patients were less likely to experience any morbidity (3.1% versus 17.8%; p < 0.0001), a serious morbidity (1.4% versus 11.1%; p < 0.0001), or a surgical site infection (1.3% versus 8.4%; p < 0.0001), and less likely to die (0.3% versus 2.8%; p < 0.0001). Observed-to-expected ratios for overall morbidity ranged from 0 to 3.55; for serious morbidity, 0 to 3.23; for surgical site infection, 0 to 7.02; for mortality, 0 to 13.05., Conclusions: Although overall incidence of adverse events is low after LC, substantial morbidity and mortality are associated with OC. Additionally, controlling for patient- and operation-related factors, considerable variations exist in hospital performance when evaluating 30-day outcomes after cholecystectomy., (Copyright 2010 American College of Surgeons. Published by Elsevier Inc. All rights reserved.)

Background: The American College of Surgeons' National Surgical Quality Improvement Program (ACS NSQIP) will soon be reporting procedure-specific outcomes, and hopes to reduce the burden of data collection by collecting fewer variables. We sought to determine whether these changes threaten the robustness of the risk adjustment of hospital quality comparisons., Study Design: We used prospective, clinical data from the ACS NSQIP from 2005 to 2007 (184 hospitals, 74,887 patients). For the 5 general surgery operations in the procedure-specific NSQIP, we compared the ability of the full model (21 variables), an intermediate model (12 variables), and a limited model (5 variables) to predict patient outcomes and to risk-adjust hospital outcomes., Results: The intermediate and limited models were comparable with the full model in all analyses. In the assessment of patient risk, the limited and full models had very similar discrimination at the patient level (C-indices for all 5 procedures combined of 0.93 versus 0.91 for mortality and 0.78 versus 0.76 for morbidity) and showed good calibration across strata of patient risk. In assessing hospital-specific outcomes, results from the limited and full-risk models were highly correlated for both mortality (range 0.94 to 0.99 across the 5 operations) and morbidity (range 0.96 to 0.99)., Conclusions: Procedure-specific hospital quality measures can be adequately risk-adjusted with a limited number of variables. In the context of the ACS NSQIP, moving to a more limited model will dramatically reduce the burden of data collection for participating hospitals., (Published by Elsevier Inc.)

Background: To quantify severity of postoperative complications based on the Accordion Severity Grading System, determine the ability of severity grading to enhance National Surgical Quality Improvement Program (NSQIP) data, and develop an aggregate measure of severity of complications (the postoperative morbidity index)., Study Design: Forty-three surgical experts rated case vignettes containing postoperative complications on a severity scale. Vignettes were based on the Accordion Severity Grading System derived from the Toronto Severity Grading System. The system was adjusted using the expert severity scale results and applied to 1 year of NSQIP outcomes (1,857 patients, 704 complications) at a large tertiary care center., Results: Experts initially distinguished the 6 grades of severity in a highly significant manner (t-test probabilities all < 0.005), with 1 exception. They rated reoperation and single-system organ failure without reoperation as similar, rather than distinct, in severity. The Accordion System was adjusted to reflect this. Distinction of grades thereafter was highly significant (t-test probabilities all < 0.005). Application to American College of Surgeons NSQIP data provided important novel insights. For example, complications in 6 American College of Surgeons NSQIP categories spanned 4 or more severity grades. Severity-weighted outcomes revealed that quantitatively the greatest burden of outcomes was due to wound infection, shock, and return to the operating room, which is not revealed by unweighted outcomes. Based on this information, an aggregate measure of severity of complications-the postoperative morbidity index-was proposed., Conclusions: Quantitative severity weighting of complications is feasible. Adjustment of American College of Surgeons NSQIP outcomes using this quantitative severity grading system provides uniquely informative representations of relative burdens of morbidities., (Copyright 2010 American College of Surgeons. Published by Elsevier Inc. All rights reserved.)

Background: Studying risk-adjusted outcomes in health care relies on statistical approaches to handling missing data. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) provides risk-adjusted assessments of surgical programs, traditionally imputing certain missing data points using a single round of multivariable imputation. Such imputation assumes that data are missing at random-without systematic bias-and does not incorporate estimation uncertainty. Alternative approaches, including using multiple imputation to incorporate uncertainty or using an indicator of missingness, can enhance robustness of evaluations., Study Design: One year of de-identified data from the ACS NSQIP, representing 117 institutions and 106,113 patients, was analyzed. Using albumin variables as the missing data modeled, several imputation/adjustment models were compared, including the traditional NSQIP imputation, a new single imputation, a multiple imputation, and use of a missing indicator., Results: Coefficients for albumin values changed under new single imputation and multiple imputation approaches. Multiple imputation resulted in increased standard errors, as expected. An indicator of missingness was highly explanatory, disproving the missing-at-random assumption. The effects of changes in approach differed for different outcomes, such as mortality and morbidity, and effects were greatest in smaller datasets. However, ultimate changes in patient risk assessment and institutional assessment were minimal., Conclusions: Newer statistical approaches to modeling missing (albumin) values result in noticeable statistical distinctions, including improved incorporation of imputation uncertainty. In addition, the missing-at-random assumption is incorrect for albumin. Despite these findings, effects on institutional assessments are small. Although effects can be most important with smaller data-sets, the current approach to imputing missing values in the ACS NSQIP appears reasonably robust.

The history and development of the NSQIP, from its inception in the Veterans Administration Health System to its implementation within the private sector sponsored by the ACS, documents the growth of a program that has substantially improved the quality of surgical care and has had a considerable influence on the culture of quality improvement in the profession. The success of the ACS NSQIP is the result of providing hospitals with rigorous, clinical data, networking opportunities, and resources to improve their risk-adjusted outcomes. In this manner, the ACS NSQIP challenges its hospitals and health care providers to continually improve the care they provide. In addition to reducing the complications and mortality experienced by patients after surgical procedures, hospitals that participate in the ACS NSQIP have seen the financial rewards of their quality improvement efforts. Continued growth of the ACS NSQIP will facilitate achievement of the primary goal surrounding the current health care reform debate: efficient, high-quality care.

Background: Data used for evaluating quality of medical care need to be of high reliability to ensure valid quality assessment and benchmarking. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) has continually emphasized the collection of highly reliable clinical data through its program infrastructure., Study Design: We provide a detailed description of the various mechanisms used in ACS NSQIP to assure collection of high quality data, including training of data collectors (surgical clinical reviewers) and ongoing audits of data reliability. For the 2005 through 2008 calendar years, inter-rater reliability was calculated overall and for individual variables using percentages of agreement between the data collector and the auditor. Variables with > 5% disagreement are flagged for educational efforts to improve accurate collection. Cohen's kappa was estimated for selected variables from the 2007 audit year., Results: Inter-rater reliability audits show that overall disagreement rates on variables have fallen from 3.15% in 2005 (the first year of public enrollment in ACS NSQIP) to 1.56% in 2008. In addition, disagreement levels for individual variables have continually improved, with 26 individual variables demonstrating > 5% disagreement in 2005, to only 2 such variables in 2008. Estimated kappa values suggest substantial or almost perfect agreement for most variables., Conclusions: The ACS NSQIP has implemented training and audit procedures for its hospital participants that are highly effective in collecting robust data. Audit results show that data have been reliable since the program's inception and that reliability has improved every year., (Copyright (c) 2010 American College of Surgeons. Published by Elsevier Inc. All rights reserved.)

Background: Length of postoperative stay (LOS) has gained increasing attention as a potential indicator of surgical efficiency. Our objective was to examine the feasibility of assessing LOS at 182 hospitals to identify institutions with outlying performance., Methods: Patients were identified who underwent colorectal surgery at 182 hospitals participating in the American College of Surgeon's National Surgical Quality Improvement Program (ACS NSQIP) from 2006 to 2007. Regression models for extended LOS (greater than the 75th percentile) were developed to identify hospitals whose ratios of observed to expected events (O/E) were significantly better (low outlier) or worse (high outlier) than expected after adjustment for case mix. To evaluate strategies for evaluating LOS that would be minimally influenced by the occurrence of complications, separate models were developed for patients categorized either by (1) the nonoccurrence or occurrence of any postoperative complication or (2) tercile of preoperative morbidity risk., Results: The 23,098 patients selected for this study were partitioned into groups without complications (0% complications), with complications (100%) or into terciles of preoperative morbidity risk (with 22.4%, 38.7%, and 60.0% of patients having complications, respectively). In general, the greater the complication rate the longer the LOS and the fewer the number of statistical outliers that were identified., Conclusions: ACS NSQIP data can provide individual hospitals with riskadjusted LOS measures that can be used to identify outlying performance and motivate quality improvement efforts.

Background: Although logistic regression has commonly been used to adjust for risk differences in patient and case mix to permit quality comparisons across hospitals, hierarchical modeling has been advocated as the preferred methodology, because it accounts for clustering of patients within hospitals. It is unclear whether hierarchical models would yield important differences in quality assessments compared with logistic models when applied to American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) data. Our objective was to evaluate differences in logistic versus hierarchical modeling for identifying hospitals with outlying outcomes in the ACS-NSQIP., Study Design: Data from ACS-NSQIP patients who underwent colorectal operations in 2008 at hospitals that reported at least 100 operations were used to generate logistic and hierarchical prediction models for 30-day morbidity and mortality. Differences in risk-adjusted performance (ratio of observed-to-expected events) and outlier detections from the two models were compared., Results: Logistic and hierarchical models identified the same 25 hospitals as morbidity outliers (14 low and 11 high outliers), but the hierarchical model identified 2 additional high outliers. Both models identified the same eight hospitals as mortality outliers (five low and three high outliers). The values of observed-to-expected events ratios and p values from the two models were highly correlated. Results were similar when data were permitted from hospitals providing < 100 patients., Conclusions: When applied to ACS-NSQIP data, logistic and hierarchical models provided nearly identical results with respect to identification of hospitals' observed-to-expected events ratio outliers. As hierarchical models are prone to implementation problems, logistic regression will remain an accurate and efficient method for performing risk adjustment of hospital quality comparisons.

In honor of the seminal contributions of Dr. Shukri Khuri to the foundation and development of the National Surgical Quality Improvement Program (NSQIP), a review of recent work and new directions within the American College of Surgeons (ACS) NSQIP is presented, according to the following outline:

Background/objective: The National Surgical Quality Improvement Program (NSQIP) has demonstrated quality improvement in the VA and pilot study of 14 academic institutions. The objective was to show that American College of Surgeons (ACS)-NSQIP helps all enrolled hospitals., Methods: ACS-NSQIP data was used to evaluate improvement in hospitals longitudinally over 3 years (2005-2007). Improvement was defined as reduction in risk-adjusted "Observed/Expected" (O/E) ratios between periods with risk adjustment held constant. Multivariable logistic regression-based adjustment was performed and included indicators for procedure groups. Additionally, morbidity counts were modeled using a negative binomial model, to estimate the number of avoided complications., Results: Multiple perspectives reflected improvement over time. In the analysis of 118 hospitals (2006-2007), 66% of hospitals improved risk-adjusted mortality (mean O/E improvement: 0.174; P < 0.05) and 82% improved risk adjusted complication rates (mean improvement: 0.114; P < 0.05). Correlations between starting O/E and improvement (0.834 for mortality, 0.652 for morbidity), as well as relative risk, revealed that initially worse-performing hospitals had more likelihood of improvement. Nonetheless, well-performing hospitals also improved. Modeling morbidity counts, 183 hospitals (2007), avoided ~9598 potential complications: ~52/hospital. Due to sampling this may represent only 1 of 5 to 1of 10 of the true total. Improvement reflected aggregate performance across all types of hospitals (academic/community, urban/rural). Changes in patient risk over time had important contributions to the effect., Conclusions: ACS-NSQIP indicates that surgical outcomes improve across all participating hospitals in the private sector. Improvement is reflected for both poor- and well-performing facilities. NSQIP hospitals appear to be avoiding substantial numbers of complications- improving care, and reducing costs. Changes in risk over time merit further study.

Background: Surgical peer review might be characterized by assessment heterogeneity., Methods: We performed a prospective, anonymous, peer review of surgeon and system performance during a morbidity and mortality conference., Results: Twenty-two cases were reviewed by a mean of 48.9 respondents each, including attendings, fellows, and residents. A mean of 50% (standard deviation, 23%) of respondents identified some quality issue in each case, reflecting high heterogeneity. The mean percentage in identifying a system issue was 27%, and in identifying a physician issue was 37%. When identifying a physician issue, physician care was judged as appropriate by 72%, as controversial by 26%, or as inappropriate by 2%. Residents were more likely than attendings to identify system issues (odds ratio, 2.23) and physician issues (odds ratio, 3.58), but attendings were more likely to rate care controversial or inappropriate (odds ratio, 2.53)., Conclusions: Surgical peer reviews, even after group discussion, display substantial heterogeneity. Review methods should account for this heterogeneity.

Background: Surgical decision-making and informed patient consent both benefit from having accurate information about risk. But currently available risk estimating systems have one or more limitations associated with lack of specificity to operation type, size of sample (reliability), range of outcomes predicted, and appreciation of hospital effects., Study Design: Data from the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) patients who underwent colorectal surgery in 2006 to 2007 were used to generate logistic prediction models for 30-day morbidity, serious morbidity, and mortality. Results for these three models were then used to construct a universal multivariable model to predict risk for all three outcomes. Model performance was externally validated against 2005 data., Results: For 2006 to 2007, 28,863 patients were identified who underwent major colorectal operations at 182 hospitals. A single 15-variable predictor model exhibited discrimination (c-statistic) close to that observed for the separate models on all three outcomes. Similar discrimination was found when the 2006 to 2007 universal model was applied to 3,037 operations conducted in 2005 at 37 hospitals., Conclusions: The ACS NSQIP colorectal risk calculator allows surgeons to preoperatively provide patients with detailed information about their personal risks of overall morbidity, serious morbidity, and mortality. Because ACS NSQIP can also categorize hospitals as performing better or worse than expected (or as expected), surgeons have the opportunity to adjust risk probabilities for patients at their institutions accordingly.

Objective: To examine the influence of American Society of Anesthesiologists Physical Status Classification (ASA) and preoperative Functional Health Status (FHS) variables on risk-adjusted estimates of surgical quality and to assess whether classifications are inflated at some hospitals., Background: ASA and FHS are influential in risk-adjusted comparisons of surgical quality. However, because ASA and FHS are subjective they can be inflated, making patients appear more ill than they actually are, and crediting hospitals for a sicker patient population., Methods: We identified 28,751 colorectal surgery patients at 170 hospitals participating in the American College of Surgeon's National Surgical Quality Improvement Program (ACS NSQIP) during 2006 to 2007. Logistic regression models were developed for morbidity and mortality with and without inclusion of ASA and FHS. Hospital quality rankings from the different models were compared., Results: Morbidity and mortality rates were 24.3% and 3.9%, respectively. Percents of patients in ASA classes I through V were 3.3%, 46.4%, 41.5%,8.3%, and 0.7% and that were independent or partially or totally dependent were 89.2%, 7.2%, and 3.6%, respectively. Models that included ASA and FHS exhibited slightly better fit (Hosmer-Lemshow statistic) and discrimination(c-statistic) than models without both these variables, though magnitudes of differences were consistent with chance. There was inconsistent evidence for improper assignment of ASA and FHS., Conclusions: The small improvements in model quality when both ASA and FHS are present versus absent, suggest that they make a unique contribution to assessing severity of preoperative risk. With little indication that these subjective variables are subject to an important level of institutional bias, it is appropriate that they be used to assess risk-adjusted surgical quality. Periodic monitoring for inappropriate inflation of ASA status is warranted.

Background: Studies using Medicare data have suggested that African American race is an independent predictor of death after major surgery. We hypothesized that the apparent adverse effect of race on surgical outcomes is due to confounding by comorbidity, not race itself., Methods: We identified all non-Hispanic white and African American general surgery, private sector patients included in the National Surgery Quality Improvement Program (NSQIP) Patient Safety in Surgery Study (2001-2004). Patient characteristics, comorbidities, and postoperative outcomes were collected/analyzed using NSQIP methodology. Characteristics between races were compared using Student t and chi(2) tests. Odds ratios (OR) for 30-day morbidity and mortality were calculated using multivariable logistic regression., Results: We identified 34,141 white and 5068 African American patients. African Americans were younger but more likely to undergo emergency surgery and present with hypertension, dyspnea, diabetes, renal failure, open wounds/infection, or advanced American Society of Anesthesiology class (all P < 0.001). African Americans underwent less complex procedures but had higher unadjusted 30-day morbidity (14.33% vs. 12.35%; P < 0.001) and mortality (2.09% vs. 1.65%; P = 0.02). After controlling for comorbidity, African American race had no independent effect on mortality (OR 0.95, (0.74-1.23)) but was associated with a higher risk of postoperative cardiac arrest (OR 2.49, (1.80-3.45)) and renal insufficiency/failure (OR 1.70 (1.32-2.18))., Conclusion: African American race is associated with greater comorbidity and cardiac/renal complications but is not an independent predictor of perioperative mortality after general surgery. Efforts to improve postoperative outcomes in African Americans should focus on reducing the need for emergency surgery and improving perioperative management of comorbid conditions.