Search

Objective . Very few predictive models have been externally validated in a prospective cohort following the implementation of an artificial intelligence analytic system. This type of real-world validation is critically important due to the risk of data drift, or changes in data definitions or clinical practices over time, that could impact model performance in contemporaneous real-world cohorts. In this work, we report the model performance of a predictive analytics tool developed before COVID-19 and demonstrate model performance during the COVID-19 pandemic. Approach . The analytic system (CoMETⓇ, Nihon Kohden Digital Health Solutions LLC, Irvine, CA) was implemented in a randomized controlled trial that enrolled 10 422 patient visits in a 1:1 display-on display-off design. The CoMET scores were calculated for all patients but only displayed in the display-on arm. Only the control/display-off group is reported here because the scores could not alter care patterns. Main results. Of the 5184 visits in the display-off arm, 311 experienced clinical deterioration and care escalation, resulting in transfer to the intensive care unit, primarily due to respiratory distress. The model performance of CoMET was assessed based on areas under the receiver operating characteristic curve, which ranged from 0.725 to 0.737. Significance. The models were well-calibrated, and there were dynamic increases in the model scores in the hours preceding the clinical deterioration events. A hypothetical alerting strategy based on a rise in score and duration of the rise would have had good performance, with a positive predictive value more than 10-fold the event rate. We conclude that predictive statistical models developed five years before study initiation had good model performance despite the passage of time and the impact of the COVID-19 pandemic., (Creative Commons Attribution license.)

Background: In randomized clinical trials among critically ill patients, it is uncertain how choices regarding the measurement and analysis of nonmortal outcomes measured in terms of duration, such as intensive care unit (ICU) length of stay (LOS), affect studies' conclusions., Objectives: Assess the definitions and analytic methods used for ICU LOS analyses in published randomized clinical trials., Research Design: This is a systematic review and statistical simulation study., Results: Among the 80 of 150 trials providing sufficient information regarding the chosen definition of ICU LOS, 3 different start times (ICU admission, trial enrollment/randomization, receipt of intervention) and 2 end times (discharge readiness, actual discharge) were used. In roughly three quarters of these studies, ICU LOS was compared using approaches that did not explicitly account for death, either by ignoring it entirely or stratifying the analyses by survival status. The remaining studies used time-to-event (discharge) models censoring at death or applied a fixed LOS value to patients who died. In statistical simulations, we showed that each analytic approach tested a different question regarding ICU LOS, and that approaches that do not explicitly account for death often produce misleading or ambiguous conclusions when treatments produce small effects on mortality, even if those are not detected as significant in the trial., Conclusions: There is considerable variability in how ICU LOS is measured and analyzed which impairs the ability to compare results across trials and can produce spurious conclusions. Analyses of duration-based outcomes such as LOS should jointly assess the impact of the intervention on mortality to yield correct interpretations.

Clinical endpoints measured in terms of duration, such as intensive care unit (ICU) length of stay (LOS), are widely used in randomized clinical trials (RCTs) and observational research. In analyses of patient-level data from a recent RCT, in which ICU LOS was the primary endpoint, and in administrative data, we showed that additional ICU time is often accrued by patients after they are deemed ready for discharge. This "immutable" time (which cannot plausibly be altered by interventions under study) varies by day, week, and year, adding on average one-third of a day to total LOS. We then used statistical simulations, informed by the administrative data and RCT, to assess the impact of immutable time on the measurement and statistical comparison of patients' ICU LOS. These simulations demonstrated that immutable time combines with clinically necessary ICU time (neither of which is likely to be normally distributed) to produce overall LOS distributions that might either mask true treatment effects or suggest false treatment effects relative to analyses of time to discharge readiness. The extent and direction of bias were complex functions of the statistical method used, mortality rates and distributions, and the magnitude of immutable time relative to intervention-associated reductions in LOS., (© The Author(s) 2017. Published by Oxford University Press on behalf of the Johns Hopkins Bloomberg School of Public Health. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.)

Importance: Predictions of long-term survival and functional outcomes influence decision making for critically ill patients, yet little is known regarding their accuracy., Objective: To determine the discriminative accuracy of intensive care unit (ICU) physicians and nurses in predicting 6-month patient mortality and morbidity, including ambulation, toileting, and cognition., Design, Setting, and Participants: Prospective cohort study conducted in 5 ICUs in 3 hospitals in Philadelphia, Pennsylvania, and enrolling patients who spent at least 3 days in the ICU from October 2013 until May 2014 and required mechanical ventilation, vasopressors, or both. These patients' attending physicians and bedside nurses were also enrolled. Follow-up was completed in December 2014., Main Outcomes and Measures: ICU physicians' and nurses' binary predictions of in-hospital mortality and 6-month outcomes, including mortality, return to original residence, ability to toilet independently, ability to ambulate up 10 stairs independently, and ability to remember most things, think clearly, and solve day-to-day problems (ie, normal cognition). For each outcome, physicians and nurses provided a dichotomous prediction and rated their confidence in that prediction on a 5-point Likert scale. Outcomes were assessed via interviews with surviving patients or their surrogates at 6 months. Discriminative accuracy was measured using positive and negative likelihood ratios (LRs), C statistics, and other operating characteristics., Results: Among 340 patients approached, 303 (89%) consented (median age, 62 years [interquartile range, 53-71]; 57% men; 32% African American); 6-month follow-up was completed for 299 (99%), of whom 169 (57%) were alive. Predictions were made by 47 physicians and 128 nurses. Physicians most accurately predicted 6-month mortality (positive LR, 5.91 [95% CI, 3.74-9.32]; negative LR, 0.41 [95% CI, 0.33-0.52]; C statistic, 0.76 [95% CI, 0.72-0.81]) and least accurately predicted cognition (positive LR, 2.36 [95% CI, 1.36-4.12]; negative LR, 0.75 [95% CI, 0.61-0.92]; C statistic, 0.61 [95% CI, 0.54-0.68]). Nurses most accurately predicted in-hospital mortality (positive LR, 4.71 [95% CI, 2.94-7.56]; negative LR, 0.61 [95% CI, 0.49-0.75]; C statistic, 0.68 [95% CI, 0.62-0.74]) and least accurately predicted cognition (positive LR, 1.50 [95% CI, 0.86-2.60]; negative LR, 0.88 [95% CI, 0.73-1.06]; C statistic, 0.55 [95% CI, 0.48-0.62]). Discriminative accuracy was higher when physicians and nurses were confident about their predictions (eg, for physicians' confident predictions of 6-month mortality: positive LR, 33.00 [95% CI, 8.34-130.63]; negative LR, 0.18 [95% CI, 0.09-0.35]; C statistic, 0.90 [95% CI, 0.84-0.96]). Compared with a predictive model including objective clinical variables, a model that also included physician and nurse predictions had significantly higher discriminative accuracy for in-hospital mortality, 6-month mortality, and return to original residence (P < .01 for all)., Conclusions and Relevance: ICU physicians' and nurses' discriminative accuracy in predicting 6-month outcomes of critically ill patients varied depending on the outcome being predicted and confidence of the predictors. Further research is needed to better understand how clinicians derive prognostic estimates of long-term outcomes.

Importance: Although the end-of-life care patients receive is known to vary across nations, regions, and centers, these differences are best explored within a group of patients with presumably similar care preferences., Objective: To examine the proportions of patients admitted to the intensive care unit (ICU) with limitations on life-sustaining treatments and the proportions of such patients who receive aggressive care across individual ICUs., Design, Settings, and Participants: Retrospective cohort study using the Project IMPACT database (from April 1, 2001, to December 31, 2008) including 141 ICUs in 105 hospitals in the United States and 277,693 ICU patient visits. We used logistic regression analysis models adjusted for available patient characteristics and clustered visits by individual ICU. The full analysis was last performed in October 2014., Main Outcomes and Measures: Outcomes included the provision of (1) cardiopulmonary resuscitation, (2) new forms of life support, and the (3) addition or (4) reversal of treatment limitations., Results: Of the ICU admissions evaluated, 4.8% (95% CI, 4.7%-4.9%) had previously established treatment limitations. Patients admitted with treatment limitations were more likely to be older with more functional limitations and comorbidities. Among patients who survived to hospital discharge, more experienced reversals of existing treatment limitations during the ICU stay (17.8% [95% CI, 17.0%-18.7%]) than additions of new limits (11.7% [95% CI, 11.1%-12.4%]) (P < .01). Among patients who died, 15.7% (95% CI, 14.7-16.8%) had received cardiopulmonary resuscitation. After risk adjustment, ICUs varied widely in the proportions of patients admitted with treatment limitations (median, 4.0%; range, <1.0%-20.9%), the proportions of those who received cardiopulmonary resuscitation (37.7% [95% CI, 3.8%-92.4%]), the proportions of new forms of life support (30.0% [95% CI, 6.0%-84.2%]), and, among survivors, the proportion who had new treatment limitations established (11.2% [95% CI, 1.9%-57.3%]) and reversal of treatment limitations during or following ICU admission (20.2% [95% CI, 1.8%-76.2%]). The observed variability could not be consistently explained using measurable center-level characteristics., Conclusions and Relevance: Intensive care units vary dramatically in how they manage care for patients admitted with treatment limitations. Among patients who survive, escalations in the aggressiveness of care are more common during the ICU stay than are de-escalations in aggressiveness. This study cannot directly measure whether care received was consistent with patients' preferences but suggests that ICU culture and physicians' practice styles contribute to the aggressiveness of care.

Background: ICU readmissions are associated with increased mortality and costs; however, it is unclear whether these outcomes are caused by readmissions or by residual confounding by illness severity. An assessment of temporal changes in ICU readmission in response to a specific policy change could help disentangle these possibilities. We sought to determine whether ICU readmission rates changed after 2003 Accreditation Council for Graduate Medical Education Resident Duty Hours reform ("reform") and whether there were temporally corresponding changes in other ICU outcomes., Methods: We used a difference-in-differences approach using Project IMPACT (Improved Methods of Patient Information Access of Core Clinical Tasks). Piecewise regression models estimated changes in outcomes immediately before and after reform in 274,491 critically ill medical and surgical patients in 151 community and academic US ICUs. Outcome measures included ICU readmission, ICU mortality, and in-hospital post-ICU-discharge mortality., Results: In ICUs with residents, ICU readmissions increased before reform (OR, 1.5; 95% CI, 1.22-1.84; P < .01), and decreased after (OR, 0.85; 95% CI, 0.73-0.98; P = .03). This abrupt decline in ICU readmissions after reform differed significantly from an increase in readmissions observed in ICUs without residents at this time (difference-in-differences P < .01). No comparable changes in mortality were observed between ICUs with vs without residents., Conclusions: The changes in ICU readmission rates after reform, without corresponding changes in mortality, suggest that ICU readmissions are not causally related to other untoward patient outcomes. Instead, ICU readmission rates likely reflect operational aspects of care that are not patient-centered, making them less useful indicators of ICU quality.

Background: The magnitude and implication of variation in end-of-life decision-making among ICUs in the United States is unknown., Methods: We reviewed data on decisions to forgo life-sustaining therapy (DFLSTs) in 269,002 patients admitted to 153 ICUs in the United States between 2001 and 2009. We used fixed-effects logistic regression to create a multivariable model for DFLST and then calculated adjusted rates of DFLST for each ICU., Results: Patient factors associated with increased odds of DFLST included advanced age, female sex, white race, and poor baseline functional status (all P < .001). However, associations with several of these factors varied among ICUs (eg, black race had an OR for DFLST from 0.18 to 2.55 across ICUs). The ICU staffing model was also found to be associated with DFLST, with an open ICU staffing model associated with an increased odds of a DFLST (OR = 1.19). The predicted probability of DFLST varied approximately sixfold among ICUs after adjustment for the fixed patient and ICU effects and was directly correlated with the standardized mortality ratios of ICUs (r = 0.53, 0.41-0.68)., Conclusion: Although patient factors explain much of the variability in DFLST practices, significant effects of ICU culture and practice influence end-of-life decision-making. The observation that an ICU's risk-adjusted propensity to withdraw life support is directly associated with its standardized mortality ratio suggests problems with using the latter as a quality measure.

Objective: Good quality indicators should have face validity, relevance to patients, and be able to be measured reliably. Beyond these general requirements, good quality indicators should also have certain statistical properties, including sufficient variability to identify poor performers, relative insensitivity to severity adjustment, and the ability to capture what providers do rather than patients' characteristics. We assessed the performance of candidate indicators of ICU quality on these criteria. Indicators included ICU readmission, mortality, several length of stay outcomes, and the processes of venous-thromboembolism and stress ulcer prophylaxis provision., Design: Retrospective cohort study., Setting: One hundred thirty-eight U.S. ICUs from 2001-2008 in the Project IMPACT database., Patients: Two hundred sixty-eight thousand eight hundred twenty-four patients discharged from U.S. ICUs., Interventions: None., Measurements and Main Results: We assessed indicators' (1) variability across ICU-years; (2) degree of influence by patient vs. ICU and hospital characteristics using the Omega statistic; (3) sensitivity to severity adjustment by comparing the area under the receiver operating characteristic curve (AUC) between models including vs. excluding patient variables, and (4) correlation between risk adjusted quality indicators using a Spearman correlation. Large ranges of among-ICU variability were noted for all quality indicators, particularly for prolonged length of stay (4.7-71.3%) and the proportion of patients discharged home (30.6-82.0%), and ICU and hospital characteristics outweighed patient characteristics for stress ulcer prophylaxis (ω, 0.43; 95% CI, 0.34-0.54), venous thromboembolism prophylaxis (ω, 0.57; 95% CI, 0.53-0.61), and ICU readmissions (ω, 0.69; 95% CI, 0.52-0.90). Mortality measures were the most sensitive to severity adjustment (area under the receiver operating characteristic curve % difference, 29.6%); process measures were the least sensitive (area under the receiver operating characteristic curve % differences: venous thromboembolism prophylaxis, 3.4%; stress ulcer prophylaxis, 2.1%). None of the 10 indicators was clearly and consistently correlated with a majority of the other nine indicators., Conclusions: No indicator performed optimally across assessments. Future research should seek to define and operationalize quality in a way that is relevant to both patients and providers.

Rationale: Intensive care unit (ICU)-based randomized clinical trials (RCTs) among adult critically ill patients commonly fail to detect treatment benefits., Objectives: Appraise the rates of success, outcomes used, statistical power, and design characteristics of published trials., Methods: One hundred forty-six ICU-based RCTs of diagnostic, therapeutic, or process/systems interventions published from January 2007 to May 2013 in 16 high-impact general or critical care journals were studied., Measurement and Main Results: Of 146 RCTs, 54 (37%) were positive (i.e., the a priori hypothesis was found to be statistically significant). The most common primary outcomes were mortality (n = 40 trials), infection-related outcomes (n = 33), and ventilation-related outcomes (n = 30), with positive results found in 10, 58, and 43%, respectively. Statistical power was discussed in 135 RCTs (92%); 92 cited a rationale for their power parameters. Twenty trials failed to achieve at least 95% of their reported target sample size, including 11 that were stopped early due to insufficient accrual/logistical issues. Of 34 superiority RCTs comparing mortality between treatment arms, 13 (38%) accrued a sample size large enough to find an absolute mortality reduction of 10% or less. In 22 of these trials the observed control-arm mortality rate differed from the predicted rate by at least 7.5%., Conclusions: ICU-based RCTs are commonly negative and powered to identify what appear to be unrealistic treatment effects, particularly when using mortality as the primary outcome. Additional concerns include a lack of standardized methods for assessing common outcomes, unclear justifications for statistical power calculations, insufficient patient accrual, and incorrect predictions of baseline event rates.

Rationale: Time-varying demand for critical care may strain the capacities of intensive care units (ICUs) to provide optimal care. Intensivists and ICU nurses may be the best judges of the strain on their ICU. Yet, it is not clear what ICU and hospital factors contribute to this perceived sense of strain among ICU providers., Objectives: To identify measureable ICU and hospital factors associated with perceived strain by intensivists and ICU nurses., Methods: During a 6-month prospective cohort study, we surveyed nurses and physicians responsible for bed management regarding the ability of a 24-bed medical ICU (MICU) to provide optimal critical care. We simultaneously assessed time-varying ICU-level factors, including patient census, number of admissions, average patient acuity, number of interhospital transfer requests, and censuses of other hospital units. To identify factors associated with strain, we used an algorithm for covariate selection in regression models that selects variables that contribute sufficiently to model prediction to justify their inclusion., Measurements and Main Results: Of 254 surveys, 226 (89%) were completed by 18 charge nurses and 17 physicians. On a scale of 1 to 10 (where a higher score indicated more strain), the median perceived strain score among nurses was 6 (interquartile range, 3-7) and among physicians was 5 (interquartile range, 3-7), with moderate correlation within days (interclass correlation coefficient, 0.45; 95% confidence interval: 0.30, 0.60). Average patient acuity, MICU census, number of MICU admissions, and general ward census were included in the most efficient model of strain perceived by nurses. Only MICU census was strongly associated with strain perceived by physicians., Conclusions: A model containing commonly available metrics of ICU census, average patient acuity, and the proportion of new admissions has validity as a model of ICU nurses' perceived ICU capacity strain. However, only ICU census was associated with increased perceived capacity strain by physicians, highlighting the need for involvement of multiple stakeholder groups to improve our understanding of ICU capacity strain.

Rationale: The aging population may strain intensive care unit (ICU) capacity and adversely affect patient outcomes. Existing fluctuations in demand for ICU care offer an opportunity to explore such relationships., Objectives: To determine whether transient increases in ICU strain influence patient mortality, and to identify characteristics of ICUs that are resilient to surges in capacity strain., Methods: Retrospective cohort study of 264,401 patients admitted to 155 U.S. ICUs from 2001 to 2008. We used logistic regression to examine relationships of measures of ICU strain (census, average acuity, and proportion of new admissions) near the time of ICU admission with mortality., Measurements and Main Results: A total of 36,465 (14%) patients died in the hospital. ICU census on the day of a patient's admission was associated with increased mortality (odds ratio [OR], 1.02 per standardized unit increase; 95% confidence interval [CI]: 1.00, 1.03). This effect was greater among ICUs employing closed (OR, 1.07; 95% CI: 1.02, 1.12) versus open (OR, 1.01; 95% CI: 0.99, 1.03) physician staffing models (interaction P value = 0.02). The relationship between census and mortality was stronger when the census was composed of higher acuity patients (interaction P value < 0.01). Averaging strain over the first 3 days of patients' ICU stays yielded similar results except that the proportion of new admissions was now also associated with mortality (OR, 1.04 for each 10% increase; 95% CI: 1.02, 1.06)., Conclusions: Several sources of ICU strain are associated with small but potentially important increases in patient mortality, particularly in ICUs employing closed staffing models. Although closed ICUs may promote favorable outcomes under static conditions, they are susceptible to being overwhelmed by patient influxes.

Background: Strains on the capacities of intensive care units (ICUs) may influence the quality of ICU-to-floor transitions., Objective: To determine how 3 metrics of ICU capacity strain (ICU census, new admissions, and average acuity) measured on days of patient discharges influence ICU length of stay (LOS) and post-ICU discharge outcomes., Design: Retrospective cohort study from 2001 to 2008., Setting: 155 ICUs in the United States., Patients: 200 730 adults discharged from ICUs to hospital floors., Measurements: Associations between ICU capacity strain metrics and discharged patient ICU LOS, 72-hour ICU readmissions, subsequent in-hospital death, post-ICU discharge LOS, and hospital discharge destination., Results: Increases in the 3 strain variables on the days of ICU discharge were associated with shorter preceding ICU LOS (all P < 0.001) and increased odds of ICU readmissions (all P < 0.050). Going from the 5th to 95th percentiles of strain was associated with a 6.3-hour reduction in ICU LOS (95% CI, 5.3 to 7.3 hours) and a 1.0% increase in the odds of ICU readmission (CI, 0.6% to 1.5%). No strain variable was associated with increased odds of subsequent death, reduced odds of being discharged home from the hospital, or longer total hospital LOS., Limitation: Long-term outcomes could not be measured., Conclusion: When ICUs are strained, triage decisions seem to be affected such that patients are discharged from the ICU more quickly and, perhaps consequentially, have slightly greater odds of being readmitted to the ICU. However, short-term patient outcomes are unaffected. These results suggest that bed availability pressures may encourage physicians to discharge patients from the ICU more efficiently and that ICU readmissions are unlikely to be causally related to patient outcomes., Primary Funding Source: Agency for Healthcare Research and Quality; National Heart, Lung, and Blood Institute; and Society of Critical Care Medicine.

Background: Intensive care unit (ICU) readmission rates are commonly viewed as indicators of ICU quality. However, definitions of ICU readmissions vary, and it is unknown which, if any, readmissions are associated with ICU quality., Objective: Empirically derive the optimal interval between ICU discharge and readmission for purposes of considering ICU readmission as an ICU quality indicator., Research Design: Retrospective cohort study., Subjects: A total of 214,692 patients discharged from 157 US ICUs participating in the Project IMPACT database, 2001-2008., Measures: We graphically examined how patient characteristics and ICU discharge circumstances (eg, ICU census) were related to the odds of ICU readmissions as the allowable interval between ICU discharge and readmission was lengthened. We defined the optimal interval by identifying inflection points where these relationships changed significantly and permanently., Results: A total of 2242 patients (1.0%) were readmitted to the ICU within 24 hours; 9062 (4.2%) within 7 days. Patient characteristics exhibited stronger associations with readmissions after intervals >48-60 hours. By contrast, ICU discharge circumstances and ICU interventions (eg, mechanical ventilation) exhibited weaker relationships as intervals lengthened, with inflection points at 30-48 hours. Because of the predominance of afternoon readmissions regardless of time of discharge, using intervals defined by full calendar days rather than fixed numbers of hours produced more valid results., Discussion: It remains uncertain whether ICU readmission is a valid quality indicator. However, having established 2 full calendar days (not 48 h) after ICU discharge as the optimal interval for measuring ICU readmissions, this study will facilitate future research designed to determine its validity.

Rationale: The incidence of intensive care unit (ICU) readmissions across the United States is unknown., Objectives: To determine incidence of ICU readmissions in United States hospitals, and describe the distribution of time between ICU discharges and readmissions., Methods: This retrospective cohort study used 196,202 patients in 156 medical and surgical ICUs in 106 community and academic hospitals participating in Project IMPACT from April 1, 2001, to December 31, 2007. We used mixed-effects logistic regression, adjusting for patient and hospital characteristics, to describe how ICU readmission rates differed across patient types, ICU models, and hospital types., Measurements and Main Results: Measurements consisted of 48- and 120-hour ICU readmission rates and time to readmission. A total of 3,905 patients (2%) were readmitted to the ICU within 48 hours, and 7,171 (3.7%) within 120 hours. In adjusted analysis, there was no difference in ICU readmissions across patient types or ICU models. Among medical patients, those in academic hospitals had higher odds of 48- and 120-hour readmission than patients in community hospitals without residents (1.51 [95% confidence interval, 1.12-2.02] and 1.63 [95% confidence interval, 1.24-2.16]). Median time to ICU readmission was 3.07 days (interquartile range, 1.27-6.58). Closed ICUs had the longest times to readmission (3.55 d [interquartile range, 1.42-7.50])., Conclusions: Approximately 2% and 4% of ICU patients discharged to the ward are readmitted within 48 and 120 hours, within a median time of 3 days. Medical patients in academic hospitals are more likely to be readmitted than patients in community hospitals without residents. ICU readmission rates could be useful for policy makers and investigations into their causes and consequences.