Search

Objective: Early detection of clinical deterioration using machine learning early warning scores may improve outcomes. However, most implemented scores were developed using logistic regression, only underwent retrospective validation, and were not tested in important subgroups. Our objective was to develop and prospectively validate a gradient boosted machine model (eCARTv5) for identifying clinical deterioration on the wards., Design: Multicenter retrospective and prospective observational study., Setting: Inpatient admissions to the medical-surgical wards at seven hospitals in three health systems for model development (2006-2022) and at 21 hospitals from three health systems for retrospective (2009-2023) and prospective (2023-2024) external validation., Patients: All adult patients hospitalized at each participating health system during the study years., Interventions: None MEASUREMENTS AND MAIN RESULTS: Predictor variables (demographics, vital signs, documentation, and laboratory values) were used in a gradient boosted trees algorithm to predict intensive care unit transfer or death in the next 24 hours. The developed model (eCART) was compared to the Modified Early Warning Score (MEWS) and the National Early Warning Score (NEWS) using the area under the receiver operating characteristic curve (AUROC). The development cohort included 901,491 admissions, the retrospective validation cohort included 1,769,461 admissions, and the prospective validation cohort included 205,946 admissions. In retrospective validation, eCART had the highest AUROC (0.835; 95%CI 0.834, 0.835), followed by NEWS (0.766 (95%CI 0.766, 0.767)), and MEWS (0.704 (95%CI 0.703, 0.704)). eCART's performance remained high (AUROC ≥0.80) across a range of patient demographics, clinical conditions, and during prospective validation., Conclusions: We developed eCART, which performed better than the NEWS and MEWS retrospectively, prospectively, and across a range of subgroups. These results served as the foundation for Food and Drug Administration clearance for its use in identifying deterioration in hospitalized ward patients.

Background and Objective: To develop the COVid Veteran (COVet) score for clinical deterioration in Veterans hospitalized with COVID-19 and further validate this model in both Veteran and non-Veteran samples. No such score has been derived and validated while incorporating a Veteran sample., Derivation Cohort: Adults (age ≥ 18 yr) hospitalized outside the ICU with a diagnosis of COVID-19 for model development to the Veterans Health Administration (VHA) (n = 80 hospitals)., Validation Cohort: External validation occurred in a VHA cohort of 34 hospitals, as well as six non-Veteran health systems for further external validation (n = 21 hospitals) between 2020 and 2023., Prediction Model: eXtreme Gradient Boosting machine learning methods were used, and performance was assessed using the area under the receiver operating characteristic curve and compared with the National Early Warning Score (NEWS). The primary outcome was transfer to the ICU or death within 24 hours of each new variable observation. Model predictor variables included demographics, vital signs, structured flowsheet data, and laboratory values., Results: A total of 96,908 admissions occurred during the study period, of which 59,897 were in the Veteran sample and 37,011 were in the non-Veteran sample. During external validation in the Veteran sample, the model demonstrated excellent discrimination, with an area under the receiver operating characteristic curve of 0.88. This was significantly higher than NEWS (0.79; p < 0.01). In the non-Veteran sample, the model also demonstrated excellent discrimination (0.86 vs. 0.79 for NEWS; p < 0.01). The top three variables of importance were eosinophil percentage, mean oxygen saturation in the prior 24-hour period, and worst mental status in the prior 24-hour period., Conclusions: We used machine learning methods to develop and validate a highly accurate early warning score in both Veterans and non-Veterans hospitalized with COVID-19. The model could lead to earlier identification and therapy, which may improve outcomes., Competing Interests: Dr. Churpek and Alexandra Spicer have funding from the Department of Defense: Peer Reviewed Medical Research Program (PRMRP) W81XWH-21-1-0009. Drs. Churpek and Edelson both have patents for electronic Cardiac Arrest Risk Triage (eCART), a non-Veteran deterioration analytic. Dr. Edelson has ownership interest in Quant HC and is president/cofounder of AgileMD with licensing agreements with Philips Healthcare and EarlySense. The remaining authors have disclosed that they do not have any potential conflicts of interest.