Search

Open in new tab Download slide OBJECTIVES The aim of this study was to investigate the performance of the EuroSCORE II over time and dynamics in values of predictors included in the model. METHODS A cohort study was performed using data from the Netherlands Heart Registration. All cardiothoracic surgical procedures performed between 1 January 2013 and 31 December 2019 were included for analysis. Performance of the EuroSCORE II was assessed across 3-month intervals in terms of calibration and discrimination. For subgroups of major surgical procedures, performance of the EuroSCORE II was assessed across 12-month time intervals. Changes in values of individual EuroSCORE II predictors over time were assessed graphically. RESULTS A total of 103 404 cardiothoracic surgical procedures were included. Observed mortality risk ranged between 1.9% [95% confidence interval (CI) 1.6–2.4] and 3.6% (95% CI 2.6–4.4) across 3-month intervals, while the mean predicted mortality risk ranged between 3.4% (95% CI 3.3–3.6) and 4.2% (95% CI 3.9–4.6). The corresponding observed:expected ratios ranged from 0.50 (95% CI 0.46–0.61) to 0.95 (95% CI 0.74–1.16). Discriminative performance in terms of the c -statistic ranged between 0.82 (95% CI 0.78–0.89) and 0.89 (95% CI 0.87–0.93). The EuroSCORE II consistently overestimated mortality compared to observed mortality. This finding was consistent across all major cardiothoracic surgical procedures. Distributions of values of individual predictors varied broadly across predictors over time. Most notable trends were a decrease in elective surgery from 75% to 54% and a rise in patients with no or New York Heart Association I class heart failure from 27% to 33%. CONCLUSIONS The EuroSCORE II shows good discriminative performance, but consistently overestimates mortality risks of all types of major cardiothoracic surgical procedures in the Netherlands. [ABSTRACT FROM AUTHOR]

• All studied discrimination and calibration measures independently impacted expected risk reductions (ΔNTCP) and patient selection outcomes and should thus be assessed at external validation. • AUC had the largest impact on patient selection, and it was therefore suggested, if multiple models predicting the same outcome are compared at external validation, to select the model with the highest AUC, and consider model recalibration if needed. • Some prediction errors have more impact on clinical decision making than others and thus we wish to stress the importance of treatment decision protocols, linking predictions to treatment decisions, as they facilitate investigation of impact of prediction errors on actual clinical decisions. Normal-tissue complication probability (NTCP) models predict complication risk in patients receiving radiotherapy, considering radiation dose to healthy tissues, and are used to select patients for proton therapy, based on their expected reduction in risk after proton therapy versus photon radiotherapy (ΔNTCP). Recommended model evaluation measures include area under the receiver operating characteristic curve (AUC), overall calibration (CITL), and calibration slope (CS), whose precise relation to patient selection is still unclear. We investigated how each measure relates to patient selection outcomes. The model validation and consequent patient selection process was simulated within empirical head and neck cancer patient data. By manipulating performance measures independently via model perturbations, the relation between model performance and patient selection was studied. Small reductions in AUC (-0.02) yielded mean changes in ΔNTCP between 0.9–3.2 %, and single-model patient selection differences between 2–19 %. Deviations (-0.2 or +0.2) in CITL or CS yielded mean changes in ΔNTCP between 0.3–1.4 %, and single-model patient selection differences between 1–10 %. Each measure independently impacts ΔNTCP and patient selection and should thus be assessed in a representative sufficiently large external sample. Our suggested practical model selection approach is considering the model with the highest AUC, and recalibrating it if needed. [ABSTRACT FROM AUTHOR]

Background: General practitioners (GPs) often assess patients with acute infections. It is challenging for GPs to recognize patients needing immediate hospital referral for sepsis while avoiding unnecessary referrals. This study aimed to predict adverse sepsis-related outcomes from telephone triage information of patients presenting to out-of-hours GP cooperatives. Methods: A retrospective cohort study using linked routine care databases from out-of-hours GP cooperatives, general practices, hospitals and mortality registration. We included adult patients with complaints possibly related to an acute infection, who were assessed (clinic consultation or home visit) by a GP from a GP cooperative between 2017–2019. We used telephone triage information to derive a risk prediction model for sepsis-related adverse outcome (infection-related ICU admission within seven days or infection-related death within 30 days) using logistic regression, random forest, and neural network machine learning techniques. Data from 2017 and 2018 were used for derivation and from 2019 for validation. Results: We included 155,486 patients (median age of 51 years; 59% females) in the analyses. The strongest predictors for sepsis-related adverse outcome were age, type of contact (home visit or clinic consultation), patients considered ABCD unstable during triage, and the entry complaints"general malaise", "shortness of breath" and "fever". The multivariable logistic regression model resulted in a C-statistic of 0.89 (95% CI 0.88–0.90) with good calibration. Machine learning models performed similarly to the logistic regression model. A "sepsis alert" based on a predicted probability >1% resulted in a sensitivity of 82% and a positive predictive value of 4.5%. However, most events occurred in patients receiving home visits, and model performance was substantially worse in this subgroup (C-statistic 0.70). Conclusion: Several patient characteristics identified during telephone triage of patients presenting to out-of-hours GP cooperatives were associated with sepsis-related adverse outcomes. Still, on a patient level, predictions were not sufficiently accurate for clinical purposes. [ABSTRACT FROM AUTHOR]