Search

Clinical decisions are often guided by clinical prediction models or diagnostic tests. Decision curve analysis (DCA) combines classical assessment of predictive performance with the consequences of using these strategies for clinical decision‐making. In DCA, the best decision strategy is the one that maximizes the net benefit: the net number of true positives (or negatives) provided by a given strategy. Here, we employ Bayesian approaches to DCA, addressing four fundamental concerns when evaluating clinical decision strategies: (i) which strategies are clinically useful, (ii) what is the best available decision strategy, (iii) which of two competing strategies is better, and (iv) what is the expected net benefit loss associated with the current level of uncertainty. While often consistent with frequentist point estimates, fully Bayesian DCA allows for an intuitive probabilistic interpretation framework and the incorporation of prior evidence. We evaluate the methods using simulation and provide a comprehensive case study. Software implementation is available in the bayesDCA R package. Ultimately, the Bayesian DCA workflow may help clinicians and health policymakers adopt better‐informed decisions. [ABSTRACT FROM AUTHOR]

Background: The aim of this systematic review was to investigate how clinical prediction models compare in terms of their methodological development, validation, and predictive capabilities, for patients with blunt chest trauma presenting to the Emergency Department. Methods: A systematic review was conducted across databases from 1st Jan 2000 until 1st April 2024. Studies were categorised into three types of multivariable prediction research and data extracted regarding methodological issues and the predictive capabilities of each model. Risk of bias and applicability were assessed. Results: 41 studies were included that discussed 22 different models. The most commonly observed study design was a single-centre, retrospective, chart review. The most widely externally validated clinical prediction models with moderate to good discrimination were the Thoracic Trauma Severity Score and the STUMBL Score. Discussion: This review demonstrates that the predictive ability of some of the existing clinical prediction models is acceptable, but high risk of bias and lack of subsequent external validation limits the extensive application of the models. The Thoracic Trauma Severity Score and STUMBL Score demonstrate better predictive accuracy in both development and external validation studies than the other models, but require recalibration and / or update and evaluation of their clinical and cost effectiveness. Review registration: PROSPERO database (https://www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID=351638). [ABSTRACT FROM AUTHOR]

STUDY QUESTION Can we develop a prediction model for the chance of a live birth following the transfer of an embryo created using donated oocytes? SUMMARY ANSWER Three primary models that included patient, past treatment, and cycle characteristics were developed using Australian data to predict the chance of a live birth following the transfer of an embryo created using donated oocytes; these models were well-calibrated to the population studied, achieved reasonable predictive power and generalizability when tested on New Zealand data. WHAT IS KNOWN ALREADY Nearly 9% of ART embryo transfer cycles performed globally use embryos created using donated oocytes. This percentage rises to one-quarter and one-half in same-sex couples and women aged over 45 years, respectively. STUDY DESIGN, SIZE, DURATION This study uses population-based Australian clinical registry data comprising 9384 embryo transfer cycles that occurred between 2015 and 2021 for model development, with an external validation cohort of 1493 New Zealand embryo transfer cycles. PARTICIPANTS/MATERIALS, SETTING, METHODS Three prediction models were compared that incorporated patient characteristics, but differed in whether they considered use of prior autologous treatment factors and current treatment parameters. We internally validated the models on Australian data using grouped cross-validation and reported several measures of model discrimination and calibration. Variable importance was measured through calculating the change in predictive performance that resulted from variable permutation. The best-performing model was externally validated on data from New Zealand. MAIN RESULTS AND THE ROLE OF CHANCE The best-performing model had an internal validation AUC-ROC of 0.60 and Brier score of 0.20, and external validation AUC-ROC of 0.61 and Brier score of 0.23. While these results indicate ∼15% less discriminatory ability compared to models assessed on an autologous cohort from the same population the performance of the models was clearly statistically significantly better than random, demonstrated generalizability, and was well-calibrated to the population studied. The most important variables for predicting the chance of a live birth were the oocyte donor age, the number of prior oocyte recipient embryo transfer cycles, whether the transferred embryo was cleavage or blastocyst stage and oocyte recipient age. Of lesser importance were the oocyte-recipient parity, whether donor or partner sperm was used, the number of prior autologous embryo transfer cycles and the number of embryos transferred. LIMITATIONS, REASONS FOR CAUTION The models had relatively weak discrimination suggesting further features need to be added to improve their predictive power. Variation in donor oocyte cohorts across countries due to differences such as whether anonymous and compensated donation are allowed may necessitate the models be recalibrated prior to application in non-Australian cohorts. WIDER IMPLICATIONS OF THE FINDINGS These results confirm the well-established importance of oocyte age and ART treatment history as the key prognostic factors in predicting treatment outcomes. One of the developed models has been incorporated into a consumer-facing website (YourIVFSuccess.com.au/Estimator) to allow patients to obtain personalized estimates of their chance of success using donor oocytes. STUDY FUNDING/COMPETING INTEREST(S) This research was funded by the Australian government as part of the Medical Research Future Fund (MRFF) Emerging Priorities and Consumer Driven Research initiative: EPCD000007. L.R. declares personal consulting fees from Abbott and Merck, lecture fees from Abbott, receipt of an educational grant from Merck, past presidency of the Fertility Society of Australia & New Zealand and World Endometriosis Society and being a minor shareholder in Monash IVF Group (ASX:MVF). G.M.C. declares receipt of Australian government grant funding for the research study and the development and maintenance of the YourIVFSuccess website. O.F. J.N. and A.P. report no conflicts of interest. TRIAL REGISTRATION NUMBER N/A. [ABSTRACT FROM AUTHOR]

Objective To analyze the risk factors of type 2 diabetic peripheral neuropathy (DPN) and to construct a clinical prediction model for DPN. Methods A retrospective review covered 581 patients with type 2 diabetes treated in the Department of Endocrinology and Metabolism Diseases of Changchun University of Traditional Chinese Medicine from September 2020 to November 2023; 296 patients without diabetic kidney disease were classified as NDPD group and 285 patients with diabetic kidney disease were classified as DPN group. The clinical data of patients were collected; univariate analysis was performed followed by multivariate Logistic regression analysis to identify the variables with statistically significant differences to find independent risk factors. R software was used to construct a nomogram, and plot the receiver operating characteristic (ROC) curve and then calculated the cut-off value, and the discrimination of the model was represented by the area under curve (AUC) value. The calibration diagram of the model was drawn, and the Hosmer-Lemeshow test combined with the calibration curve was used to evaluate the prediction accuracy of the model. Results Seven risk factors were selected as age, disease duration, smoking history, hemoglobinA1c(HbA1c), total cholesterol(TC),triglyceride(TG), low density lipo-protein-cholesterol(LDL)and a prediction model was preliminarily established based on the above risk factors. The AUC value of the area under the ROC curve was 0.722 (95% CI: 0.673-0.771), and the cut-off value was 0.477 (0.620, 0.729) indicating that the model had certain predictive capacity and accuracy for DPN. The results of Hosmer-Lemeshow test showed that χ2=10.683, P=0.220, indicating that the model fit was good. The results of the calibration chart showed that the prediction curve and the calibration curve had a good degree of coincidence, indicating that the accuracy of the model was good. Conclusions The risk factors for peripheral neuropathy in patients with type 2 diabetes mellitus include age, course of disease, smoking history, HbA1c, TC, TG, LDL. The clinical prediction model based on these factors can provide a reference for early clinical screening and early identification of DPN patients.

Abstract Background The aim of this systematic review was to investigate how clinical prediction models compare in terms of their methodological development, validation, and predictive capabilities, for patients with blunt chest trauma presenting to the Emergency Department. Methods A systematic review was conducted across databases from 1st Jan 2000 until 1st April 2024. Studies were categorised into three types of multivariable prediction research and data extracted regarding methodological issues and the predictive capabilities of each model. Risk of bias and applicability were assessed. Results 41 studies were included that discussed 22 different models. The most commonly observed study design was a single-centre, retrospective, chart review. The most widely externally validated clinical prediction models with moderate to good discrimination were the Thoracic Trauma Severity Score and the STUMBL Score. Discussion This review demonstrates that the predictive ability of some of the existing clinical prediction models is acceptable, but high risk of bias and lack of subsequent external validation limits the extensive application of the models. The Thoracic Trauma Severity Score and STUMBL Score demonstrate better predictive accuracy in both development and external validation studies than the other models, but require recalibration and / or update and evaluation of their clinical and cost effectiveness. Review registration PROSPERO database ( https://www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID=351638 ).

Abstract Background Successful deployment of clinical prediction models for clinical deterioration relates not only to predictive performance but to integration into the decision making process. Models may demonstrate good discrimination and calibration, but fail to match the needs of practising acute care clinicians who receive, interpret, and act upon model outputs or alerts. We sought to understand how prediction models for clinical deterioration, also known as early warning scores (EWS), influence the decision-making of clinicians who regularly use them and elicit their perspectives on model design to guide future deterioration model development and implementation. Methods Nurses and doctors who regularly receive or respond to EWS alerts in two digital metropolitan hospitals were interviewed for up to one hour between February 2022 and March 2023 using semi-structured formats. We grouped interview data into sub-themes and then into general themes using reflexive thematic analysis. Themes were then mapped to a model of clinical decision making using deductive framework mapping to develop a set of practical recommendations for future deterioration model development and deployment. Results Fifteen nurses (n = 8) and doctors (n = 7) were interviewed for a mean duration of 42 min. Participants emphasised the importance of using predictive tools for supporting rather than supplanting critical thinking, avoiding over-protocolising care, incorporating important contextual information and focusing on how clinicians generate, test, and select diagnostic hypotheses when managing deteriorating patients. These themes were incorporated into a conceptual model which informed recommendations that clinical deterioration prediction models demonstrate transparency and interactivity, generate outputs tailored to the tasks and responsibilities of end-users, avoid priming clinicians with potential diagnoses before patients were physically assessed, and support the process of deciding upon subsequent management. Conclusions Prediction models for deteriorating inpatients may be more impactful if they are designed in accordance with the decision-making processes of acute care clinicians. Models should produce actionable outputs that assist with, rather than supplant, critical thinking.

Abstract Background There are currently no effective prediction methods for evaluating the occurrence of cognitive impairment in patients with cerebral small vessel disease (CSVD). Aims To investigate the risk factors for cognitive dysfunction in patients with CSVD and to construct a risk prediction model. Methods A retrospective study was conducted on 227 patients with CSVD. All patients were assessed by brain magnetic resonance imaging (MRI), and the Montreal Cognitive Assessment (MoCA) was used to assess cognitive status. In addition, the patient’s medical records were also recorded. The clinical data were divided into a normal cognitive function group and a cognitive impairment group. A MoCA score

Background: Successful deployment of clinical prediction models for clinical deterioration relates not only to predictive performance but to integration into the decision making process. Models may demonstrate good discrimination and calibration, but fail to match the needs of practising acute care clinicians who receive, interpret, and act upon model outputs or alerts. We sought to understand how prediction models for clinical deterioration, also known as early warning scores (EWS), influence the decision-making of clinicians who regularly use them and elicit their perspectives on model design to guide future deterioration model development and implementation. Methods: Nurses and doctors who regularly receive or respond to EWS alerts in two digital metropolitan hospitals were interviewed for up to one hour between February 2022 and March 2023 using semi-structured formats. We grouped interview data into sub-themes and then into general themes using reflexive thematic analysis. Themes were then mapped to a model of clinical decision making using deductive framework mapping to develop a set of practical recommendations for future deterioration model development and deployment. Results: Fifteen nurses (n = 8) and doctors (n = 7) were interviewed for a mean duration of 42 min. Participants emphasised the importance of using predictive tools for supporting rather than supplanting critical thinking, avoiding over-protocolising care, incorporating important contextual information and focusing on how clinicians generate, test, and select diagnostic hypotheses when managing deteriorating patients. These themes were incorporated into a conceptual model which informed recommendations that clinical deterioration prediction models demonstrate transparency and interactivity, generate outputs tailored to the tasks and responsibilities of end-users, avoid priming clinicians with potential diagnoses before patients were physically assessed, and support the process of deciding upon subsequent management. Conclusions: Prediction models for deteriorating inpatients may be more impactful if they are designed in accordance with the decision-making processes of acute care clinicians. Models should produce actionable outputs that assist with, rather than supplant, critical thinking. Highlights: • This article explored decision-making processes of clinicians using a clinical prediction model for deteriorating patients, also known as an early warning score. • Our study identified that the clinical utility of deterioration models may lie in their assistance in generating, evaluating, and selecting diagnostic hypotheses, an important part of clinical decision making that is underrepresented in the prediction modelling literature. • Nurses in particular stressed the need for models that encourage critical thinking and further investigation rather than prescribe strict care protocols. [ABSTRACT FROM AUTHOR]

Objectives: To develop a machine learning logistic regression algorithm that can classify patients with an idiopathic macular hole (IMH) into those with good or poor vison at 6 months after a vitrectomy. In addition, to determine its accuracy and the contribution of the preoperative OCT characteristics to the algorithm. Methods: This was a single-center, cohort study. The classifier was developed using preoperative clinical information and the optical coherence tomographic (OCT) findings of 43 eyes of 43 patients who had undergone a vitrectomy. The explanatory variables were selected using a filtering method based on statistical significance and variance inflation factor (VIF) values, and the objective variable was the best-corrected visual acuity (BCVA) at 6 months postoperation. The discrimination threshold of the BCVA was the 0.15 logarithm of the minimum angle of the resolution (logMAR) units. Results: The performance of the classifier was 0.92 for accuracy, 0.73 for recall, 0.60 for precision, 0.74 for F-score, and 0.84 for the area under the curve (AUC). In logistic regression, the standard regression coefficients were 0.28 for preoperative BCVA, 0.13 for outer nuclear layer defect length (ONL_DL), −0.21 for outer plexiform layer defect length (OPL_DL) − (ONL_DL), and −0.17 for (OPL_DL)/(ONL_DL). In the IMH form, a stenosis pattern with a narrowing from the OPL to the ONL of the MH had a significant effect on the postoperative BCVA at 6 months. Conclusions: Our results indicate that (OPL_DL) − (ONL_DL) had a similar contribution to preoperative visual acuity in predicting the postoperative visual acuity. This model had a strong performance, suggesting that the preoperative visual acuity and MH characteristics in the OCT images were crucial in forecasting the postoperative visual acuity in IMH patients. Thus, it can be used to classify MH patients into groups with good or poor postoperative visual acuity, and the classification was comparable to that of previous studies using deep learning. [ABSTRACT FROM AUTHOR]

Background: There are currently no effective prediction methods for evaluating the occurrence of cognitive impairment in patients with cerebral small vessel disease (CSVD). Aims: To investigate the risk factors for cognitive dysfunction in patients with CSVD and to construct a risk prediction model. Methods: A retrospective study was conducted on 227 patients with CSVD. All patients were assessed by brain magnetic resonance imaging (MRI), and the Montreal Cognitive Assessment (MoCA) was used to assess cognitive status. In addition, the patient's medical records were also recorded. The clinical data were divided into a normal cognitive function group and a cognitive impairment group. A MoCA score < 26 (an additional 1 point for education < 12 years) is defined as cognitive dysfunction. Results: A total of 227 patients (mean age 66.7 ± 6.99 years) with CSVD were included in this study, of whom 68.7% were male and 100 patients (44.1%) developed cognitive impairment. Age (OR = 1.070; 95% CI = 1.015 ~ 1.128, p < 0.05), hypertension (OR = 2.863; 95% CI = 1.438 ~ 5.699, p < 0.05), homocysteine(HCY) (OR = 1.065; 95% CI = 1.005 ~ 1.127, p < 0.05), lacunar infarct score(Lac_score) (OR = 2.732; 95% CI = 1.094 ~ 6.825, P < 0.05), and CSVD total burden (CSVD_score) (OR = 3.823; 95% CI = 1.496 ~ 9.768, P < 0.05) were found to be independent risk factors for cognitive decline in the present study. The above 5 variables were used to construct a nomogram, and the model was internally validated by using bootstrapping with a C-index of 0.839. The external model validation C-index was 0.867. Conclusions: The nomogram model based on brain MR images and clinical data helps in individualizing the probability of cognitive impairment progression in patients with CSVD. [ABSTRACT FROM AUTHOR]

Abstract Background Middle-aged and older adults with physical disabilities exhibit more common and severe depressive symptoms than those without physical disabilities. Such symptoms can greatly affect the physical and mental health and life expectancy of middle-aged and older persons with disabilities. Method This study selected 2015 and 2018 data from the China Longitudinal Study of Health and Retirement. After analyzing the effect of age on depression, we used whether middle-aged and older adults with physical disabilities were depressed as the dependent variable and included a total of 24 predictor variables, including demographic factors, health behaviors, physical functioning and socialization, as independent variables. The data were randomly divided into training and validation sets on a 7:3 basis. LASSO regression analysis combined with binary logistic regression analysis was performed in the training set to screen the predictor variables of the model. Construct models in the training set and perform model evaluation, model visualization and internal validation. Perform external validation of the model in the validation set. Result A total of 1052 middle-aged and elderly persons with physical disabilities were included in this study, and the prevalence of depression in the elderly group > middle-aged group. Restricted triple spline indicated that age had different effects on depression in the middle-aged and elderly groups. LASSO regression analysis combined with binary logistic regression screened out Gender, Location of Residential Address, Shortsightedness, Hearing, Any possible helper in the future, Alcoholic in the Past Year, Difficulty with Using the Toilet, Difficulty with Preparing Hot Meals, and Unable to work due to disability constructed the Chinese Depression Prediction Model for Middle-aged and Older People with Physical Disabilities. The nomogram shows that living in a rural area, lack of assistance, difficulties with activities of daily living, alcohol abuse, visual and hearing impairments, unemployment and being female are risk factors for depression in middle-aged and older persons with physical disabilities. The area under the ROC curve for the model, internal validation and external validation were all greater than 0.70, the mean absolute error was less than 0.02, and the recall and precision were both greater than 0.65, indicating that the model performs well in terms of discriminability, accuracy and generalisation. The DCA curve and net gain curve of the model indicate that the model has high gain in predicting depression. Conclusion In this study, we showed that being female, living in rural areas, having poor vision and/or hearing, lack of assistance from others, drinking alcohol, having difficulty using the restroom and preparing food, and being unable to work due to a disability were risk factors for depression among middle-aged and older adults with physical disabilities. We developed a depression prediction model to assess the likelihood of depression in Chinese middle-aged and older adults with physical disabilities based on the above risk factors, so that early identification, intervention, and treatment can be provided to middle-aged and older adults with physical disabilities who are at high risk of developing depression.

Early prediction of the onset, progression and prognosis of acute ischemic stroke (AIS) is helpful for treatment decision-making and proactive management. Although several biomarkers have been found to predict the progression and prognosis of AIS, these biomarkers have not been widely used in routine clinical practice. Xanthine oxidase (XO) is a form of xanthine oxidoreductase (XOR), which is widespread in various organs of the human body and plays an important role in redox reactions and ischemia‒reperfusion injury. Our previous studies have shown that serum XO levels on admission have certain clinical predictive value for AIS. The purpose of this study was to utilize serum XO levels and clinical data to establish machine learning models for predicting the onset, progression, and prognosis of AIS. We enrolled 328 consecutive patients with AIS and 107 healthy controls from October 2020 to September 2021. Serum XO levels and stroke-related clinical data were collected. We established 5 machine learning models—the logistic regression (LR), support vector machine (SVM), decision tree, random forest, and K-nearest neighbor (KNN) models—to predict the onset, progression, and prognosis of AIS. The area under the receiver operating characteristic curve (AUROC), accuracy, sensitivity, specificity, negative predictive value, and positive predictive value were used to evaluate the predictive performance of each model. Among the 5 machine learning models predicting AIS onset, the AUROC values of 4 prediction models were over 0.7, while that of the KNN model was lower (AUROC = 0.6708, 95% CI 0.576–0.765). The LR model showed the best AUROC value (AUROC = 0.9586, 95% CI 0.927–0.991). Although the 5 machine learning models showed relatively poor predictive value for the progression of AIS (all AUROCs <0.7), the LR model still showed the highest AUROC value (AUROC = 0.6543, 95% CI 0.453–0.856). We compared the value of 5 machine learning models in predicting the prognosis of AIS, and the LR model showed the best predictive value (AUROC = 0.8124, 95% CI 0.715–0.910). The tested machine learning models based on serum levels of XO could predict the onset and prognosis of AIS. Among the 5 machine learning models, we found that the LR model showed the best predictive performance. Machine learning algorithms improve accuracy in the early diagnosis of AIS and can be used to make treatment decisions. [ABSTRACT FROM AUTHOR]

BackgroundIn light of the global effort to eradicate stunting in childhood, the objective of this research endeavor was to assess the prevalence of stunting and associated factors, simultaneously construct and validate a risk prediction model for stunting among children under the age of three in Shenzhen, China.MethodsUsing the stratified random sampling method, we selected 9,581 children under the age of three for research and analysis. The dataset underwent a random allocation into training and validation sets, adhering to a 8:2 split ratio. Within the training set, a combined approach of LASSO regression analysis and binary logistic regression analysis was implemented to identify and select the predictive variables for the model. Subsequently, model construction was conducted in the training set, encompassing model evaluation, visualization, and internal validation procedures. Finally, to assess the model's generalizability, external validation was performed using the validation set.ResultsA total of 684 (7.14%) had phenotypes of stunt. Utilizing a combined approach of LASSO regression and logistic regression, key predictors of stunting among children under three years of age were identified, including sex, age in months, mother's education, father's age, birth order, feeding patterns, delivery mode, average daily parent-child reading time, average time spent in child-parent interactions, and average daily outdoor time. These variables were subsequently employed to develop a comprehensive prediction model for childhood stunting. A nomogram model was constructed based on these factors, demonstrating excellent consistency and accuracy. Calibration curves validated the agreement between the nomogram predictions and actual observations. Furthermore, ROC and DCA analyses indicated the strong predictive performance of the nomograms.ConclusionsThe developed model for forecasting stunt risk, which integrates a spectrum of variables. This analytical framework presents actionable intelligence to medical professionals, laying down a foundational framework and a pivot for the conception and execution of preemptive strategies and therapeutic interventions.

ObjectiveAfter deploying a clinical prediction model, subsequently collected data can be used to fine-tune its predictions and adapt to temporal shifts. Because model updating carries risks of over-updating/fitting, we study online methods with performance guarantees.Materials and methodsWe introduce 2 procedures for continual recalibration or revision of an underlying prediction model: Bayesian logistic regression (BLR) and a Markov variant that explicitly models distribution shifts (MarBLR). We perform empirical evaluation via simulations and a real-world study predicting Chronic Obstructive Pulmonary Disease (COPD) risk. We derive "Type I and II" regret bounds, which guarantee the procedures are noninferior to a static model and competitive with an oracle logistic reviser in terms of the average loss.ResultsBoth procedures consistently outperformed the static model and other online logistic revision methods. In simulations, the average estimated calibration index (aECI) of the original model was 0.828 (95%CI, 0.818-0.938). Online recalibration using BLR and MarBLR improved the aECI towards the ideal value of zero, attaining 0.265 (95%CI, 0.230-0.300) and 0.241 (95%CI, 0.216-0.266), respectively. When performing more extensive logistic model revisions, BLR and MarBLR increased the average area under the receiver-operating characteristic curve (aAUC) from 0.767 (95%CI, 0.765-0.769) to 0.800 (95%CI, 0.798-0.802) and 0.799 (95%CI, 0.797-0.801), respectively, in stationary settings and protected against substantial model decay. In the COPD study, BLR and MarBLR dynamically combined the original model with a continually refitted gradient boosted tree to achieve aAUCs of 0.924 (95%CI, 0.913-0.935) and 0.925 (95%CI, 0.914-0.935), compared to the static model's aAUC of 0.904 (95%CI, 0.892-0.916).DiscussionDespite its simplicity, BLR is highly competitive with MarBLR. MarBLR outperforms BLR when its prior better reflects the data.ConclusionsBLR and MarBLR can improve the transportability of clinical prediction models and maintain their performance over time.

Abstract Background Over time, the performance of clinical prediction models may deteriorate due to changes in clinical management, data quality, disease risk and/or patient mix. Such prediction models must be updated in order to remain useful. In this study, we investigate dynamic model updating of clinical survival prediction models. In contrast to discrete or one-time updating, dynamic updating refers to a repeated process for updating a prediction model with new data. We aim to extend previous research which focused largely on binary outcome prediction models by concentrating on time-to-event outcomes. We were motivated by the rapidly changing environment seen during the COVID-19 pandemic where mortality rates changed over time and new treatments and vaccines were introduced. Methods We illustrate three methods for dynamic model updating: Bayesian dynamic updating, recalibration, and full refitting. We use a simulation study to compare performance in a range of scenarios including changing mortality rates, predictors with low prevalence and the introduction of a new treatment. Next, the updating strategies were applied to a model for predicting 70-day COVID-19-related mortality using patient data from QResearch, an electronic health records database from general practices in the UK. Results In simulated scenarios with mortality rates changing over time, all updating methods resulted in better calibration than not updating. Moreover, dynamic updating outperformed ad hoc updating. In the simulation scenario with a new predictor and a small updating dataset, Bayesian updating improved the C-index over not updating and refitting. In the motivating example with a rare outcome, no single updating method offered the best performance. Conclusions We found that a dynamic updating process outperformed one-time discrete updating in the simulations. Bayesian updating offered good performance overall, even in scenarios with new predictors and few events. Intercept recalibration was effective in scenarios with smaller sample size and changing baseline hazard. Refitting performance depended on sample size and produced abrupt changes in hazard ratio estimates between periods.

Abstract Background Deep learning models have had a lot of success in various fields. However, on structured data they have struggled. Here we apply four state-of-the-art supervised deep learning models using the attention mechanism and compare against logistic regression and XGBoost using discrimination, calibration and clinical utility. Methods We develop the models using a general practitioners database. We implement a recurrent neural network, a transformer with and without reverse distillation and a graph neural network. We measure discrimination using the area under the receiver operating characteristic curve (AUC) and the area under the precision recall curve (AUPRC). We assess smooth calibration using restricted cubic splines and clinical utility with decision curve analysis. Results Our results show that deep learning approaches can improve discrimination up to 2.5% points AUC and 7.4% points AUPRC. However, on average the baselines are competitive. Most models are similarly calibrated as the baselines except for the graph neural network. The transformer using reverse distillation shows the best performance in clinical utility on two out of three prediction problems over most of the prediction thresholds. Conclusion In this study, we evaluated various approaches in supervised learning using neural networks and attention. Here we do a rigorous comparison, not only looking at discrimination but also calibration and clinical utility. There is value in using deep learning models on electronic health record data since it can improve discrimination and clinical utility while providing good calibration. However, good baseline methods are still competitive.

BackgroundAlthough community-acquired pneumonia (CAP) is one of the most common infections in children, no tools exist to risk stratify children with suspected CAP. We developed and validated a prediction model to risk stratify and inform hospitalization decisions in children with suspected CAP.MethodsWe performed a prospective cohort study of children aged 3 months to 18 years with suspected CAP in a pediatric emergency department. Primary outcome was disease severity, defined as mild (discharge home or hospitalization for 24 hours), or severe (intensive care unit stay for >24 hours, septic shock, vasoactive agents, positive-pressure ventilation, chest drainage, extracorporeal membrane oxygenation, or death). Ordinal logistic regression and bootstrapped backwards selection were used to derive and internally validate our model.ResultsOf 1128 children, 371 (32.9%) developed moderate disease and 48 (4.3%) severe disease. Severity models demonstrated excellent discrimination (optimism-corrected c-indices of 0.81) and outstanding calibration. Severity predictors in the final model included respiratory rate, systolic blood pressure, oxygenation, retractions, capillary refill, atelectasis or pneumonia on chest radiograph, and pleural effusion.ConclusionsWe derived and internally validated a score that accurately predicts disease severity in children with suspected CAP. Once externally validated, this score has potential to facilitate management decisions by providing individualized risk estimates that can be used in conjunction with clinical judgment to improve the care of children with suspected CAP.

Abstract Background Knee osteoarthritis (KOA) is a multifactorial, slow-progressing, non-inflammatory degenerative disease primarily affecting synovial joints. It is usually induced by advanced age and/or trauma and eventually leads to irreversible destruction of articular cartilage and other tissues of the joint. Current research on KOA progression has limited clinical application significance. In this study, we constructed a prediction model for KOA progression based on multiple clinically relevant factors to provide clinicians with an effective tool to intervene in KOA progression. Method This study utilized the data set from the Dryad database which included patients with Kellgren–Lawrence (KL) grades 2 and 3. The KL grades was determined as the dependent variable, while 15 potential predictors were identified as independent variables. Patients were randomized into training set and validation set. The training set underwent LASSO analysis, model creation, visualization, decision curve analysis and internal validation using R language. The validation set is externally validated and F1-score, precision, and recall are computed. Result A total of 101 patients with KL2 and 94 patients with KL3 were selected. We randomly split the data set into a training set and a validation set by 8:2. We filtered “BMI”, “TC”, “Hypertension treatment”, and “JBS3 (%)” to build the prediction model for progression of KOA. Nomogram used to visualize the model in R language. Area under ROC curve was 0.896 (95% CI 0.847–0.945), indicating high discrimination. Mean absolute error (MAE) of calibration curve = 0.041, showing high calibration. MAE of internal validation error was 0.043, indicating high model calibration. Decision curve analysis showed high net benefit. External validation of the metabolic syndrome column-line graph prediction model was performed by the validation set. The area under the ROC curve was 0.876 (95% CI 0.767–0.984), indicating that the model had a high degree of discrimination. Meanwhile, the calibration curve Mean absolute error was 0.113, indicating that the model had a high degree of calibration. The F1 score is 0.690, the precision is 0.667, and the recall is 0.714. The above metrics represent a good performance of the model. Conclusion We found that KOA progression was associated with four variable predictors and constructed a predictive model for KOA progression based on the predictors. The clinician can intervene based on the nomogram of our prediction model. Key information This study is a clinical predictive model of KOA progression. KOA progression prediction model has good credibility and clinical value in the prevention of KOA progression.

Background: Deep learning models have had a lot of success in various fields. However, on structured data they have struggled. Here we apply four state-of-the-art supervised deep learning models using the attention mechanism and compare against logistic regression and XGBoost using discrimination, calibration and clinical utility. Methods: We develop the models using a general practitioners database. We implement a recurrent neural network, a transformer with and without reverse distillation and a graph neural network. We measure discrimination using the area under the receiver operating characteristic curve (AUC) and the area under the precision recall curve (AUPRC). We assess smooth calibration using restricted cubic splines and clinical utility with decision curve analysis. Results: Our results show that deep learning approaches can improve discrimination up to 2.5% points AUC and 7.4% points AUPRC. However, on average the baselines are competitive. Most models are similarly calibrated as the baselines except for the graph neural network. The transformer using reverse distillation shows the best performance in clinical utility on two out of three prediction problems over most of the prediction thresholds. Conclusion: In this study, we evaluated various approaches in supervised learning using neural networks and attention. Here we do a rigorous comparison, not only looking at discrimination but also calibration and clinical utility. There is value in using deep learning models on electronic health record data since it can improve discrimination and clinical utility while providing good calibration. However, good baseline methods are still competitive. [ABSTRACT FROM AUTHOR]

Background: Knee osteoarthritis (KOA) is a multifactorial, slow-progressing, non-inflammatory degenerative disease primarily affecting synovial joints. It is usually induced by advanced age and/or trauma and eventually leads to irreversible destruction of articular cartilage and other tissues of the joint. Current research on KOA progression has limited clinical application significance. In this study, we constructed a prediction model for KOA progression based on multiple clinically relevant factors to provide clinicians with an effective tool to intervene in KOA progression. Method: This study utilized the data set from the Dryad database which included patients with Kellgren–Lawrence (KL) grades 2 and 3. The KL grades was determined as the dependent variable, while 15 potential predictors were identified as independent variables. Patients were randomized into training set and validation set. The training set underwent LASSO analysis, model creation, visualization, decision curve analysis and internal validation using R language. The validation set is externally validated and F1-score, precision, and recall are computed. Result: A total of 101 patients with KL2 and 94 patients with KL3 were selected. We randomly split the data set into a training set and a validation set by 8:2. We filtered "BMI", "TC", "Hypertension treatment", and "JBS3 (%)" to build the prediction model for progression of KOA. Nomogram used to visualize the model in R language. Area under ROC curve was 0.896 (95% CI 0.847–0.945), indicating high discrimination. Mean absolute error (MAE) of calibration curve = 0.041, showing high calibration. MAE of internal validation error was 0.043, indicating high model calibration. Decision curve analysis showed high net benefit. External validation of the metabolic syndrome column-line graph prediction model was performed by the validation set. The area under the ROC curve was 0.876 (95% CI 0.767–0.984), indicating that the model had a high degree of discrimination. Meanwhile, the calibration curve Mean absolute error was 0.113, indicating that the model had a high degree of calibration. The F1 score is 0.690, the precision is 0.667, and the recall is 0.714. The above metrics represent a good performance of the model. Conclusion: We found that KOA progression was associated with four variable predictors and constructed a predictive model for KOA progression based on the predictors. The clinician can intervene based on the nomogram of our prediction model. Key information: This study is a clinical predictive model of KOA progression. KOA progression prediction model has good credibility and clinical value in the prevention of KOA progression. [ABSTRACT FROM AUTHOR]

Abstract Background Early distinction between mild and serious infections (SI) is challenging in children in ambulatory care. Clinical prediction models (CPMs), developed to aid physicians in clinical decision-making, require broad external validation before clinical use. We aimed to externally validate four CPMs, developed in emergency departments, in ambulatory care. Methods We applied the CPMs in a prospective cohort of acutely ill children presenting to general practices, outpatient paediatric practices or emergency departments in Flanders, Belgium. For two multinomial regression models, Feverkidstool and Craig model, discriminative ability and calibration were assessed, and a model update was performed by re-estimation of coefficients with correction for overfitting. For two risk scores, the SBI score and PAWS, the diagnostic test accuracy was assessed. Results A total of 8211 children were included, comprising 498 SI and 276 serious bacterial infections (SBI). Feverkidstool had a C-statistic of 0.80 (95% confidence interval 0.77–0.84) with good calibration for pneumonia and 0.74 (0.70–0.79) with poor calibration for other SBI. The Craig model had a C-statistic of 0.80 (0.77–0.83) for pneumonia, 0.75 (0.70–0.80) for complicated urinary tract infections and 0.63 (0.39–0.88) for bacteraemia, with poor calibration. The model update resulted in improved C-statistics for all outcomes and good overall calibration for Feverkidstool and the Craig model. SBI score and PAWS performed extremely weak with sensitivities of 0.12 (0.09–0.15) and 0.32 (0.28–0.37). Conclusions Feverkidstool and the Craig model show good discriminative ability for predicting SBI and a potential for early recognition of SBI, confirming good external validity in a low prevalence setting of SBI. The SBI score and PAWS showed poor diagnostic performance. Trial registration ClinicalTrials.gov, NCT02024282. Registered on 31 December 2013.

The quantitative effect of seasonality on the prevalence of infectious diseases has been widely studied in epidemiological models. However, its influence in clinical prediction models has not been analyzed in great depth. In this work, we study the different approaches that can be employed to deal with seasonality when using white-box models related to infections, including two new proposals based on sliding windows and ensembles. We additionally consider the effects of class imbalance and high dimensionality, as they are common problems that must be confronted when building clinical prediction models. These approaches were tested with four datasets: two created synthetically and two extracted from the MIMIC-III database. Our proposed methods obtained the best results in the majority of the experiments, although traditional approaches attained good results in certain cases. On the whole, our results corroborate the theory that clinical prediction models for infections can be improved by considering the effect of seasonality, although the techniques employed to obtain the best results are highly dependent on both the dataset and the modeling technique considered. [ABSTRACT FROM AUTHOR]

Clinical prediction models are increasingly used across healthcare to support clinical decision making. Existing methods and models are time-invariant and thus ignore the changes in populations and healthcare practice that occur over time. We aimed to compare the performance of time-invariant with time-variant models in UK National Adult Cardiac Surgery Audit data from Manchester University NHS Foundation Trust between 2009 and 2019. Data from 2009-2011 were used for initial model fitting, and data from 2012-2019 for validation and updating. We fitted four models to the data: a time-invariant logistic regression model (not updated), a logistic model which was updated every year and validated it in each subsequent year, a logistic regression model where the intercept is a function of calendar time (not updated), and a continually updating Bayesian logistic model which was updated with each new observation and continuously validated. We report predictive performance over the complete validation cohort and for each year in the validation data. Over the complete validation data, the Bayesian model had the best predictive performance. [ABSTRACT FROM AUTHOR]

Rationale & objectiveThe Kidney Failure Risk Equation (KFRE) is a simple widely validated prediction model using age, sex, estimated glomerular filtration rate, and urinary albumin-creatinine ratio to predict the risk for end-stage kidney disease. Data are limited for its applicability to kidney transplant recipients.Study designValidation study of the KFRE as a post hoc analysis of the Folic Acid for Vascular Outcomes Reduction in Transplantation (FAVORIT) Trial.Setting & participantsAdult kidney transplant recipients with functioning kidney allografts at least 6 months posttransplantation from 30 centers in the United States, Canada, and Brazil. Participants with estimated glomerular filtration rates

Abstract Introduction Personalized disease management informed by quantitative risk prediction has the potential to improve patient care and outcomes. The integration of risk prediction into clinical workflow should be informed by the experiences and preferences of stakeholders, and the impact of such integration should be evaluated in prospective comparative studies. The objectives of the IMplementing Predictive Analytics towards efficient chronic obstructive pulmonary disease (COPD) treatments (IMPACT) study are to integrate an exacerbation risk prediction tool into routine care and to determine its impact on prescription appropriateness (primary outcome), medication adherence, quality of life, exacerbation rates, and sex and gender disparities in COPD care (secondary outcomes). Methods IMPACT will be conducted in two phases. Phase 1 will include the systematic and user-centered development of two decision support tools: (1) a decision tool for pulmonologists called the ACCEPT decision intervention (ADI), which combines risk prediction from the previously developed Acute COPD Exacerbation Prediction Tool with treatment algorithms recommended by the Canadian Thoracic Society’s COPD pharmacotherapy guidelines, and (2) an information pamphlet for COPD patients (patient tool), tailored to their prescribed medication, clinical needs, and lung function. In phase 2, we will conduct a stepped-wedge cluster randomized controlled trial in two outpatient respiratory clinics to evaluate the impact of the decision support tools on quality of care and patient outcomes. Clusters will be practicing pulmonologists (n ≥ 24), who will progressively switch to the intervention over 18 months. At the end of the study, a qualitative process evaluation will be carried out to determine the barriers and enablers of uptake of the tools. Discussion The IMPACT study coincides with a planned harmonization of electronic health record systems across tertiary care centers in British Columbia, Canada. The harmonization of these systems combined with IMPACT’s implementation-oriented design and partnership with stakeholders will facilitate integration of the tools into routine care, if the results of the proposed study reveal positive association with improvement in the process and outcomes of clinical care. The process evaluation at the end of the trial will inform subsequent design iterations before largescale implementation. Trial registration NCT05309356.

Abstract Background Supporting decisions for patients who present to the emergency department (ED) with COVID-19 requires accurate prognostication. We aimed to evaluate prognostic models for predicting outcomes in hospitalized patients with COVID-19, in different locations and across time. Methods We included patients who presented to the ED with suspected COVID-19 and were admitted to 12 hospitals in the New York City (NYC) area and 4 large Dutch hospitals. We used second-wave patients who presented between September and December 2020 (2137 and 3252 in NYC and the Netherlands, respectively) to evaluate models that were developed on first-wave patients who presented between March and August 2020 (12,163 and 5831). We evaluated two prognostic models for in-hospital death: The Northwell COVID-19 Survival (NOCOS) model was developed on NYC data and the COVID Outcome Prediction in the Emergency Department (COPE) model was developed on Dutch data. These models were validated on subsequent second-wave data at the same site (temporal validation) and at the other site (geographic validation). We assessed model performance by the Area Under the receiver operating characteristic Curve (AUC), by the E-statistic, and by net benefit. Results Twenty-eight-day mortality was considerably higher in the NYC first-wave data (21.0%), compared to the second-wave (10.1%) and the Dutch data (first wave 10.8%; second wave 10.0%). COPE discriminated well at temporal validation (AUC 0.82), with excellent calibration (E-statistic 0.8%). At geographic validation, discrimination was satisfactory (AUC 0.78), but with moderate over-prediction of mortality risk, particularly in higher-risk patients (E-statistic 2.9%). While discrimination was adequate when NOCOS was tested on second-wave NYC data (AUC 0.77), NOCOS systematically overestimated the mortality risk (E-statistic 5.1%). Discrimination in the Dutch data was good (AUC 0.81), but with over-prediction of risk, particularly in lower-risk patients (E-statistic 4.0%). Recalibration of COPE and NOCOS led to limited net benefit improvement in Dutch data, but to substantial net benefit improvement in NYC data. Conclusions NOCOS performed moderately worse than COPE, probably reflecting unique aspects of the early pandemic in NYC. Frequent updating of prognostic models is likely to be required for transportability over time and space during a dynamic pandemic.

Background: Early distinction between mild and serious infections (SI) is challenging in children in ambulatory care. Clinical prediction models (CPMs), developed to aid physicians in clinical decision-making, require broad external validation before clinical use. We aimed to externally validate four CPMs, developed in emergency departments, in ambulatory care. Methods: We applied the CPMs in a prospective cohort of acutely ill children presenting to general practices, outpatient paediatric practices or emergency departments in Flanders, Belgium. For two multinomial regression models, Feverkidstool and Craig model, discriminative ability and calibration were assessed, and a model update was performed by re-estimation of coefficients with correction for overfitting. For two risk scores, the SBI score and PAWS, the diagnostic test accuracy was assessed. Results: A total of 8211 children were included, comprising 498 SI and 276 serious bacterial infections (SBI). Feverkidstool had a C-statistic of 0.80 (95% confidence interval 0.77–0.84) with good calibration for pneumonia and 0.74 (0.70–0.79) with poor calibration for other SBI. The Craig model had a C-statistic of 0.80 (0.77–0.83) for pneumonia, 0.75 (0.70–0.80) for complicated urinary tract infections and 0.63 (0.39–0.88) for bacteraemia, with poor calibration. The model update resulted in improved C-statistics for all outcomes and good overall calibration for Feverkidstool and the Craig model. SBI score and PAWS performed extremely weak with sensitivities of 0.12 (0.09–0.15) and 0.32 (0.28–0.37). Conclusions: Feverkidstool and the Craig model show good discriminative ability for predicting SBI and a potential for early recognition of SBI, confirming good external validity in a low prevalence setting of SBI. The SBI score and PAWS showed poor diagnostic performance. Trial registration: ClinicalTrials.gov, NCT02024282. Registered on 31 December 2013. [ABSTRACT FROM AUTHOR]

Despite an exponential increase in publications on clinical prediction models over recent years, the number of models deployed in clinical practice remains fairly limited. In this paper, we identify common obstacles that impede effective deployment of prediction models in healthcare, and investigate their underlying causes. We observe a key underlying cause behind most obstacles — the improper development and evaluation of prediction models. Inherent heterogeneities in clinical data complicate the development and evaluation of clinical prediction models. Many of these heterogeneities in clinical data are unreported because they are deemed to be irrelevant, or due to privacy concerns. We provide real-life examples where failure to handle heterogeneities in clinical data, or sources of biases, led to the development of erroneous models. The purpose of this paper is to familiarize modeling practitioners with common sources of biases and heterogeneities in clinical data, both of which have to be dealt with to ensure proper development and evaluation of clinical prediction models. Proper model development and evaluation, together with complete and thorough reporting, are important prerequisites for a prediction model to be effectively deployed in healthcare. [ABSTRACT FROM AUTHOR]

Cardiovascular diseases and their associated disorder of heart failure (HF) are major causes of death globally, making it a priority for doctors to detect and predict their onset and medical consequences. Artificial Intelligence (AI) allows doctors to discover clinical indicators and enhance their diagnoses and treatments. Specifically, “eXplainable AI” (XAI) offers tools to improve the clinical prediction models that experience poor interpretability of their results. This work presents an explainability analysis and evaluation of two HF survival prediction models using a dataset that includes 299 patients who have experienced HF. The first model utilizes survival analysis, considering death events and time as target features, while the second model approaches the problem as a classification task to predict death. The model employs an optimization data workflow pipeline capable of selecting the best machine learning algorithm as well as the optimal collection of features. Moreover, different post hoc techniques have been used for the explainability analysis of the model. The main contribution of this paper is an explainability-driven approach to select the best HF survival prediction model balancing prediction performance and explainability. Therefore, the most balanced explainable prediction models are Survival Gradient Boosting model for the survival analysis and Random Forest for the classification approach with a c-index of 0.714 and balanced accuracy of 0.74 (std 0.03) respectively. The selection of features by the SCI-XAI in the two models is similar where “serum_creatinine”, “ejection_fraction”, and “sex” are selected in both approaches, with the addition of “diabetes” for the survival analysis model. Moreover, the application of post hoc XAI techniques also confirm common findings from both approaches by placing the “serum_creatinine” as the most relevant feature for the predicted outcome, followed by “ejection_fraction”. The explainable prediction models for HF survival presented in this paper would improve the further adoption of clinical prediction models by providing doctors with insights to better understand the reasoning behind usually “black-box” AI clinical solutions and make more reasonable and data-driven decisions.

Aims: Multinomial logistic regression models allow one to predict the risk of a categorical outcome with > 2 categories. When developing such a model, researchers should ensure the number of participants (n) is appropriate relative to the number of events (E k) and the number of predictor parameters (p k) for each category k. We propose three criteria to determine the minimum n required in light of existing criteria developed for binary outcomes. Proposed criteria: The first criterion aims to minimise the model overfitting. The second aims to minimise the difference between the observed and adjusted R 2 Nagelkerke. The third criterion aims to ensure the overall risk is estimated precisely. For criterion (i), we show the sample size must be based on the anticipated Cox-snell R 2 of distinct 'one-to-one' logistic regression models corresponding to the sub-models of the multinomial logistic regression, rather than on the overall Cox-snell R 2 of the multinomial logistic regression. Evaluation of criteria: We tested the performance of the proposed criteria (i) through a simulation study and found that it resulted in the desired level of overfitting. Criterion (ii) and (iii) were natural extensions from previously proposed criteria for binary outcomes and did not require evaluation through simulation. Summary: We illustrated how to implement the sample size criteria through a worked example considering the development of a multinomial risk prediction model for tumour type when presented with an ovarian mass. Code is provided for the simulation and worked example. We will embed our proposed criteria within the pmsampsize R library and Stata modules. [ABSTRACT FROM AUTHOR]

Aims: The Thrombolysis in Myocardial Infarction Risk Score for Heart Failure (HF) in Diabetes (TRS‐HFDM) prognosticates HF hospitalization in people with type 2 diabetes (T2D). This study aimed to externally validate and extend its use for those with recent acute coronary syndrome (ACS). Materials and Methods: The TRS‐HFDM was externally validated in the Examination of Cardiovascular Outcomes with Alogliptin versus Standard of Care (EXAMINE) trial (n = 5380) and extended with natriuretic biomarkers. Missing data were multiply imputed. Initial TRS‐HFDM variables were previous HF (2 points), atrial fibrillation (1 point), coronary artery disease (1 point), estimated glomerular filtration rate <60 ml/min/1.73 m2 (1 point), and urine albumin‐to‐creatinine ratio 30‐300 mg/g (1 point) and >300 mg/g (2 points). Results: In total, HF hospitalization occurred in 193 (3.6%) patients. Based on the TRS‐HFDM, 25% of patients were classified as intermediate risk (1 point), 30% were classified as high risk (2 points), 19% were classified as very‐high risk (3 points) and 26% were classified as severe risk (≥4 points). Before model extension, discrimination (C‐index 0.76, 95%·CI 0.73‐0.80) and calibration (calibration slope 0.82, 95%·CI 0.65‐1.0; calibration‐in‐the‐large −0.15, 95%·CI −0.37‐0.64) were moderate‐to‐good in individuals with T2D and recent ACS. The extension of TRS‐HFDM with the addition of N‐terminal pro‐B‐type natriuretic peptide (NT‐ProBNP) improved discrimination (C‐index 0.82, 95%·CI 0.79‐0.85) and calibration (calibration slope 0.84, 95%·CI 0.66‐1.02; calibration‐in‐the‐large −0.12, 95%·CI −0.33‐0.081) for this higher‐risk population. Conclusion: The TRS‐HFDM with the extension of NT‐ProBNP improves risk stratification and generalizes the use of the risk score for patients with T2D and ACS. Future validation studies in ACS populations may be warranted. [ABSTRACT FROM AUTHOR]

Introduction: The digitization of hospital systems, including integrated electronic medical records, has provided opportunities to improve the prediction performance of inpatient fall risk models and their application to computerized clinical decision support systems. This review describes the data sources and scope of methods reported in studies that developed inpatient fall prediction models, including machine learning and more traditional approaches to inpatient fall risk prediction. Methods: This scoping review used methods recommended by the Arksey and O'Malley framework and its recent advances. PubMed, CINAHL, IEEE Xplore, and EMBASE databases were systematically searched. Studies reporting the development of inpatient fall risk prediction approaches were included. There was no restriction on language or recency. Reference lists and manual searches were also completed. Reporting quality was assessed using adherence to Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis statement (TRIPOD), where appropriate. Results: Database searches identified 1,396 studies, 63 were included for scoping assessment and 45 for reporting quality assessment. There was considerable overlap in data sources and methods used for model development. Fall prediction models typically relied on features from patient assessments, including indicators of physical function or impairment, or cognitive function or impairment. All but two studies used patient information at or soon after admission and predicted fall risk over the entire admission, without consideration of post-admission interventions, acuity changes or length of stay. Overall, reporting quality was poor, but improved in the past decade. Conclusion: There was substantial homogeneity in data sources and prediction model development methods. Use of artificial intelligence, including machine learning with high-dimensional data, remains underexplored in the context of hospital falls. Future research should consider approaches with the potential to utilize high-dimensional data from digital hospital systems, which may contribute to greater performance and clinical usefulness. [ABSTRACT FROM AUTHOR]

Self-harm risk prediction models developed using health system data (electronic health records and insurance claims information) often use patient information from up to several years prior to the index visit when the prediction is made. Measurements from some time periods may not be available for all patients. Using the framework of algorithm-agnostic variable importance, we study the predictive potential of variables corresponding to different time horizons prior to the index visit and demonstrate the application of variable importance techniques in the biomedical informatics setting. We use variable importance to quantify the potential of recent (up to three months before the index visit) and distant (more than one year before the index visit) patient mental health information for predicting self-harm risk using data from seven health systems. We quantify importance as the decrease in predictiveness when the variable set of interest is excluded from the prediction task. We define predictiveness using discriminative metrics: area under the receiver operating characteristic curve (AUC), sensitivity, and positive predictive value. Mental health predictors corresponding to the three months prior to the index visit show strong signal of importance; in one setting, excluding these variables decreased AUC from 0.85 to 0.77. Predictors corresponding to more distant information were less important. Predictors from the months immediately preceding the index visit are highly important. Implementation of self-harm prediction models may be challenging in settings where recent data are not completely available (e.g., due to lags in insurance claims processing) at the time a prediction is made. Clinically derived variables from different time frames exhibit varying levels of importance for predicting self-harm. Variable importance analyses can inform whether and how to implement risk prediction models into clinical practice given real-world data limitations. These analyses be applied more broadly in biomedical informatics research to provide insight into general clinical risk prediction tasks. [Display omitted] [ABSTRACT FROM AUTHOR]

Background: The accurate prediction of the outcome of hepatitis B virus-related acute-on-chronic liver failure (HBV-ACLF) is impeded by population heterogeneity. The study aimed to assess the impact of underlying cirrhosis on the performance of clinical prediction models (CPMs). Methods: Using data from two multicenter, prospective cohorts of patients with HBV-ACLF, the discrimination, calibration, and clinical benefit were assessed for CPMs predicting 28-day and 90-day outcomes in patients with cirrhosis and those without, respectively. Results: A total of 919 patients with HBV-ACLF were identified by Chinese Group on the Study of Severe Hepatitis B (COSSH) criteria, including 675 with cirrhosis and 244 without. COSSH-ACLF IIs, COSSH-ACLFs, Chronic Liver Failure-Consortium Acute-on-Chronic Liver Failure score (CLIF-C ACLFs), Tongji Prognostic Predictor Model score (TPPMs), Model for End-Stage Liver Disease score (MELDs), and MELD-Sodium score (MELD-Nas) were all strong predictors of short-term mortality in patients with HBV-ACLF. In contrast to a high model discriminative capacity in ACLF without cirrhosis, each prognostic model represents a marked decline of C-index, net reclassification index (NRI), and integrated discrimination improvement (IDI) in predicting either 28-day or 90-day prognosis of patients with cirrhosis. The hazard analysis identified largely overlapping risk factors of poor outcomes in both subgroups, while serum bilirubin was specifically associated with short-term mortality in patients with cirrhosis and blood urea nitrogen in patients without cirrhosis. A subgroup analysis in patients with cirrhosis showed a decline of discrimination of CPMS in those with ascites or infections compared to that in those without. Conclusion: Predicting the short-term outcome of HBV-ACLF by CPMs is optimal in patients without cirrhosis but limited in those with cirrhosis, at least partially due to the complicated ascites or infections. [ABSTRACT FROM AUTHOR]

Background The number of studies applying machine learning (ML) to predict acute kidney injury (AKI) has grown steadily over the past decade. We assess and critically appraise the state of the art in ML models for AKI prediction, considering performance, methodological soundness, and applicability. Methods We searched PubMed and ArXiv, extracted data, and critically appraised studies based on the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD), Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS), and Prediction Model Risk of Bias Assessment Tool (PROBAST) guidelines. Results Forty-six studies from 3166 titles were included. Thirty-eight studies developed a model, five developed and externally validated one, and three studies externally validated one. Flexible ML methods were used more often than deep learning, although the latter was common with temporal variables and text as predictors. Predictive performance showed an area under receiver operating curves ranging from 0.49 to 0.99. Our critical appraisal identified a high risk of bias in 39 studies. Some studies lacked internal validation, whereas external validation and interpretability of results were rarely considered. Fifteen studies focused on AKI prediction in the intensive care setting, and the US-derived Medical Information Mart for Intensive Care (MIMIC) data set was commonly used. Reproducibility was limited as data and code were usually unavailable. Conclusions Flexible ML methods are popular for the prediction of AKI, although more complex models based on deep learning are emerging. Our critical appraisal identified a high risk of bias in most models: Studies should use calibration measures and external validation more often, improve model interpretability, and share data and code to improve reproducibility. [ABSTRACT FROM AUTHOR]

OBJECTIVE: Currently, there are still no convincing clinical models predicting closed lower extremity fracture-associated deep vein thrombosis in patients treated through thromboprophylactic methods. We aimed at using two retrospective cohorts to develop and externally verify a clinical prediction model for deep vein thrombosis in patients treated with anticoagulants after suffering closed lower extremity fractures. PATIENTS AND METHODS: We evaluated the patients’ pre- and post-operatively, to accurately determine the predictive power of the biomarkers and clinical risk factors. Two retrospective cohorts were used for the development and external verification of a pre-operative clinical prediction model (development: n = 2,253; verification: n = 833) and post-operative clinical prediction model (development: n = 1,422; verification: n = 449), respectively. RESULTS: The C-indices were used to show the predicted incidence of objective thrombosis at the pre- and post-operative stage, which were then compared with the observed incidence of thrombosis in both cohorts. Biomarkers and clinical indicators were included in pre- and post-operative nomograms, which were adequately calibrated in both cohorts. The cross-validated C-indices of the pre- and post-operative clinical prediction models in the verification cohort were 0.706 (95% Cl, 0.67-0.74) and 0.875 (95% Cl, 0.84-0.91), respectively. CONCLUSIONS: We present our findings of novel pre- and post-operative nomograms for the prediction of deep venous thrombosis in patients who received thromboprophylaxis after suffering closed lower extremity fractures. [ABSTRACT FROM AUTHOR]

Objectives: To determine the extent to which current Large Language Models (LLMs) can serve as substitutes for traditional machine learning (ML) as clinical predictors using data from electronic health records (EHRs), we investigated various factors that can impact their adoption, including overall performance, calibration, fairness, and resilience to privacy protections that reduce data fidelity., Materials and Methods: We evaluated GPT-3.5, GPT-4, and ML (as gradient-boosting trees) on clinical prediction tasks in EHR data from Vanderbilt University Medical Center and MIMIC IV. We measured predictive performance with AUROC and model calibration using Brier Score. To evaluate the impact of data privacy protections, we assessed AUROC when demographic variables are generalized. We evaluated algorithmic fairness using equalized odds and statistical parity across race, sex, and age of patients. We also considered the impact of using in-context learning by incorporating labeled examples within the prompt., Results: Traditional ML (AUROC: 0.847, 0.894 (VUMC, MIMIC)) substantially outperformed GPT-3.5 (AUROC: 0.537, 0.517) and GPT-4 (AUROC: 0.629, 0.602) (with and without in-context learning) in predictive performance and output probability calibration (Brier Score (ML vs GPT-3.5 vs GPT-4): 0.134 versus 0.384 versus 0.251, 0.042 versus 0.06 versus 0.219). Traditional ML is more robust than GPT-3.5 and GPT-4 to generalizing demographic information to protect privacy. GPT-4 is the fairest model according to our selected metrics but at the cost of poor model performance., Conclusion: These findings suggest that LLMs are much less effective and robust than locally-trained ML for clinical prediction tasks, but they are getting better over time.

Objectives: We describe the steps for implementing a dynamic updating pipeline for clinical prediction models and illustrate the proposed methods in an application of 5-year survival prediction in cystic fibrosis., Study Design and Setting: Dynamic model updating refers to the process of repeated updating of a clinical prediction model with new information to counter performance degradation. We describe 2 types of updating pipeline: "proactive updating" where candidate model updates are tested any time new data are available, and "reactive updating" where updates are only made when performance of the current model declines or the model structure changes. Methods for selecting the best candidate updating model are based on measures of predictive performance under the 2 pipelines. The methods are illustrated in our motivating example of a 5-year survival prediction model in cystic fibrosis. Over a dynamic updating period of 10 years, we report the updating decisions made and the performance of the prediction models selected under each pipeline., Results: Both the proactive and reactive updating pipelines produced survival prediction models that overall had better performance in terms of calibration and discrimination than a model that was not updated. Further, use of the dynamic updating pipelines ensured that the prediction model's performance was consistently and frequently reviewed in new data., Conclusion: Implementing a dynamic updating pipeline will help guard against model performance degradation while ensuring that the updating process is principled and data-driven., Competing Interests: Declaration of competing interest There are no competing interests for any author., (Copyright © 2024 The Author(s). Published by Elsevier Inc. All rights reserved.)

The quantitative effect of seasonality on the prevalence of infectious diseases has been widely studied in epidemiological models. However, its influence in clinical prediction models has not been analyzed in great depth. In this work, we study the different approaches that can be employed to deal with seasonality when using white-box models related to infections, including two new proposals based on sliding windows and ensembles. We additionally consider the effects of class imbalance and high dimensionality, as they are common problems that must be confronted when building clinical prediction models. These approaches were tested with four datasets: two created synthetically and two extracted from the MIMIC-III database. Our proposed methods obtained the best results in the majority of the experiments, although traditional approaches attained good results in certain cases. On the whole, our results corroborate the theory that clinical prediction models for infections can be improved by considering the effect of seasonality, although the techniques employed to obtain the best results are highly dependent on both the dataset and the modeling technique considered.

This study's goal is to determine the accuracy of a linear classifier that predicts the prognosis of patients with macular edema (ME) due to a branch retinal vein occlusion during the maintenance phase of antivascular endothelial growth factor (anti-VEGF) therapy. The classifier was created using the clinical information and optical coherence tomographic (OCT) findings obtained up to the time of the first resolution of ME. In total, 66 eyes of 66 patients received an initial intravitreal injection of anti-VEGF followed by repeated injections with the pro re nata (PRN) regimen for 12 months. The patients were divided into two groups: those with and those without good vision during the PRN phase. The mean AUC of the classifier was 0.93, and the coefficients of the explanatory variables were: best-corrected visual acuity (BCVA) at baseline was 0.66, BCVA at first resolution of ME was 0.51, age was 0.21, the average brightness of the ellipsoid zone (EZ) was −0.12, the intactness of the external limiting membrane (ELM) was −0.14, the average brightness of the ELM was −0.17, the brightness value of EZ was −0.17, the area of the outer segments of the photoreceptors was −0.20, and the intactness of the EZ was −0.24. This algorithm predicted the prognosis over time for individual patients during the PRN phase. [ABSTRACT FROM AUTHOR]

Summary: Background: The Acute Chronic Obstructive Pulmonary Disease (COPD) Exacerbation Prediction Tool (ACCEPT) was developed for individualised prediction of COPD exacerbations. ACCEPT was well calibrated overall and had a high discriminatory power, but overestimated risk among individuals without recent exacerbations. The objectives of this study were to 1) fine-tune ACCEPT to make better predictions for individuals with a negative exacerbation history, 2) develop more parsimonious models, and 3) externally validate the models in a new dataset. Methods: We recalibrated ACCEPT using data from the Evaluation of COPD Longitudinally to Identify Predictive Surrogate End-points (ECLIPSE, a three-year observational study, 1,803 patients, 2,117 exacerbations) study by applying non-parametric regression splines to the predicted rates. We developed three reduced versions of ACCEPT by removing symptom score and/or baseline medications as predictors. We examined the discrimination, calibration, and net benefit of ACCEPT 2·0 in the placebo arm of the Towards a Revolution in COPD Health (TORCH, a three-year randomised clinical trial of inhaled therapies in COPD, 1,091 patients, 1,064 exacerbations) study. The primary outcome for prediction was the occurrence of ≥2 moderate or ≥1 severe exacerbation in the next 12 months; the secondary outcomes were prediction of the occurrence of any moderate/severe exacerbation or any severe exacerbation. Findings: ACCEPT 2·0 had an area-under-the-curve (AUC) of 0·76 for predicting the primary outcome. Exacerbation history alone (current standard of care) had an AUC of 0·68. The model was well calibrated in patients with positive or negative exacerbation histories. Changes in AUC in reduced versions were minimal for the primary outcome as well as for predicting the occurrence of any moderate/severe exacerbations (ΔAUC