Search

Diagnostic accuracy studies assess the sensitivity and specificity of a new index test in relation to an established comparator or the reference standard. The development and selection of the index test are usually assumed to be conducted prior to the accuracy study. In practice, this is often violated, for instance, if the choice of the (apparently) best biomarker, model or cutpoint is based on the same data that is used later for validation purposes. In this work, we investigate several multiple comparison procedures which provide family-wise error rate control for the emerging multiple testing problem. Due to the nature of the co-primary hypothesis problem, conventional approaches for multiplicity adjustment are too conservative for the specific problem and thus need to be adapted. In an extensive simulation study, five multiple comparison procedures are compared with regard to statistical error rates in least-favourable and realistic scenarios. This covers parametric and non-parametric methods and one Bayesian approach. All methods have been implemented in the new open-source R package cases which allows us to reproduce all simulation results. Based on our numerical results, we conclude that the parametric approaches (maxT and Bonferroni) are easy to apply but can have inflated type I error rates for small sample sizes. The two investigated Bootstrap procedures, in particular the so-called pairs Bootstrap, allow for a family-wise error rate control in finite samples and in addition have a competitive statistical power. [ABSTRACT FROM AUTHOR]

Validation of prediction models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. This article describes measures to evaluate predictions and the potential improvement in decision making from survival models based on Cox proportional hazards regression. Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. This article describes measures to evaluate predictions and the potential improvement in decision making from survival models based on Cox proportional hazards regression. As a motivating case study, the authors consider the prediction of the composite outcome of recurrence or death (the "event") in patients with breast cancer after surgery. They developed a simple Cox regression model with 3 predictors, as in the Nottingham Prognostic Index, in 2982 women (1275 events over 5 years of follow-up) and externally validated this model in 686 women (285 events over 5 years). Improvement in performance was assessed after the addition of progesterone receptor as a prognostic biomarker. The model predictions can be evaluated across the full range of observed follow-up times or for the event occurring by the end of a fixed time horizon of interest. The authors first discuss recommended statistical measures that evaluate model performance in terms of discrimination, calibration, or overall performance. Further, they evaluate the potential clinical utility of the model to support clinical decision making according to a net benefit measure. They provide SAS and R code to illustrate internal and external validation. The authors recommend the proposed set of performance measures for transparent reporting of the validity of predictions from survival models. [ABSTRACT FROM AUTHOR]

In Germany, the current COVID-19 cases are managed and reported by the local health authorities. The workload of their employees during the pandemic is high, especially in periods of high infection numbers. In this work a decision support toolkit for local health authorities is introduced. A demonstrator web application was developed with the R Shiny framework and is publicly accessible online. It contains five separate tools based on statistical models for specific use cases and corresponding questions of COVID-19 cases and their contacts. The underlying statistical methods have been implemented in a new open-source R package. The toolkit has the potential to support local health authorities' employees in their daily work. A simulated-based validation of the statistical models and a usability evaluation of the demonstrator application in a user study will be carried out in the future. [ABSTRACT FROM AUTHOR]

Major advances have been made regarding the utilization of machine learning techniques for disease diagnosis and prognosis based on complex and high‐dimensional data. Despite all justified enthusiasm, overoptimistic assessments of predictive performance are still common in this area. However, predictive models and medical devices based on such models should undergo a throughout evaluation before being implemented into clinical practice. In this work, we propose a multiple testing framework for (comparative) phase III diagnostic accuracy studies with sensitivity and specificity as co‐primary endpoints. Our approach challenges the frequent recommendation to strictly separate model selection and evaluation, that is, to only assess a single diagnostic model in the evaluation study. We show that our parametric simultaneous test procedure asymptotically allows strong control of the family‐wise error rate. A multiplicity correction is also available for point and interval estimates. Moreover, we demonstrate in an extensive simulation study that our multiple testing strategy on average leads to a better final diagnostic model and increased statistical power. To plan such studies, we propose a Bayesian approach to determine the optimal number of models to evaluate simultaneously. For this purpose, our algorithm optimizes the expected final model performance given previous (hold‐out) data from the model development phase. We conclude that an assessment of multiple promising diagnostic models in the same evaluation study has several advantages when suitable adjustments for multiple comparisons are employed. [ABSTRACT FROM AUTHOR]

Model selection and performance assessment for prediction models are important tasks in machine learning, e.g. for the development of medical diagnosis or prognosis rules based on complex data. A common approach is to select the best model via cross-validation and to evaluate this final model on an independent dataset. In this work, we propose to instead evaluate several models simultaneously. These may result from varied hyperparameters or completely different learning algorithms. Our main goal is to increase the probability to correctly identify a model that performs sufficiently well. In this case, adjusting for multiplicity is necessary in the evaluation stage to avoid an inflation of the family wise error rate. We apply the so-called maxT-approach which is based on the joint distribution of test statistics and suitable to (approximately) control the family-wise error rate for a wide variety of performance measures. We conclude that evaluating only a single final model is suboptimal. Instead, several promising models should be evaluated simultaneously, e.g. all models within one standard error of the best validation model. This strategy has proven to increase the probability to correctly identify a good model as well as the final model performance in extensive simulation studies. [ABSTRACT FROM AUTHOR]

Background: Transition to adult health services is a vulnerable phase in young persons with chronic disease. We describe how young persons with inflammatory bowel disease in Germany and Austria experience care during the transitional age, focusing on differences by type of provider (pediatric vs. adult specialist, no specialist). Methods: This was a follow up survey in patients previously registered with a pediatric IBD registry. Patients aged 15 to 25 received a postal questionnaire, including a measure of health care experience and satisfaction. Descriptive analyses were stratified by age. Sub-analyses in the 18–20 year age group compared health care experience by type of provider. Determinants of early or late transfer were examined using multinomial logistic regression. Results: 619 patients responded to the survey; 605 questionnaires were available for analysis. Usual age of completing transition was 18. Earlier transfer was more common with low parental SES (OR 1.8, 95% CI 0.7 to 4.6), and less common with advanced schooling (OR 0.5, 95% CI 0.2 to 1.2). Structured transition was uncommon. 48% of the respondents had not received any preceding transition advice. Overall satisfaction with IBD care was high, especially with respect to interpersonal aspects, but less so in aspects of continuity of care. Conclusions: Despite high overall patient satisfaction, relevant deficiencies in transitional care were documented. Some of these were associated with lower parental social status. Differences in health care satisfaction by type of provider (adult vs. pediatric) were small. [ABSTRACT FROM AUTHOR]

Purpose: To describe early signs for restrictive subtype of chronic lung allograft dysfunction (CLAD) after lung transplantation in computed tomography (CT) and to evaluate the predictive value for disease progression and survival.Material and Methods: 52 CT examinations in lung transplant patients at CLAD onset were scored for CT features referring to airways disease, parenchymal or pleural abnormality. Patients with and without later development of restrictive CLAD (TLC≤80%) were compared. A radiological score for inflammation including pleural effusion, central and peripheral ground glass opacities and consolidations was calculated and used for survival analysis.Results: CT of patients with later development of restrictive CLAD showed significantly more often abnormalities at CLAD onset, in particular consolidations (57% vs. 4%; p<0.001) and ground glass attenuations (71% vs. 7%; p<0.001) than those of patients without the restrictive phenotype. CT score for inflammation was significantly higher in patients with than without later restrictive CLAD (3.4 vs. 0.6; p<0.001). Survival of patients with a high score (>2) for inflammation in CT at CLAD onset was significantly lower than of those with a low score (443 vs. 2415 days; p=0.019).Conclusions: CT at CLAD onset differs in patients with/without later development of the restrictive phenotype. It is therefore an indicator for future development of restrictive CLAD and predictor for survival. It should be implemented in the diagnostic work-up at diagnosis of CLAD. [ABSTRACT FROM AUTHOR]