Search

Artificial intelligence (AI) solutions that automatically extract information from digital histology images have shown great promise for improving pathological diagnosis. Prior to routine use, it is important to evaluate their predictive performance and obtain regulatory approval. This assessment requires appropriate test datasets. However, compiling such datasets is challenging and specific recommendations are missing. A committee of various stakeholders, including commercial AI developers, pathologists, and researchers, discussed key aspects and conducted extensive literature reviews on test datasets in pathology. Here, we summarize the results and derive general recommendations for the collection of test datasets. We address several questions: Which and how many images are needed? How to deal with low-prevalence subsets? How can potential bias be detected? How should datasets be reported? What are the regulatory requirements in different countries? The recommendations are intended to help AI developers demonstrate the utility of their products and to help regulatory agencies and end users verify reported performance measures. Further research is needed to formulate criteria for sufficiently representative test datasets so that AI solutions can operate with less user intervention and better support diagnostic workflows in the future.

Diagnostic accuracy studies assess sensitivity and specificity of a new index test in relation to an established comparator or the reference standard. The development and selection of the index test is usually assumed to be conducted prior to the accuracy study. In practice, this is often violated, for instance if the choice of the (apparently) best biomarker, model or cutpoint is based on the same data that is used later for validation purposes. In this work, we investigate several multiple comparison procedures which provide family-wise error rate control for the emerging multiple testing problem. Due to the nature of the co-primary hypothesis problem, conventional approaches for multiplicity adjustment are too conservative for the specific problem and thus need to be adapted. In an extensive simulation study, five multiple comparison procedures are compared with regards to statistical error rates in least-favorable and realistic scenarios. This covers parametric and nonparamtric methods and one Bayesian approach. All methods have been implemented in the new open-source R package DTAmc which allows to reproduce all simulation results. Based on our numerical results, we conclude that the parametric approaches (maxT, Bonferroni) are easy to apply but can have inflated type I error rates for small sample sizes. The two investigated Bootstrap procedures, in particular the so-called pairs Bootstrap, allow for a family-wise error rate control in finite samples and in addition have a competitive statistical power., Comment: v1: 13 pages, 3 figures; v2: 15 pages, 4 figures, added a real-world data example (incl. figure), new design for some figures, fixed minor errors, overall (simulation) results remain unchanged; v3: added high dpi figures and a few minor fixes compared to v2

In February 2020 the first human infection with SARS-CoV-2 was reported in Germany. Since then the local public health offices have been responsible to monitor and react to the dynamics of the pandemic. One of their major tasks is to contain the spread of the virus after potential spreading events, for example when one or multiple participants have a positive test result after a group meeting (e.g. at school, at a sports event or at work). In this case, contacts of the infected person have to be traced and potentially are quarantined (at home) for a period of time. When all relevant contact persons obtain a negative polymerase chain reaction (PCR) test result, the quarantine may be stopped. However, tracing and testing of all contacts is time-consuming, costly and (thus) not always feasible. This motivates our work, in which we present a statistical model for the probability that no transmission of Sars-CoV-2 occurred given an arbitrary number of test results at potentially different timepoints. Hereby, the time-dependent sensitivity and specificity of the conducted PCR test are taken in account. We employ a parametric Bayesian model which can be adopted to different situations when specific prior knowledge is available. This is illustrated for group events in German school classes and applied to exemplary real-world data from this context. Our approach has the potential to support important quarantine decisions with the goal to achieve a better balance between necessary containment of the pandemic and preservation of social and economic life. The focus of future work should be on further refinement and evaluation of quarantine decisions based on our statistical model., Comment: This manuscript is a preprint and is submitted to IJERPH

Major advances have been made regarding the utilization of artificial intelligence in health care. In particular, deep learning approaches have been successfully applied for automated and assisted disease diagnosis and prognosis based on complex and high-dimensional data. However, despite all justified enthusiasm, overoptimistic assessments of predictive performance are still common. Automated medical testing devices based on machine-learned prediction models should thus undergo a throughout evaluation before being implemented into clinical practice. In this work, we propose a multiple testing framework for (comparative) phase III diagnostic accuracy studies with sensitivity and specificity as co-primary endpoints. Our approach challenges the frequent recommendation to strictly separate model selection and evaluation, i.e. to only assess a single diagnostic model in the evaluation study. We show that our parametric simultaneous test procedure asymptotically allows strong control of the family-wise error rate. Moreover, we demonstrate in extensive simulation studies that our multiple testing strategy on average leads to a better final diagnostic model and increased statistical power. To plan such studies, we propose a Bayesian approach to determine the optimal number of models to evaluate. For this purpose, our algorithm optimizes the expected final model performance given previous (hold-out) data from the model development phase. We conclude that an assessment of multiple promising diagnostic models in the same evaluation study has several advantages when suitable adjustments for multiple comparisons are implemented., Comment: 32 pages, 5 figures; v2: minor linguistic revision, no content-related changes

Statistical inference in high-dimensional settings is challenging when standard unregularized methods are employed. In this work, we focus on the case of multiple correlated proportions for which we develop a Bayesian inference framework. For this purpose, we construct an $m$-dimensional Beta distribution from a $2^m$-dimensional Dirichlet distribution, building on work by Olkin and Trikalinos (2015). This readily leads to a multivariate Beta-binomial model for which simple update rules from the common Dirichlet-multinomial model can be adopted. From the frequentist perspective, this approach amounts to adding pseudo-observations to the data and allows a joint shrinkage estimation of mean vector and covariance matrix. For higher dimensions ($m > 10$), the extensive model based on $2^m$ parameters starts to become numerically infeasible. To counter this problem, we utilize a reduced parametrisation which has only $1 + m(m + 1)/2$ parameters describing first and second order moments. A copula model can then be used to approximate the (posterior) multivariate Beta distribution. A natural inference goal is the construction of multivariate credible regions. The properties of different credible regions are assessed in a simulation study in the context of investigating the accuracy of multiple binary classifiers. It is shown that the extensive and copula approach lead to a (Bayes) coverage probability very close to the target level. In this regard, they outperform credible regions based on a normal approximation of the posterior distribution, in particular for small sample sizes. Additionally, they always lead to credible regions which lie entirely in the parameter space which is not the case when the normal approximation is used., Comment: 25 pages, 7 figures; v2: one reference updated, minor cosmetic changes, contents unaffected; v3: slight abstract overhaul, typos fixed, different template, contents unaffected; v4: added comment regarding limitation of copula approach in discussion section, no other changes