1. Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: The CADDementia challenge
- Author
-
Rozanna Meijboom, Vladimir S. Fonov, Mads Nielsen, Marion Smits, Hugo Vrenken, Katherine R. Gray, Simon Fristed Eskildsen, Christian Ledig, Ahmed Abdulkadir, Garry Smith, Wiro J. Niessen, Giuseppe Di Fatta, Olaf Ronneberger, Pierrick Coupé, Alexandre Routier, Tong Tong, Frederik Barkhof, Katja Franke, Rebecca M. E. Steketee, Sabina Tangaro, Khan M. Iftekharuddin, Alessia Sarica, Andrea Chincarini, Esther E. Bron, Elaheh Moradi, Stanley Durrleman, Roberto Bellotti, Philip Scheltens, Christian Wachinger, Martin Reuter, Paolo Inglese, Zhivko Stoyanov, Andrés Marino Álvarez-Meza, Francesco Sensi, John C. van Swieten, Stefan Klein, Nicola Amoroso, Joana R. Meireles, Lauge Sørensen, Janne M. Papma, António J. Bastos-Leite, Jussi Tohka, Wiesje M. van der Flier, Madalena Pinto, Chester V. Dolph, Ricardo Guerrero, Christian Gaser, Carolina Méndez Orellana, Carolina Garrett, David Cárdenas-Peña, Laboratoire d'Etude du Rayonnement et de la Matière en Astrophysique (LERMA), École normale supérieure - Paris (ENS Paris), Université Paris sciences et lettres (PSL)-Université Paris sciences et lettres (PSL)-Université Pierre et Marie Curie - Paris 6 (UPMC)-Institut national des sciences de l'Univers (INSU - CNRS)-Observatoire de Paris, Université Paris sciences et lettres (PSL)-Université de Cergy Pontoise (UCP), Université Paris-Seine-Université Paris-Seine-Centre National de la Recherche Scientifique (CNRS), Erasmus University Medical Center [Rotterdam] (Erasmus MC), Wageningen University and Research [Wageningen] (WUR), VU University Medical Center [Amsterdam], Laboratoire Exploitation, Perception, Simulateurs et Simulations (IFSTTAR/COSYS/LEPSIS), Institut Français des Sciences et Technologies des Transports, de l'Aménagement et des Réseaux (IFSTTAR)-Communauté Université Paris-Est, Hospital de São João [Porto], Faculdade de Medicina da Universidade do Porto (FMUP), Universidade do Porto, University of Freiburg [Freiburg], Department of Computer Science [Freiburg], Centre for Biological Signaling Studies [Freiburg] (BIOSS), Istituto Nazionale di Fisica Nucleare, sezione di Bari (INFN, sezione di Bari), Istituto Nazionale di Fisica Nucleare (INFN), Università degli studi di Bari Aldo Moro (UNIBA), Universidad Nacional de Colombia [Bogotà] (UNAL), Old Dominion University [Norfolk] (ODU), Aarhus University [Aarhus], Laboratoire Bordelais de Recherche en Informatique (LaBRI), Université de Bordeaux (UB)-Centre National de la Recherche Scientifique (CNRS)-École Nationale Supérieure d'Électronique, Informatique et Radiocommunications de Bordeaux (ENSEIRB), McConnell Brain Imaging Centre (MNI), Montreal Neurological Institute and Hospital, McGill University = Université McGill [Montréal, Canada]-McGill University = Université McGill [Montréal, Canada], Jena University Hospital [Jena], Biomedical Image Analysis Group [London] (BioMedIA), Imperial College London, Department of Signal Processing [Tampere], Tampere University of Technology [Tampere] (TUT), Algorithms, models and methods for images and signals of the human brain (ARAMIS), Inria Paris-Rocquencourt, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-Université Pierre et Marie Curie - Paris 6 (UPMC)-Institut du Cerveau et de la Moëlle Epinière = Brain and Spine Institute (ICM), Université Pierre et Marie Curie - Paris 6 (UPMC)-Institut National de la Santé et de la Recherche Médicale (INSERM)-Centre National de la Recherche Scientifique (CNRS)-CHU Pitié-Salpêtrière [AP-HP], Sorbonne Université (SU)-Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Sorbonne Université (SU)-Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Université Pierre et Marie Curie - Paris 6 (UPMC)-Institut National de la Santé et de la Recherche Médicale (INSERM)-Centre National de la Recherche Scientifique (CNRS)-CHU Pitié-Salpêtrière [AP-HP], Sorbonne Université (SU)-Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Sorbonne Université (SU)-Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP), Università degli Studi 'Magna Graecia' di Catanzaro [Catanzaro, Italie] (UMG), School of Systems Engineering [Reading], University of Reading (UOR), Universita degli studi di Genova, Centre for Integrative Neuroscience and Neurodynamics [Reading] (CINN), Department of Computer Science [Copenhagen] (DIKU), Faculty of Science [Copenhagen], University of Copenhagen = Københavns Universitet (KU)-University of Copenhagen = Københavns Universitet (KU), Computer Aided Medical Procedures & Augmented Reality (CAMPAR), Technische Universität Munchen - Université Technique de Munich [Munich, Allemagne] (TUM), Massachusetts General Hospital [Boston], Computer Science and Artificial Intelligence Laboratory [Cambridge] (CSAIL), Massachusetts Institute of Technology (MIT), Delft University of Technology (TU Delft), École normale supérieure - Paris (ENS-PSL), Universidade do Porto = University of Porto, Università degli studi di Bari Aldo Moro = University of Bari Aldo Moro (UNIBA), Université de Bordeaux (UB)-École Nationale Supérieure d'Électronique, Informatique et Radiocommunications de Bordeaux (ENSEIRB)-Centre National de la Recherche Scientifique (CNRS), Université Pierre et Marie Curie - Paris 6 (UPMC)-Institut National de la Santé et de la Recherche Médicale (INSERM)-CHU Pitié-Salpêtrière [AP-HP], Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Sorbonne Université (SU)-Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Sorbonne Université (SU)-Centre National de la Recherche Scientifique (CNRS)-Université Pierre et Marie Curie - Paris 6 (UPMC)-Institut National de la Santé et de la Recherche Médicale (INSERM)-CHU Pitié-Salpêtrière [AP-HP], Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Sorbonne Université (SU)-Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Sorbonne Université (SU)-Centre National de la Recherche Scientifique (CNRS), Università degli Studi 'Magna Graecia' di Catanzaro = University of Catanzaro (UMG), Università degli studi di Genova = University of Genoa (UniGe), University of Copenhagen = Københavns Universitet (UCPH)-University of Copenhagen = Københavns Universitet (UCPH), Coupé, Pierrick, Medical Informatics, Radiology & Nuclear Medicine, Neurology, Physics and medical technology, Radiology and nuclear medicine, Human genetics, NCA - neurodegeneration, and NCA - Brain imaging technology
- Subjects
NATIONAL INSTITUTE ,Male ,[INFO.INFO-IM] Computer Science [cs]/Medical Imaging ,MCI PATIENTS ,DISEASE ,RECOMMENDATIONS ,030218 nuclear medicine & medical imaging ,0302 clinical medicine ,BRAIN ATROPHY ,Diagnosis, Computer-Assisted ,Challenge ,Medical diagnosis ,Aged, 80 and over ,Radiology, Nuclear Medicine & Medical Imaging ,11 Medical And Health Sciences ,ALZHEIMERS ASSOCIATION WORKGROUPS ,Middle Aged ,Alzheimer's disease ,Classification ,Magnetic Resonance Imaging ,Neurology ,Female ,Algorithm ,Life Sciences & Biomedicine ,Algorithms ,Alzheimer's Disease Neuroimaging Initiative ,Cognitive Neuroscience ,Feature extraction ,BIOMARKERS ,Neuroimaging ,Sensitivity and Specificity ,Article ,17 Psychology And Cognitive Sciences ,03 medical and health sciences ,Alzheimer Disease ,Image Interpretation, Computer-Assisted ,medicine ,[INFO.INFO-IM]Computer Science [cs]/Medical Imaging ,Dementia ,Humans ,Cognitive Dysfunction ,Generalizability theory ,Aged ,Science & Technology ,Neurology & Neurosurgery ,business.industry ,Neurosciences ,Mild cognitive impairment ,AD ,Computer-aided diagnosis ,medicine.disease ,Data set ,Structural MRI ,Test set ,Neurosciences & Neurology ,DIMENSIONAL PATTERN-CLASSIFICATION ,business ,030217 neurology & neurosurgery - Abstract
Algorithms for computer-aided diagnosis of dementia based on structural MRI have demonstrated high performance in the literature, but are difficult to compare as different data sets and methodology were used for evaluation. In addition, it is unclear how the algorithms would perform on previously unseen data, and thus, how they would perform in clinical practice when there is no real opportunity to adapt the algorithm to the data at hand. To address these comparability, generalizability and clinical applicability issues, we organized a grand challenge that aimed to objectively compare algorithms based on a clinically representative multi-center data set. Using clinical practice as the starting point, the goal was to reproduce the clinical diagnosis. Therefore, we evaluated algorithms for multi-class classification of three diagnostic groups: patients with probable Alzheimer's disease, patients with mild cognitive impairment and healthy controls. The diagnosis based on clinical criteria was used as reference standard, as it was the best available reference despite its known limitations. For evaluation, a previously unseen test set was used consisting of 354 T1-weighted MRI scans with the diagnoses blinded. Fifteen research teams participated with a total of 29 algorithms. The algorithms were trained on a small training set (n = 30) and optionally on data from other sources (e.g., the Alzheimer's Disease Neuroimaging Initiative, the Australian Imaging Biomarkers and Lifestyle flagship study of aging). The best performing algorithm yielded an accuracy of 63.0% and an area under the receiver-operating-characteristic curve (AUC) of 78.8%. In general, the best performances were achieved using feature extraction based on voxel-based morphometry or a combination of features that included volume, cortical thickness, shape and intensity. The challenge is open for new submissions via the web-based framework: http://caddementia.grand-challenge.org. (C) 2015 Elsevier Inc. All rights reserved.
- Published
- 2015
- Full Text
- View/download PDF