Monique Elmaleh, Freddy Cliquet, Nicolas Guigui, David Germanaud, Ayoub Ghriss, Balázs Kégl, Anita Beggiato, Roberto Toro, Richard Delorme, Amicie de Pierrefeu, Valentina Zantedeschi, Joris Van den Bossche, Nicolas Traut, Laurent Bonasse-Gahot, Alexandre Boucaud, Alban Bethegies, Guillaume Lemaitre, Thomas Bourgeron, Katja Heuer, Meng Wang, Gael Varauquaux, Weidong Cai, Stanislas Chambon, Institut Pasteur [Paris], Centre de Recherche Interdisciplinaire / Center for Research and Interdisciplinarity [Paris, France] (CRI), Institut National de la Santé et de la Recherche Médicale (INSERM)-Université Paris Cité (UPC), Max Planck Institute for Human Cognitive and Brain Sciences [Leipzig] (IMPNSC), Max-Planck-Gesellschaft, Méthodes computationnelles et mathématiques pour comprendre la société et la santé à partir de données (SODA), Inria Saclay - Ile de France, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria), Université Paris-Saclay, AP-HP Hôpital universitaire Robert-Debré [Paris], Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP), Service NEUROSPIN (NEUROSPIN), Direction de Recherche Fondamentale (CEA) (DRF (CEA)), Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Université Paris-Saclay, hosa.io, Centre d'Analyse et de Mathématique sociales (CAMS), École des hautes études en sciences sociales (EHESS)-Centre National de la Recherche Scientifique (CNRS), Stanford School of Medicine [Stanford], Stanford Medicine, Stanford University-Stanford University, rythm.co, University of Colorado [Boulder], University of Chinese Academy of Sciences [Beijing] (UCAS), Institute of Automation - Chinese Academy of Sciences, Laboratoire Hubert Curien [Saint Etienne] (LHC), Institut d'Optique Graduate School (IOGS)-Université Jean Monnet [Saint-Étienne] (UJM)-Centre National de la Recherche Scientifique (CNRS), HUAWEI Technologies France (HUAWEI), Montreal Neurological Institute and Hospital, McGill University = Université McGill [Montréal, Canada], Département de Neuroscience - Department of Neuroscience, Institut Pasteur [Paris] (IP)-Centre National de la Recherche Scientifique (CNRS)-Université Paris Cité (UPCité), Institut National de la Santé et de la Recherche Médicale (INSERM)-Université Paris Cité (UPCité), Université Paris-Saclay-Direction de Recherche Fondamentale (CEA) (DRF (CEA)), Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Commissariat à l'énergie atomique et aux énergies alternatives (CEA), Université Paris sciences et lettres (PSL), Institut d'Optique Graduate School (IOGS)-Université Jean Monnet - Saint-Étienne (UJM)-Centre National de la Recherche Scientifique (CNRS), Huawei Technologies France, Huawei Technologies France [Boulogne-Billancourt], Lassailly-Bondaz, Anne, and Laboratoire Hubert Curien (LHC)
MRI has been extensively used to identify anatomical and functional differences in Autism Spectrum Disorder (ASD). Yet, many of these findings have proven difficult to replicate because studies rely on small cohorts and are built on many complex, undisclosed, analytic choices. We conducted an international challenge to predict ASD diagnosis from MRI data, where we provided preprocessed anatomical and functional MRI data from > 2,000 individuals. Evaluation of the predictions was rigorously blinded. 146 challengers submitted prediction algorithms, which were evaluated at the end of the challenge using unseen data and an additional acquisition site. On the best algorithms, we studied the importance of MRI modalities, brain regions, and sample size. We found evidence that MRI could predict ASD diagnosis: the 10 best algorithms reliably predicted diagnosis with AUC∼0.80 – far superior to what can be currently obtained using genotyping data in cohorts 20-times larger. We observed that functional MRI was more important for prediction than anatomical MRI, and that increasing sample size steadily increased prediction accuracy, providing an efficient strategy to improve biomarkers. We also observed that despite a strong incentive to generalise to unseen data, model development on a given dataset faces the risk of overfitting: performing well in cross-validation on the data at hand, but not generalising. Finally, we were able to predict ASD diagnosis on an external sample added after the end of the challenge (EU-AIMS), although with a lower prediction accuracy (AUC=0.72). This indicates that despite being based on a large multisite cohort, our challenge still produced biomarkers fragile in the face of dataset shifts.