1. Evaluation of algorithms for multi-modality whole heart segmentation: An open-access grand challenge
- Author
-
Thierry Brouard, Qianqian Tong, Mattias P. Heinrich, Guoyan Zheng, Jennifer Keegan, Pheng-Ann Heng, Jean-Yves Ramel, Darko Štern, Weixin Si, Guodong Zeng, Zenglin Shi, Chengjia Wang, David N. Firmin, Xin Yang, Chenchen Sun, Örjan Smedby, Martin Urschler, Lei Li, David E. Newby, Chunliang Wang, Ulas Bagci, Julien Oster, Aliasghar Mortazi, Christian Payer, Raad Mohiaddin, Kawal Rhode, Tom MacGillivray, Xiangyun Liao, Sebastien Ourselin, Gaetan Galisot, Guanyu Yang, Xiahai Zhuang, Guang Yang, Cheng Bian, British Heart Foundation, Computer science department [University College London] (UCL-CS), University College of London [London] (UCL), Shanghai Jiao Tong University [Shanghai], Graz University of Technology [Graz] (TU Graz), Ludwig Boltzmann Institute for Clinical Forensic Imaging [Graz] (LBI-CFI), Institute of Medical Informatics [Lübeck], Universität zu Lübeck [Lübeck], Imagerie Adaptative Diagnostique et Interventionnelle (IADI), Institut National de la Santé et de la Recherche Médicale (INSERM)-Université de Lorraine (UL), Royal Institute of Technology [Stockholm] (KTH ), Department of Biomedical Engineering and Health Systems [Stockholm], Shenzhen University [Shenzhen], Peking University [Beijing], The Chinese University of Hong Kong [Hong Kong], Center for Research in Computer Vision [Orlando] (CRCV), University of Central Florida [Orlando] (UCF), Nanjing Southeast University, Laboratoire d'Informatique Fondamentale et Appliquée de Tours (LIFAT), Centre National de la Recherche Scientifique (CNRS)-Université de Tours-Institut National des Sciences Appliquées - Centre Val de Loire (INSA CVL), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA), Reconnaissance des formes et analyse d'images (RFAI), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Centre National de la Recherche Scientifique (CNRS)-Université de Tours-Institut National des Sciences Appliquées - Centre Val de Loire (INSA CVL), School of Computer Science [Wuhan], Wuhan University [China], Shenzhen Institutes of Advanced Technology (SIAT), Chinese Academy of Sciences [Changchun Branch] (CAS), Institute for Surgical Technology and Biomechanics [Bern] (ISTB), University of Bern, Centre for Cardiovascular Science [Edinburgh] (BHF), University of Edinburgh, Division of Imaging Sciences, King‘s College London, Department of Imaging Royal Brompton Hospital, Royal Brompton Hospital, National Heart and Lung Institute [London] (NHLI), Imperial College London-Royal Brompton and Harefield NHS Foundation Trust, This work was funded in part by the National Natural Science Foundation of China (NSFC) grant (61971142), the Science and Technology Commission of Shanghai Municipality grant (17JC1401600) and the British Heart Foundation Project grant (PG/16/78/32402)., Université de Tours (UT)-Institut National des Sciences Appliquées - Centre Val de Loire (INSA CVL), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Centre National de la Recherche Scientifique (CNRS), and Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Centre National de la Recherche Scientifique (CNRS)-Université de Tours (UT)-Institut National des Sciences Appliquées - Centre Val de Loire (INSA CVL)
- Subjects
FOS: Computer and information sciences ,Computer science ,Computer Vision and Pattern Recognition (cs.CV) ,Computer Science - Computer Vision and Pattern Recognition ,Datasets as Topic ,Health Informatics ,Overfitting ,Benchmark ,Article ,09 Engineering ,030218 nuclear medicine & medical imaging ,Set (abstract data type) ,03 medical and health sciences ,0302 clinical medicine ,[INFO.INFO-TS]Computer Science [cs]/Signal and Image Processing ,[INFO.INFO-IM]Computer Science [cs]/Medical Imaging ,Image Processing, Computer-Assisted ,Humans ,symbols.heraldic_charge ,Radiology, Nuclear Medicine and imaging ,Segmentation ,[INFO.INFO-HC]Computer Science [cs]/Human-Computer Interaction [cs.HC] ,Challenge ,Multi-modality ,11 Medical and Health Sciences ,Radiological and Ultrasound Technology ,business.industry ,Deep learning ,Whole Heart Segmentation ,Heart shape ,[INFO.INFO-CV]Computer Science [cs]/Computer Vision and Pattern Recognition [cs.CV] ,Heart ,Benchmarking ,Magnetic Resonance Imaging ,Computer Graphics and Computer-Aided Design ,[INFO.INFO-TT]Computer Science [cs]/Document and Text Processing ,Nuclear Medicine & Medical Imaging ,Benchmark (computing) ,symbols ,Computer Vision and Pattern Recognition ,Artificial intelligence ,Tomography, X-Ray Computed ,business ,Algorithm ,Algorithms ,030217 neurology & neurosurgery ,Test data - Abstract
Highlights • This work presents the methodologies and evaluation results for the WHS algorithms selected from the submissions to the Multi-Modality Whole Heart Segmentation (MM-WHS) challenge, in conjunction with MICCAI 2017. • This work introduces the related information to the challenge, discusses the results from the conventional methods and deep learning-based algorithms, and provides insights to the future research. • The challenge provides a fair and intuitive comparison framework for methods developed and being developed for WHS. • The challenge provides the training datasets with manually delineated ground truths and evaluation for an ongoing development of MM-WHS algorithms., Graphical abstract This manuscript presents the methodologies and evaluation results for the WHS algorithms selected from the submissions to the Multi-Modality Whole Heart Segmentation (MMWHS) challenge, in conjunction with MICCAI-STACOM 2017. The challenge provides 120 three-dimensional cardiac images covering the whole heart, including 60 CT and 60 MRI volumes, all acquired in clinical environments with manual delineation. Ten algorithms for CT data and eleven algorithms for MRI data, submitted from twelve groups, have been evaluated. The results show that many of the deep learning (DL) based methods achieved high accuracy, even though the number of training datasets were limited. Several of them also reported poor results in the blinded evaluation, probably due to over fitting in their training. The conventional algorithms, mainly based on multi-atlas segmentation, demonstrated robust and stable performance, even though the accuracy is not as good as the best DL method in CT segmentation. The challenge, including provision of the annotated training data and the blinded evaluation for submitted algorithms on the test data, continues as an ongoing benchmarking resource., Knowledge of whole heart anatomy is a prerequisite for many clinical applications. Whole heart segmentation (WHS), which delineates substructures of the heart, can be very valuable for modeling and analysis of the anatomy and functions of the heart. However, automating this segmentation can be challenging due to the large variation of the heart shape, and different image qualities of the clinical data. To achieve this goal, an initial set of training data is generally needed for constructing priors or for training. Furthermore, it is difficult to perform comparisons between different methods, largely due to differences in the datasets and evaluation metrics used. This manuscript presents the methodologies and evaluation results for the WHS algorithms selected from the submissions to the Multi-Modality Whole Heart Segmentation (MM-WHS) challenge, in conjunction with MICCAI 2017. The challenge provided 120 three-dimensional cardiac images covering the whole heart, including 60 CT and 60 MRI volumes, all acquired in clinical environments with manual delineation. Ten algorithms for CT data and eleven algorithms for MRI data, submitted from twelve groups, have been evaluated. The results showed that the performance of CT WHS was generally better than that of MRI WHS. The segmentation of the substructures for different categories of patients could present different levels of challenge due to the difference in imaging and variations of heart shapes. The deep learning (DL)-based methods demonstrated great potential, though several of them reported poor results in the blinded evaluation. Their performance could vary greatly across different network structures and training strategies. The conventional algorithms, mainly based on multi-atlas segmentation, demonstrated good performance, though the accuracy and computational efficiency could be limited. The challenge, including provision of the annotated training data and the blinded evaluation for submitted algorithms on the test data, continues as an ongoing benchmarking resource via its homepage (www.sdspeople.fudan.edu.cn/zhuangxiahai/0/mmwhs/).
- Published
- 2019
- Full Text
- View/download PDF