Rethinking pre-training on medical imaging.

Transfer learning from natural image datasets, such as ImageNet, is common for applying deep learning to medical imaging. However, the modalities of natural and medical images differ considerably, and the reason for the latest medical research preferring ImageNet to medical data is questionable. In this study, we investigated the properties of medical pre-training and its transfer effectiveness on various medical tasks. Through an intuitive convolution-based analysis, we determined the modality characteristics of images. Surprisingly, medical pre-training showed exceptional performance for a classification task but not for a segmentation task since medical data are visually homogeneous and lack morphological information. Using data with diverse modalities helped overcome such drawbacks, resulting in medical pre-training achieving performance comparable to pre-training with ImageNet with considerably fewer samples than ImageNet for both aforementioned tasks. Finally, a study of learned representations and realistic scenarios indicated that while ImageNet is the best choice for medical imaging, medical pre-training has significant potential. • We study the transfer learning effectiveness based on medical and natural images. • We evaluate medical pre-training on downstream medical image analysis tasks. • We reveal the shortcomings and future potential of medical pre-training. [ABSTRACT FROM AUTHOR]