Hyper-fusion network for semi-automatic segmentation of skin lesions.

• We propose to use deep learning with few user-clicks to achieve accurate skin lesion segmentation results. • We propose to leverage user-inputs in optimizing the learning of skin lesion characteristics. • We propose hyper-integration modules (HIMs) to iteratively propagate user-input features and skin lesion image features to ensure the appearance of the segmented skin lesions is spatially consistent. • Our method is capable to segment skin lesions that are known to be challenging, such as those with fuzzy boundaries, inhomogeneous textures and low-contrast to the background. • We had consistently better segmentation results on three well-established public datasets (ISBI 2017, ISBI 2016 skin lesion challenge datasets and PH2 dataset). Segmentation of skin lesions is an important step for imaging-based clinical decision support systems. Automatic skin lesion segmentation methods based on fully convolutional networks (FCNs) are regarded as the state-of-the-art for accuracy. When there are, however, insufficient training data to cover all the variations in skin lesions, where lesions from different patients may have major differences in size/shape/texture, these methods failed to segment the lesions that have image characteristics, which are less common in the training datasets. FCN-based semi-automatic segmentation methods, which fuse user-inputs with high-level semantic image features derived from FCNs offer an ideal complement to overcome limitations of automatic segmentation methods. These semi-automatic methods rely on the automated state-of-the-art FCNs coupled with user-inputs for refinements, and therefore being able to tackle challenging skin lesions. However, there are a limited number of FCN-based semi-automatic segmentation methods and all these methods focused on 'early-fusion', where the first few convolutional layers are used to fuse image features and user-inputs and then derive fused image features for segmentation. For early-fusion based methods, because the user-input information can be lost after the first few convolutional layers, consequently, the user-input information will have limited guidance and constraint in segmenting the challenging skin lesions with inhomogeneous textures and fuzzy boundaries. Hence, in this work, we introduce a hyper-fusion network (HFN) to fuse the extracted user-inputs and image features over multiple stages. We separately extract complementary features which then allows for an iterative use of user-inputs along all the fusion stages to refine the segmentation. We evaluated our HFN on three well-established public benchmark datasets – ISBI Skin Lesion Challenge 2017, 2016 and PH2 – and our results show that the HFN is more accurate and generalizable than the state-of-the-art methods, in particular with challenging skin lesions. [Display omitted] [ABSTRACT FROM AUTHOR]