Instance segmentation by blend U‐Net and VOLO network

Abstract Instance segmentation is still challengeable to correctly distinguish different instances on overlapping, dense and large number of target objects. To address this, the authors simplify the instance segmentation problem to an instance classification problem and propose a novel end‐to‐end trained instance segmentation algorithm CotuNet. Firstly, the algorithm combines convolutional neural networks (CNN), Outlooker and Transformer to design a new hybrid Encoder (COT) to further feature extraction. It consists of extracting low‐level features of the image using CNN, which is passed through the Outlooker to extract more refined local data representations. Then global contextual information is generated by aggregating the data representations in local space using Transformer. Finally, the combination of cascaded upsampling and skip connection modules is used as Decoders (C‐UP) to enable the blend of multiple different scales of high‐resolution information to generate accurate masks. By validating on the CVPPP 2017 dataset and comparing with previous state‐of‐the‐art methods, CotuNet shows superior competitiveness and segmentation performance.