Hybrid Scheme of Image’s Regional Colorization Using Mask R-CNN and Poisson Editing

Image colorization is a creative process of reasonably adding colors on gray-scale images to generate well-pleasing colorized images. The existing colorization methods normally require user-supplied hints of color points and doodles, or handpicked color reference images for transferring colors, or diverse color images for predicting the colorized results; but the final colorized results generated by most of them may seem unnatural as a consequence of the unprofessional users' skills, inaccurately color transferring, or limited scale of color image collection. To overcome these limitations, a hybrid scheme consisting of two modules is proposed for images' region colorization by combining semantic segmentation and seamless fusion techniques in this paper. In the first module, the masks and category of input image's regions and background are derived from a Mask R-CNN model, and the corresponding reference images of each region are selected from a pre-classified color image database. In the second module, the background and various regions of an image are colorized by a U-Net model and a VGG model respectively. Then, the Poisson editing technique is applied for fusing all the colorized results to generate the final whole colorized image. The experiments show that our scheme can not only flexibly select the appropriate reference images for different regions of the image according their semantic information, but also effectively merge colorized results to generate a plausible colorful image. By reasonably combining different CNN-based models according to their superiority, our scheme avoids the limitation of failures caused by single-step methods, and achieves richer artistic visual effect compared with other existing methods.