Search

Image and video inpainting is a classic problem in computer vision and computer graphics, aiming to fill in the plausible and realistic content in the missing areas of images and videos. With the advance of deep learning, this problem has achieved significant progress recently. The goal of this paper is to comprehensively review the deep learning-based methods for image and video inpainting. Specifically, we sort existing methods into different categories from the perspective of their high-level inpainting pipeline, present different deep learning architectures, including CNN, VAE, GAN, diffusion models, etc., and summarize techniques for module design. We review the training objectives and the common benchmark datasets. We present evaluation metrics for low-level pixel and high-level perceptional similarity, conduct a performance evaluation, and discuss the strengths and weaknesses of representative inpainting methods. We also discuss related real-world applications. Finally, we discuss open challenges and suggest potential future research directions. [ABSTRACT FROM AUTHOR]

Specular highlight detection is an essential task with various applications in computer vision. This paper aims to detect specular highlights in single high-resolution images using deep learning while avoiding excessive GPU memory consumption. To achieve this, we present a high-resolution specular highlight detection dataset with manual annotations of specular highlights. Given our dataset, we propose a patch-level bidirectional refinement network for high-resolution specular highlight detection. The main idea is to utilize both the pathway from small-scale patch to large-scale patch and its reverse pathway to progressively refine the detection results of adjacent-scale specular highlight patches. Moreover, based on our detection network, we propose a modified inpainting framework for specular highlight removal as an application. Lastly, we provide ten potential research directions for specular highlight detection, inspiring researchers for further study. [ABSTRACT FROM AUTHOR]

Recently deep learning methods have achieved great success in image inpainting problem. However, reconstructing continuities of complex structures with non-stationary textures remains a challenging task for computer vision. In this paper the family of onion convolutions is presented, the concept of which arises from a connection between patch-based techniques and attention mechanisms. The onion convolutions are building blocks designed for the iterative completion of the missing region from its boundary to the center. It allows to continuously propagate structures and textures from the known region to the missing one and meet human criteria on high-quality image completions. As qualitative and quantitative comparisons show, our method with onion convolutions outperforms state-of-the-art methods by producing more realistic, visually plausible and semantically coherent results. [ABSTRACT FROM AUTHOR]

Multiple deep learning-based stabilization methods have been proposed recently. Some of them directly predict the optical flow to warp each unstable frame into its stabilized version, which we called direct warping. These methods primarily perform online or semi-online stabilization, prioritizing lower computational cost while achieving satisfactory results in certain scenarios. However, they fail to smooth intense instabilities and have considerably inferior results in comparison to other approaches. To improve their quality and reduce this difference, we propose: (a) NAFT, a new direct warping semi-online stabilization method, which adapts RAFT to videos by including a neighborhood-aware update mechanism, called IUNO. By using our training approach along with IUNO, we can learn the characteristics that contribute to video stability from the data patterns, rather than requiring an explicit stability definition. Furthermore, we demonstrate how leveraging an off-the-shelf video inpainting method to achieve full-frame stabilization; (b) SynthStab, a new synthetic dataset consisting of paired videos that allows supervision by camera motion instead of pixel similarities. To build SynthStab, we modeled camera motion using kinematic concepts. In addition, the unstable motion respects scene constraints, such as depth variation. We performed several experiments on SynthStab to develop and validate NAFT. We compared our results with five other methods from the literature with publicly available code. Our experimental results show that we were able to stabilize intense camera motion, outperforming other direct warping methods and bringing its performance closer to state-of-the-art methods. In terms of computational resources, our smallest network has only about 7% of model size and trainable parameters than the smallest values among the competing methods. [ABSTRACT FROM AUTHOR]

The deep image prior showed that a randomly initialized network with a suitable architecture can be trained to solve inverse imaging problems by simply optimizing it's parameters to reconstruct a single degraded image. However, it suffers from two practical limitations. First, it remains unclear how to control the prior beyond the choice of the network architecture. Second, training requires an oracle stopping criterion as during the optimization the performance degrades after reaching an optimum value. To address these challenges we introduce a frequency-band correspondence measure to characterize the spectral bias of the deep image prior, where low-frequency image signals are learned faster and better than high-frequency counterparts. Based on our observations, we propose techniques to prevent the eventual performance degradation and accelerate convergence. We introduce a Lipschitz-controlled convolution layer and a Gaussian-controlled upsampling layer as plug-in replacements for layers used in the deep architectures. The experiments show that with these changes the performance does not degrade during optimization, relieving us from the need for an oracle stopping criterion. We further outline a stopping criterion to avoid superfluous computation. Finally, we show that our approach obtains favorable results compared to current approaches across various denoising, deblocking, inpainting, super-resolution and detail enhancement tasks. Code is available at https://github.com/shizenglin/Measure-and-Control-Spectral-Bias. [ABSTRACT FROM AUTHOR]

Images can be censored by masking the region(s) of interest with a solid color or pattern. When a censored image is used for classification or matching, the mask itself may impact the results. Recent work in image inpainting and data augmentation provide two different approaches for dealing with censored images. In this paper, we perform an extensive evaluation of these methods to understand if the impact of censoring can be mitigated for image classification and retrieval. Results indicate that modern learning-based inpainting approaches outperform augmentation strategies and that metrics typically used to evaluate inpainting performance (e.g., reconstruction accuracy) do not necessarily correspond to improved classification or retrieval, especially in the case of person-shaped masked regions. [ABSTRACT FROM AUTHOR]

Image-to-image translation has been made much progress with embracing Generative Adversarial Networks (GANs). However, it's still very challenging for translation tasks that require high quality, especially at high-resolution and photo-reality. In this work, we present Discriminative Region Proposal Adversarial Network (DRPAN) for high-quality image-to-image translation. We decompose the image-to-image translation procedure into three iterated steps: the first is to generate an image with global structure but some local artifacts (via GAN), the second is to use our Discriminative Region Proposal network (DRPnet) for proposing the most fake region from the generated image, and the third is to implement "image inpainting" on the most fake region for yielding more realistic result through a reviser, so that the system (DRPAN) can be gradually optimized to synthesize images with more attention on the most artifact local part. We explore patch-based GAN to construct DRPnet for proposing the discriminative region to produce masked fake samples, further, we propose a reviser for GANs to distinguish real from masked fake for providing constructive revisions to the generator for producing realistic details, and serve as auxiliaries of the generator to synthesize high-quality results. In addition, we combine pix2pixHD with DRPAN to synthesize high-resolution results with much finer details. Moreover, we improve CycleGAN by DRPAN to address unpaired image-to-image translation with better semantic alignment. Experiments on a variety of paired and unpaired image-to-image translation tasks validate that our method outperforms the state of the art for synthesizing high-quality translation results in terms of both human perceptual studies and automatic quantitative measures. Our code is available at https://github.com/godisboy/DRPAN. [ABSTRACT FROM AUTHOR]

This paper proposes a learning-based quality evaluation framework for inpainted results that does not require any subjectively annotated training data. Image inpainting, which removes and restores unwanted regions in images, is widely acknowledged as a task whose results are quite difficult to evaluate objectively. Thus, existing learning-based image quality assessment (IQA) methods for inpainting require subjectively annotated data for training. However, subjective annotation requires huge cost and subjects' judgment occasionally differs from person to person in accordance with the judgment criteria. To overcome these difficulties, the proposed framework generates and uses simulated failure results of inpainted images whose subjective qualities are controlled as the training data. We also propose a masking method for generating training data towards fully automated training data generation. These approaches make it possible to successfully estimate better inpainted images, even though the task is quite subjective. To demonstrate the effectiveness of our approach, we test our algorithm with various datasets and show it outperforms existing IQA methods for inpainting. [ABSTRACT FROM AUTHOR]

We present the first method to handle curvature regularity in region-based image segmentation and inpainting that is independent of initialization. To this end we start from a new formulation of length-based optimization schemes, based on surface continuation constraints, and discuss the connections to existing schemes. The formulation is based on a cell complex and considers basic regions and boundary elements. The corresponding optimization problem is cast as an integer linear program. We then show how the method can be extended to include curvature regularity, again cast as an integer linear program. Here, we are considering pairs of boundary elements to reflect curvature. Moreover, a constraint set is derived to ensure that the boundary variables indeed reflect the boundary of the regions described by the region variables. We show that by solving the linear programming relaxation one gets reasonably close to the global optimum, and that curvature regularity is indeed much better suited in the presence of long and thin objects compared to standard length regularity. [ABSTRACT FROM AUTHOR]

Traditional depth estimation methods typically exploit the effect of either the variations in internal parameters such as aperture and focus (as in depth from defocus), or variations in extrinsic parameters such as position and orientation of the camera (as in stereo). When operating off-the-shelf (OTS) cameras in a general setting, these parameters influence the depth of field (DOF) and field of view (FOV). While DOF mandates one to deal with defocus blur, a larger FOV necessitates camera motion during image acquisition. As a result, for unfettered operation of an OTS camera, it becomes inevitable to account for pixel motion as well as optical defocus blur in the captured images. We propose a depth estimation framework using calibrated images captured under general camera motion and lens parameter variations. Our formulation seeks to generalize the constrained areas of stereo and shape from defocus (SFD)/focus (SFF) by handling, in tandem, various effects such as focus variation, zoom, parallax and stereo occlusions, all under one roof. One of the associated challenges in such an unrestrained scenario is the problem of removing user-defined foreground occluders in the reference depth map and image (termed inpainting of depth and image). Inpainting is achieved by exploiting the cue from motion parallax to discover (in other images) the correspondence/color information missing in the reference image. Moreover, considering the fact that the observations could be differently blurred, it is important to ensure that the degree of defocus in the missing regions (in the reference image) is coherent with the local neighbours ( defocus inpainting). [ABSTRACT FROM AUTHOR]

Non-local methods for image denoising and inpainting have gained considerable attention in recent years. This is in part due to their superior performance in textured images, a known weakness of purely local methods. Local methods on the other hand have demonstrated to be very appropriate for the recovering of geometric structures such as image edges. The synthesis of both types of methods is a trend in current research. Variational analysis in particular is an appropriate tool for a unified treatment of local and non-local methods. In this work we propose a general variational framework for non-local image inpainting, from which important and representative previous inpainting schemes can be derived, in addition to leading to novel ones. We explicitly study some of these, relating them to previous work and showing results on synthetic and real images. [ABSTRACT FROM AUTHOR]

In this paper, we propose an orientation-matching functional minimization for image denoising and image inpainting. Following the two-step TV-Stokes algorithm (Rahman et al. in Scale space and variational methods in computer vision, pp. 473-482, Springer, Heidelberg, ; Tai et al. in Image processing based on partial differential equations, pp. 3-22, Springer, Heidelberg, ; Bertalmio et al. in Proc. conf. comp. vision pattern rec., pp. 355-362, ), a regularized tangential vector field with zero divergence condition is first obtained. Then a novel approach to reconstruct the image is proposed. Instead of finding an image that fits the regularized normal direction from the first step, we propose to minimize an orientation matching cost measuring the alignment between the image gradient and the regularized normal direction. This functional yields a new nonlinear partial differential equation (PDE) for reconstructing denoised and inpainted images. The equation has an adaptive diffusivity depending on the orientation of the regularized normal vector field, providing reconstructed images which have sharp edges and smooth regions. The additive operator splitting (AOS) scheme is used for discretizing Euler-Lagrange equations. We present the results of various numerical experiments that illustrate the improvements obtained with the new functional. [ABSTRACT FROM AUTHOR]

Background subtraction is an elementary method for detection of foreground objects and their segmentations. Obviously it requires an observation image as well as a background one. In this work we attempt to remove the last requirement by reconstructing the background from the observation image and a guess on the location of the object to be segmented via variational inpainting method. A numerical evaluation of this reconstruction provides a “disocclusion measure” and the correct foreground segmentation region is expected to maximize this measure. This formulation is in fact an optimal control problem, where controls are shapes/regions and states are the corresponding inpaintings. Optimization of the disocclusion measure leads formally to a coupled contour evolution equation, an inpainting equation (the state equation) as well as a linear PDE depending on the inpainting (the adjoint state equation). The contour evolution is implemented in the framework of level sets. Finally, the proposed method is validated on various examples. We focus among others in the segmentation of calcified plaques observed in radiographs from human lumbar aortic regions. [ABSTRACT FROM AUTHOR]

Recently, “epitomes” were introduced as patch-based probability models that are learned by compiling together a large number of examples of patches from input images. In this paper, we describe how epitomes can be used to model video data and we describe significant computational speedups that can be incorporated into the epitome inference and learning algorithm. In the case of videos, epitomes are estimated so as to model most of the small space-time cubes from the input data. Then, the epitome can be used for various modeling and reconstruction tasks, of which we show results for video super-resolution, video interpolation, and object removal. Besides computational efficiency, an interesting advantage of the epitome as a representation is that it can be reliably estimated even from videos with large amounts of missing data. We illustrate this ability on the task of reconstructing the dropped frames in video broadcast using only the degraded video and also in denoising a severely corrupted video. [ABSTRACT FROM AUTHOR]

We are interested in PDE's (Partial Differential Equations) in order to smooth multi-valued images in an anisotropic manner. Starting from a review of existing anisotropic regularization schemes based on diffusion PDE's, we point out the pros and cons of the different equations proposed in the literature. Then, we introduce a new tensor-driven PDE, regularizing images while taking the curvatures of specific integral curves into account. We show that this constraint is particularly well suited for the preservation of thin structures in an image restoration process. A direct link is made between our proposed equation and a continuous formulation of the LIC's (Line Integral Convolutions by Cabral and Leedom (1993). It leads to the design of a very fast and stable algorithm that implements our regularization method, by successive integrations of pixel values along curved integral lines. Besides, the scheme numerically performs with a sub-pixel accuracy and preserves then thin image structures better than classical finite-differences discretizations. Finally, we illustrate the efficiency of our generic curvature-preserving approach – in terms of speed and visual quality – with different comparisons and various applications requiring image smoothing : color images denoising, inpainting and image resizing by nonlinear interpolation. [ABSTRACT FROM AUTHOR]

Very recently we have proposed to use a complex Ginzburg-Landau equation for high contrast inpainting, to restore higher dimensional (volumetric) data (which has applications in frame interpolation), improving sparsely sampled data and to fill in fragmentary surfaces. In this paper we review digital inpainting algorithms and compare their performance with a Ginzburg-Landau inpainting model. For the solution of the Ginzburg-Landau equation we compare the performance of several numerical algorithms. A stability and convergence analysis is given and the consequences for applications to digital inpainting are discussed. [ABSTRACT FROM AUTHOR]