Search

Group sparse residual constraint model with nonlocal priors (GSRC-NLP) has made great success in image restoration and produce state-of-the-art performance, realized through reducing the group sparsity residual. In the GSRC-NLP model, L 1 norm is used to reduce the group sparsity residual. However, the L 1 norm penalty has been known to lead to the over-shrinkage of the large sparse coefficients. In this paper, we utilize the adaptive correction procedure to reduce excessive penalties on large coefficient values while improving group sparsity. The proposed corrected group sparse residual constraint model (CGSRC) can improve the sparsity of the group sparsity residual, which lead to better performance in image restoration. We apply the iterative shrinkage/thresholding and the alternating direction method of multipliers to solve the proposed models. In addition, we study the properties of the proposed CGSRC model including the existence and uniqueness of the solution as well as the convergence of the proposed algorithms. Experimental results on image denoising and image inpainting show that the proposed models outperform several state-of-the-art image denoising and image inpainting methods in terms of both objective and perceptual quality metrics. [ABSTRACT FROM AUTHOR]

In the domain of handwritten character recognition, inpainting occluded offline characters is essential. Relying on the remarkable achievements of transformers in various tasks, we present a novel framework called “Enhanced Inpainting with Multi-head Attention and stacked long short-term memory (LSTM) Network” (E-Inpaint). This framework aims to restore occluded offline handwriting while capturing its online signal counterpart, enriched with dynamic characteristics. The proposed approach employs Convolutional Neural Network (CNN) and Multi-Layer Perceptron (MLP) in order to extract essential hidden features from the handwriting image. These features are then decoded by stacked LSTM with Multi-head Attention, achieving the inpainting process and generating the online signal corresponding to the uncorrupted version. To validate our work, we utilize the recognition system Beta-GRU on Latin, Indian, and Arabic On/Off dual datasets. The obtained results show the efficiency of using stacked-LSTM network with multi-head attention, enhancing the quality of the restored image and significantly improving the recognition rate using the innovative Beta-GRU system. Our research mainly highlights the potential of E-Inpaint in enhancing handwritten character recognition systems. [ABSTRACT FROM AUTHOR]

Hyperspectral images are typically composed of hundreds of narrow and contiguous spectral bands, each containing information regarding the material composition of the imaged scene. However, these images can be affected by various sources of noise, distortions, or data loss, which can significantly degrade their quality and usefulness. This paper introduces a convergent guaranteed algorithm, LRS-PnP-DIP(1-Lip), which successfully addresses the instability issue of DHP that has been reported before. The proposed algorithm extends the successful joint low-rank and sparse model to further exploit the underlying data structures beyond the conventional and sometimes restrictive unions of subspace models. A stability analysis guarantees the convergence of the proposed algorithm under mild assumptions, which is crucial for its application in real-world scenarios. Extensive experiments demonstrate that the proposed solution consistently delivers visually and quantitatively superior inpainting results, establishing state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Maize leaf area offers valuable insights into physiological processes, playing a critical role in breeding and guiding agricultural practices. The Azure Kinect DK possesses the real-time capability to capture and analyze the spatial structural features of crops. However, its further application in maize leaf area measurement is constrained by RGB–depth misalignment and limited sensitivity to detailed organ-level features. This study proposed a novel approach to address and optimize the limitations of the Azure Kinect DK through the multimodal coupling of RGB-D data for enhanced organ-level crop phenotyping. To correct RGB–depth misalignment, a unified recalibration method was developed to ensure accurate alignment between RGB and depth data. Furthermore, a semantic information-guided depth inpainting method was proposed, designed to repair void and flying pixels commonly observed in Azure Kinect DK outputs. The semantic information was extracted using a joint YOLOv11-SAM2 model, which utilizes supervised object recognition prompts and advanced visual large models to achieve precise RGB image semantic parsing with minimal manual input. An efficient pixel filter-based depth inpainting algorithm was then designed to inpaint void and flying pixels and restore consistent, high-confidence depth values within semantic regions. A validation of this approach through leaf area measurements in practical maize field applications—challenged by a limited workspace, constrained viewpoints, and environmental variability—demonstrated near-laboratory precision, achieving an MAPE of 6.549%, RMSE of 4.114 cm2, MAE of 2.980 cm2, and R2 of 0.976 across 60 maize leaf samples. By focusing processing efforts on the image level rather than directly on 3D point clouds, this approach markedly enhanced both efficiency and accuracy with the sufficient utilization of the Azure Kinect DK, making it a promising solution for high-throughput 3D crop phenotyping. [ABSTRACT FROM AUTHOR]

Gaze tracking technology in HMDs (Head-Mounted Displays) suffers from decreased accuracy due to highlight reflection noise from users' glasses. To address this, we present a denoising method which directly pinpoints the noisy regions through advanced segmentation models and then fills the flawed regions through advanced image inpainting algorithms. In segmentation stage, we introduce a novel model based on the recently proposed segmentation large model SAM (Segment Anything Model), called SF-SAM-Adapter (Spatial and Frequency aware SAM Adapter). It injects prior knowledge regarding the strip-like shaped in spatial and high-frequency in frequency of reflection noise into SAM by integrating specially designed trainable adapter modules into the original structure, while retaining the expressive power of the large model and better adapting to the downstream task. We achieved segmentation metrics of IoU (Intersection over Union) = 0.749 and Dice = 0.853 at a memory size of 13.9 MB, outperforming recent techniques, including UNet, UNet++, BATFormer, FANet, MSA, and SAM2-Adapter. In inpainting, we employ the advanced inpainting algorithm LAMA (Large Mask inpainting), resulting in significant improvements in gaze tracking accuracy by 0.502°, 0.182°, and 0.319° across three algorithms. The code and datasets used in current study are available in the repository: https://github.com/leiting5297/SF-SAM-Adapter.git. • Network employs large model and Adapter technology. • Architecture integrates prior spatial and frequency Knowledge. • Reflection noise removed, improving eye tracking accuracy. [ABSTRACT FROM AUTHOR]

This paper presents a comprehensive study of the application of AI-driven inpainting techniques to the restoration of historical photographs of the Czech city Most, with a focus on restoration and reconstructing the lost architectural heritage. The project combines state-of-the-art methods, including generative adversarial networks (GANs), patch-based inpainting, and manual retouching, to restore and enhance severely degraded images. The reconstructed/restored photographs of the city Most offer an invaluable visual representation of a city that was largely destroyed for industrial purposes in the 20th century. Through a series of blind and informed user tests, we assess the subjective quality of the restored images and examine how knowledge of edited areas influences user perception. Additionally, this study addresses the technical challenges of inpainting, including computational demands, interpretability, and bias in AI models. Ethical considerations, particularly regarding historical authenticity and speculative reconstruction, are also discussed. The findings demonstrate that AI techniques can significantly contribute to the preservation of cultural heritage, but must be applied with careful oversight to maintain transparency and cultural integrity. Future work will focus on improving the interpretability and efficiency of these methods, while ensuring that reconstructions remain historically and culturally sensitive. [ABSTRACT FROM AUTHOR]

With the recent rise in the development of deep learning techniques, image inpainting—the process of restoring missing or corrupted regions in images—has witnessed significant advancements. Although state-of-the-art models are effective, they often fail to inpaint complex missing areas, especially when handwritten occlusions are present in the image. To address this issue, an image inpainting model based on a residual CycleGAN is proposed. The generator takes as input the image occluded by handwritten missing patches and generates a restored image, which the discriminator then compares with the original ground truth image to determine whether it is real or fake. An adversarial trade-off between the generator and discriminator motivates the model to improve its training and produce a superior reconstructed image. Extensive experiments and analyses confirm that the proposed method generates inpainted images with superior visual quality and outperforms state-of-the-art deep learning approaches. [ABSTRACT FROM AUTHOR]

The integration of Internet of Medical Things (IoMT) technology has revolutionized healthcare, allowing rapid access to medical images and enhancing remote diagnostics in telemedicine. However, this advancement raises serious cybersecurity concerns, particularly regarding unauthorized access and data integrity. This paper presents a novel, user-friendly, visible watermarking scheme for medical images—Visual and User-Friendly Medical Image Watermarking Scheme (VUF-MIWS)—designed to secure medical image ownership while maintaining usability for diagnostic purposes. VUF-MIWS employs a unique combination of inpainting and data hiding techniques to embed hospital logos as visible watermarks, which can be removed seamlessly once image authenticity is verified, restoring the image to its original state. Experimental results demonstrate the scheme's robust performance, with the watermarking process preserving critical diagnostic information with high fidelity. The method achieved Peak Signal-to-Noise Ratios (PSNR) above 70 dB and Structural Similarity Index Measures (SSIM) of 0.99 for inpainted images, indicating minimal loss of image quality. Additionally, VUF-MIWS effectively restored the ROI region of medical images post-watermark removal, as verified through test cases with restored watermarked regions matching the original images. These findings affirm VUF-MIWS's suitability for secure telemedicine applications. [ABSTRACT FROM AUTHOR]

Employing low tensor rank decomposition in image inpainting has attracted increasing attention. This study exploited novel tensor arrangement schemes to transform an image (a low-order tensor) to a higher-order tensor without changing the total number of pixels. The developed arrangement schemes enhanced the low rankness of images under three tensor decomposition methods: matrix SVD, tensor train (TT) decomposition, and tensor singular value decomposition (t-SVD). By exploiting the schemes, we solved the image inpainting problem with three low-rank constrained models that use the matrix rank, TT rank, and tubal rank as constrained priors. The tensor tubal rank and tensor train multi-rank were developed from t-SVD and TT decomposition, respectively. Then, ADMM algorithms were efficiently exploited for solving the three models. Experimental results demonstrate that our methods are effective for image inpainting and superior to numerous close methods. [ABSTRACT FROM AUTHOR]

Maize leaf area offers valuable insights into physiological processes, playing a critical role in breeding and guiding agricultural practices. The Azure Kinect DK possesses the real-time capability to capture and analyze the spatial structural features of crops. However, its further application in maize leaf area measurement is constrained by RGB–depth misalignment and limited sensitivity to detailed organ-level features. This study proposed a novel approach to address and optimize the limitations of the Azure Kinect DK through the multimodal coupling of RGB-D data for enhanced organ-level crop phenotyping. To correct RGB–depth misalignment, a unified recalibration method was developed to ensure accurate alignment between RGB and depth data. Furthermore, a semantic information-guided depth inpainting method was proposed, designed to repair void and flying pixels commonly observed in Azure Kinect DK outputs. The semantic information was extracted using a joint YOLOv11-SAM2 model, which utilizes supervised object recognition prompts and advanced visual large models to achieve precise RGB image semantic parsing with minimal manual input. An efficient pixel filter-based depth inpainting algorithm was then designed to inpaint void and flying pixels and restore consistent, high-confidence depth values within semantic regions. A validation of this approach through leaf area measurements in practical maize field applications—challenged by a limited workspace, constrained viewpoints, and environmental variability—demonstrated near-laboratory precision, achieving an MAPE of 6.549%, RMSE of 4.114 cm2, MAE of 2.980 cm2, and R2 of 0.976 across 60 maize leaf samples. By focusing processing efforts on the image level rather than directly on 3D point clouds, this approach markedly enhanced both efficiency and accuracy with the sufficient utilization of the Azure Kinect DK, making it a promising solution for high-throughput 3D crop phenotyping.

In geophysical exploration, logging images are frequently incomplete due to the mismatch between the size of the logging instruments and that of the boreholes, which significantly impacts geological analysis. Existing methods, which rely on standard algorithms or unsupervised learning techniques, tend to be computationally intensive and time-consuming. In addition, they are difficult to inpaint regions with high-angle fractures or fine-grained textures. To address these challenges, we propose a deep learning approach for inpainting stratigraphic features. Our method utilizes pseudo-labeled training datasets to alleviate the issue of limited training labels, thereby reducing both computational cost and processing time. We introduce a Fusion-Perspective-Enhancement Module (FPEM) designed to accurately infer missing regions based on contextual guidance, thus enhancing the inpainting process for high-angle fractures. Furthermore, we present a novel discriminator known as SM-Unet, which improves fine-grained textures by adjusting the weight assigned to various regions through soft labeling during training. Our approach achieves a Peak Signal-to-Noise Ratio (PSNR) of 25.35 and a Structural Similarity Index (SSIM) of 0.901 on the logging image dataset. This performance surpasses that of state-of-the-art methods — particularly in managing high-angle fractures and fine-grained textures — while requiring less computational effort. • A pseudo-label training method combining GAN and diffusion models. • A multiscale feature fusion module combining large kernel convolution is proposed. • Improved inpainting of high-angle fractures in electric logging images. • Real-time inpainting of electric logging images is possible. [ABSTRACT FROM AUTHOR]

With increasing computer vision (CV) applications in automated construction management, the visual occlusion issue caused by crisscrossing, wide-coverage, and immovable scaffolds has become one of the most challenging. This study proposes a novel deep learning-based two-step method combining pixel-level semantic segmentation and contextual image inpainting to remove scaffolds visually and restore the occluded visual information. A low-cost data synthesis method using only unlabeled data has also been developed to alleviate the shortage of labeled data for deep neural network (DNN) training. Experiments on the synthesized test data show that the proposed method achieves performances of 92% mean intersection over union (MIoU) for scaffold segmentation and over 82% structural similarity (SSIM) for scene restoration after removing scaffolds. This research set a precedent for addressing the visual occlusion issue of scaffolds, and the proposed method is verified in real-world cases that it helps existing CV models perform better in scaffolding scenarios. • A new deep learning-based two-step method is proposed for scaffold occlusion removal. • The two-step method sequentially performs semantic segmentation and image inpainting. • Sufficient labeled data is synthesized using unlabeled data for deep-model training. • The proposed method achieves 91% MIoU and 82% SSIM in segmentation and inpainting. • The proposed method can help existing CV models perform better in scaffolding scenes. [ABSTRACT FROM AUTHOR]

We present a novel approach for synthesizing realistic novel views using Neural Radiance Fields (NeRF) with uncontrolled photos. While NeRF has shown impressive results in controlled settings, it struggles with transient objects commonly found in dynamic and time-varying scenes. Our framework called Inpainting Enhanced NeRF , or IE-NeRF addresses this issue by learning to decompose the static and dynamic components of the scene through exploring transient mask inpainting. Specifically, to model varying appearances, we propose a CNN-based appearance encoder, and a view-consistent appearance loss to transfer consistent photometric appearance in different views. Moreover, our approach extends the Multi-Layer Perceptrons (MLP) network of NeRF, enabling it to simultaneously generate intrinsic properties (static color, density) and extrinsic transient masks. During optimization we take advantage of inpainting to restore static scene image while eliminating occlusions with the guidance of transient masks. Additionally, we propose a new training strategy using frequency regularization with transient mask factor in integrated positional encoding, which facilitates faster inference and early separation of transient components during training. We evaluate our approach on internet photo collections of landmarks, demonstrating its ability to generate high-quality novel views and outperform the existing methods. • The CNN-based appearance encoder for different views input. • Decomposition of the static and dynamic by exploring transient mask inpainting. • Frequency regularization base on transient mask factor in training. • Experiments and ablation studies confirm the effectiveness of our approach. [ABSTRACT FROM AUTHOR]

Low-rank prior has important applications in image restoration tasks, particularly in filling in missing information through low-rank matrix completion models. Although the truncated nuclear norm is a classic low-rank algorithm, practical solutions often rely on convex regularized nuclear norm to approximate the rank function, which limits its approximation ability and leads to blurry edges and loss of details. To improve restoration performance, we introduce a non-convex γ -norm. Theoretical analysis shows that the γ -norm approximates the rank function more accurately than the nuclear norm, leading to a novel non-convex low-rank approximation model. Furthermore, we enhance the model by introducing transform domain sparse regularization, aimed at capturing more local details and texture information, thereby improving inpainting quality. Addressing the limitations of traditional low-rank matrix restoration models in cases of entire row or column missing, we introduce a multi-pixel window strategy based on the new model, utilizing non-local similarity to search for similar blocks in the multi-pixel neighborhood of the target block to restore the entire column and eliminate residual column artifacts. Our method demonstrates excellent performance. We compare it with several state-of-the-art image restoration techniques across multiple tasks, including pixel restoration, text and scratch removal, column inpainting, and cloud removal. Experimental results prove that our method shows significant advantages in both visual quality and quantitative evaluation. • Construct a non-convex gamma low-rank approximation model. • Introduce transform domain sparse regularization to boost low-rank model. • Introduce multi-pixel window strategy to complete column filling task. • Experimental results show that our method exhibits significant advantages. [ABSTRACT FROM AUTHOR]

We present EXE-GAN, a novel exemplar-guided facial inpainting framework using generative adversarial networks. Our approach not only preserves the quality of the input facial image but also completes the image with exemplar-like facial attributes. We achieve this by simultaneously leveraging the global style of the input image, the stochastic style generated from the random latent code, and the exemplar style of the exemplar image. We introduce a novel attribute similarity metric to encourage the networks to learn the style of facial attributes from the exemplar in a self-supervised way. To guarantee the natural transition across the boundaries of inpainted regions, we introduce a novel spatial variant gradient backpropagation technique to adjust the loss gradients based on the spatial location. We extensively evaluate EXE-GAN on public CelebA-HQ and FFHQ datasets with practical applications, which demonstrates the superior visual quality of facial inpainting. The source code is available at https://github.com/LonglongaaaGo/EXE-GAN. • A novel interactive facial inpainting framework guided by exemplar images. • A novel self-supervised attribute similarity metric for facial attributes. • A novel spatial variant gradient backpropagation algorithm for network training. • Several applications benefiting from the proposed facial inpainting algorithm. [ABSTRACT FROM AUTHOR]

Cartoon and texture are the main components of an image, and decomposing them has gained much attention in various image restoration tasks. In this paper, we propose a novel infimal convolution type model based on total generalized variation (TGV) and spatially adaptive oscillation TGV to address cartoon-texture restoration problems. The proposed spatially adaptive oscillation TGV regularizer is capable of capturing structured textures with different orientations and frequencies in localized regions. Additionally, we incorporate the second-order tensor TGV to regularize the orientation and frequency parameter. The lower semicontinuity of the new functional is established and the existence of the solutions to the proposed model is analyzed. Furthermore, we discuss suitable discretizations of the proposed model, and introduce the alternating minimization algorithm where each subproblem can be implemented by the primal-dual method. Numerical experiments on image decomposition, denoising and inpainting demonstrate that the proposed model excels in preserving textures and is competitive against existing variational and learning-based models. • A spatially adaptive oscillation TGV is proposed to represent textures of an image. • The tensor TGV is used to regularize the orientation and frequency parameter. • The existence of the solutions to the proposed model is analyzed. [ABSTRACT FROM AUTHOR]

Target occlusion or pollution has always been a common and difficult problem in 3D reconstruction, seriously affecting the reconstruction effect, especially in single image scenario. To address the issues of incomplete reconstruction caused by pixel missing, a novel framework promoting mutual enhancement between 3D reconstruction and inpainting is proposed, which is capable of reconstructing a realistic 3D face with pixel completion from single image without prior assumptions. The framework is composed of an inpainting module for facial pixel completion and a reconstruction module for 3D face modeling. The inpainting module provides complete texture and accurate 3D geometry inferring premise for 3D reconstruction, and the reconstruction module is also involved in face inpainting, constraining the inpainting module to fill plausible pixels that are closer to the ground truth. Experiments show that the accurate 3D faces with complete and fine texture can be reconstructed by the proposed framework from largely masked images, with a competitive performance even surpassing most of state-of-the-art methods with unmasked images as input. Furthermore, this framework has achieved state-of-the-art performance in image inpainting on multiple face datasets. [Display omitted] • This paper proposes a novel framework for 3D face reconstruction with missing pixel completion, which enables the generation of complete 3D face models from largely masked images. Unlike previous reconstruction methods, it possesses the superior capability to complete the missing pixels. • As an end-to-end masked face 3D reconstruction framework during the inference stage, it does not merely combine inpainting with 3D face reconstruction. Instead, it leverages facial structural information decoupled from 3D face reconstruction to optimize the facial inpainting capability. • This paper designs an efficient structure consistency loss to optimize the robustness of the facial reconstruction model, ensuring that the model consistently preserves fundamental facial features. Notably, this optimization does not entail significant computational overhead. • We have also validated the inpainting capabilities of the proposed framework on the CelebA-HQ and FFHQ datasets. The experiments show that the inpainting model in the framework achieved impressive performance, particularly for largely masked face images, which achieved state-of-the-art results. [ABSTRACT FROM AUTHOR]

In this work, we propose SwapInpaint2 to enhance the naturalness of identity-guided face inpainting. The previous version, SwapInpaint, relied on a generic inpainting model for content infer, bringing the problems of unnatural structure to the results. Our method addresses this issue by utilizing an inversion-based Background-preserving Attribute Extractor and an improved Embedding Integration Generator to provide high-quality attribute embeddings and produce high-naturalistic results. Comparison experiments with state-of-the-art models demonstrate that our new method achieves superior structural consistency and stronger style alignment both in terms of quality and quantity. • Structure-consistent attribute embeddings can improve the naturalness of inpainted results. • A inversion-based background-preserving attribute extractor. • Injecting background to modulation blocks improves style blending. [ABSTRACT FROM AUTHOR]

Document Image Inpainting (DII) has been applied to degraded documents, including financial and historical documents, to enhance the legibility of images for: (1) human readers by providing high visual quality images; and (2) machine recognizers such as Optical Character Recognition (OCR), thereby reducing recognition errors. With the advent of Deep Learning (DL), DL-based DII methods have achieved remarkable enhancements in terms of either human or machine legibility. However, focusing on improving machine legibility causes visual image degradation, affecting human readability. To address this contradiction, we propose an adaptive learning-based DII method, namely ALDII, that applies domain adaptation strategy, our approach acts like a plug-in module that is capable of constraining a total feature space before optimizing legibility of human and machine, respectively. We evaluate our ALDII on a Chinese handwritten character dataset, which includes single-character and text-line images. Compared to other state-of-the-art approaches, experimental results demonstrated superior performance of our ALDII with metrics of both human and machine legibility. [ABSTRACT FROM AUTHOR]

Semantic image synthesis aims to generate high-fidelity images from a segmentation mask, and previous methods typically train a generator to associate a global random map with the conditioning mask. However, the lack of independent control of regional content impedes their application. To address this issue, we propose an effective approach for Multi-modal conditioning-based Diverse Semantic Image Synthesis, which is referred to as McDSIS. In this model, there are a number of constituent generators incorporated to synthesize the content in semantic regions from independent random maps. The regional content can be determined by the style code associated with a random map, extracted from a reference image, or by embedding a textual description via our proposed conditioning mechanisms. As a result, the generation process is spatially disentangled, which facilitates independent synthesis of diverse content in a semantic region, while at the same time preserving other content. Due to this flexible architecture, in addition to achieving superior performance over state-of-the-art semantic image generation models, McDSIS is capable of performing various visual tasks, such as face inpainting, swapping, local editing, etc. • Multiple constituent generators are proposed for spatially disentangled synthesis. • Multi-modal conditioning mechanisms are designed for precise image editing. • Our method is capable of various conditional visual generation tasks. [ABSTRACT FROM AUTHOR]

Short-Term Load Forecasting is of great significance for the economic and stable operation of the power system. Against the background of the breakthrough in generative artificial intelligence based on the Diffusion model, the research on relevant load forecasting methods of the latter is still relatively limited. Therefore, this paper refers to many related excellent works, analyzes the commonalities between image generation tasks and load forecasting tasks, and proposes the Diffusion-based Inpainting Forecasting Method (DIFM). DIFM supports multi-variable inputs and can achieve functions such as load sequence generation, quantile forecasting and missing data imputation, making it a flexible and multifunctional method. The feasibility and performance of this method are validated across multiple datasets, with experimental results revealing that DIFM reduces the mean absolute percentage error by 24.61 % and 17.91 % respectively in short-term load forecasting and load imputation tasks compared to the optimal benchmark models. • Delved into the parallels between forecasting tasks and image inpainting tasks. • Incorporated the Diffusion model into the realm of short-term load forecasting. • Investigated various approaches to condition embedding. • The proposed method achieved generation, forecasting, and imputation. [ABSTRACT FROM AUTHOR]

Image inpainting is a recognized method for restoring the properties of pixels in damaged or incomplete images in computer vision technology. Some recent techniques based on generative adversarial network (GAN) image inpainting have outperformed traditional approaches due to their excellent deep learning capability and adaptability to various image domains. Since residual networks (ResNet) also gained popularity over time due to their property as a generative model, offering better feature representation and compatibility with other architectures, how could we leverage both of these models to result in even greater success in image inpainting? This paper proposes a novel architecture for image inpainting based on GAN and residual networks. Our proposed architecture consists of three models: Transpose Convolution-based GAN, Fast ResNet-Convolutional Neural Network, and Co-Modulation GAN. Transpose Convolution-based GAN is our newly designed architecture. It produces guided and blind image inpainting, and FR-CNN performs the object removal case. Co-Mod GAN acts as a refinement layer because it refines the results from Transpose Convolution-based GAN and FR-CNN. To train and evaluate our proposed architecture on publicly available benchmark datasets: CelebA, Places2, and ImageNet are used. Our approach proves our hypothesis, and our proposed model acquires the highest accuracy of 96.59% in the ImageNet dataset, FR-CNN acquires the highest accuracy of 96.70% in the Places2 dataset, and Co-Mod GAN acquires the highest accuracy of 96.16% in the CelebA dataset. Through an analysis of both qualitative and quantitative comparisons, it is evident that our proposed model exceeds existing architectures in performance. [ABSTRACT FROM AUTHOR]

We present a novel end-to-end identity-agnostic face reenactment system, MaskRenderer, that can generate realistic, high fidelity frames in real-time. Although recent face reenactment works have shown promising results, there are still significant challenges such as identity leakage and imitating mouth movements, especially for large pose changes and occluded faces. MaskRenderer tackles these problems by using (i) a 3DMM to model 3D face structure to better handle pose changes, occlusion, and mouth movements compared to 2D representations; (ii) a triplet loss function to embed the cross-reenactment during training for better identity preservation; and (iii) multi-scale occlusion, improving inpainting and restoring missing areas. Comprehensive quantitative and qualitative experiments conducted on the VoxCeleb1 test set, demonstrate that MaskRenderer outperforms state-of-the-art models on unseen faces, especially when the Source and Driving identities are very different. • A real-time identity-agnostic model for face reenactment. • Addressed pose changes and mouth movements by incorporating a 3D Morphable Model. • Embedding cross-identity reenactment during training by incorporating triplet loss. • Better inpainting of unwanted/missing areas by using multi-scale occlusion masks. • Improved identity/expression preservation, mouth movement in the VoxCeleb1 benchmark. [ABSTRACT FROM AUTHOR]

Image inpainting aims to recover damaged regions of a corrupted image and maintain the integrity of the structure and texture within the filled regions. Previous popular approaches have restored images with both vivid textures and structures by introducing structure priors. However, the structure prior-based approaches meet the following main challenges: (1) the fine-grained textures suffer from adverse inpainting effects because they do not fully consider the interaction between structures and textures, (2) the features of the multi-scale objects in structural and textural information cannot be extracted correctly due to the limited receptive fields in convolution operation. In this paper, we propose a texture and structure bidirectional generation network (TSBGNet) to address the above issues. We first reconstruct the texture and structure of corrupted images; then, we design a texture-enhanced-FCMSPCNN (TE-FCMSPCNN) to optimize the generated textures. We also conjoin a bidirectional information flow (BIF) module and a detail enhancement (DE) module to integrate texture and structure features globally. Additionally, we derive a multi-scale attentional feature fusion (MAFF) module to fuse multi-scale features. Experimental results demonstrate that TSBGNet effectively reconstructs realistic contents and significantly outperforms other state-of-the-art approaches on three popular datasets. Moreover, the proposed approach yields promising results on the Dunhuang Mogao Grottoes Mural dataset. • We propose a texture and structure bidirectional generation network (TSBGNet). • We derive a TE-FCMSPCNN to optimize the reconstructed texture. • We globally integrate texture and structure features. • We propose a multi-scale attentional feature fusion (MAFF) module. [ABSTRACT FROM AUTHOR]