Search

This research paper presents a novel approach for occlusion inpainting in thermal images to efficiently segment and enhance obscured regions within these images. The increasing reliance on thermal imaging in fields like surveillance, security, and defense necessitates the accurate detection of obscurants such as smoke and fog. Traditional methods often struggle with these complexities, leading to the need for more advanced solutions. Our proposed methodology uses a Generative Adversarial Network (GAN) to fill occluded areas in thermal images. This process begins with an obscured region segmentation, followed by a GAN-based pixel replacement in these areas. The methodology encompasses building, training, evaluating, and optimizing the model to ensure swift real-time performance. One of the key challenges in thermal imaging is identifying effective strategies to mitigate critical information loss due to atmospheric interference. Our approach addresses this by employing sophisticated deep-learning techniques. These techniques segment, classify and inpaint these obscured regions in a patch-wise manner, allowing for more precise and accurate image restoration. We propose utilizing architectures similar to Pix2Pix and UNet networks for generative and segmentation tasks. These networks are known for their effectiveness in image-to-image translation and segmentation tasks. Our method enhances the segmentation and inpainting process by leveraging their architectural similarities. To validate our approach, we provide a quantitative analysis and performance comparison. We include a quantitative comparison between (Pix2Pix and UNet) and our combined architecture. The comparison focuses on how well each model performs in terms of accuracy and speed, highlighting the advantages of our integrated approach. This research contributes to advancing thermal imaging techniques, offering a more robust solution for dealing with obscured regions. The integration of advanced deep learning models holds the potential to significantly improve image analysis in critical applications like surveillance and security.

Mural paintings are vital cultural expressions, enriching our lives by beautifying spaces, conveying messages, telling stories, and evoking emotions. Ancient temple murals degrade over time due to natural aging, physical damage, etc. Preserving these cultural treasures is challenging. Image inpainting is often used for digital restoration, but existing methods typically overlook naturally degraded areas, using randomly generated binary masks or small, narrow regions for repair. This study proposes a novel architecture to reconstruct large areas of naturally degraded murals, maintaining intrinsic details, avoiding color bias, and preserving artistic excellence. The architecture integrates generative adversarial networks (GANs) and the diffusion model, including a whole structure formation network (WSFN), a semantic color network (SCN), and a diffusion mixture distribution (DIMD) discriminator. The WSFN uses the original image, a line drawing, and an edge map to capture mural details, which are then texturally inpainted in the SCN using gated convolution for enhanced results. Special attention is given to globally extending the receptive field for large-area inpainting. The model is evaluated using custom-degraded mural images collected from Tamil Nadu temples. Quantitative analysis showed superior results than state-of-the-art methods, with SSIM, MSE, PSNR, and LPIPS values of 0.8853, 0.0021, 29.8826, and 0.0426, respectively.

Maize leaf area offers valuable insights into physiological processes, playing a critical role in breeding and guiding agricultural practices. The Azure Kinect DK possesses the real-time capability to capture and analyze the spatial structural features of crops. However, its further application in maize leaf area measurement is constrained by RGB–depth misalignment and limited sensitivity to detailed organ-level features. This study proposed a novel approach to address and optimize the limitations of the Azure Kinect DK through the multimodal coupling of RGB-D data for enhanced organ-level crop phenotyping. To correct RGB–depth misalignment, a unified recalibration method was developed to ensure accurate alignment between RGB and depth data. Furthermore, a semantic information-guided depth inpainting method was proposed, designed to repair void and flying pixels commonly observed in Azure Kinect DK outputs. The semantic information was extracted using a joint YOLOv11-SAM2 model, which utilizes supervised object recognition prompts and advanced visual large models to achieve precise RGB image semantic parsing with minimal manual input. An efficient pixel filter-based depth inpainting algorithm was then designed to inpaint void and flying pixels and restore consistent, high-confidence depth values within semantic regions. A validation of this approach through leaf area measurements in practical maize field applications—challenged by a limited workspace, constrained viewpoints, and environmental variability—demonstrated near-laboratory precision, achieving an MAPE of 6.549%, RMSE of 4.114 cm2, MAE of 2.980 cm2, and R2 of 0.976 across 60 maize leaf samples. By focusing processing efforts on the image level rather than directly on 3D point clouds, this approach markedly enhanced both efficiency and accuracy with the sufficient utilization of the Azure Kinect DK, making it a promising solution for high-throughput 3D crop phenotyping.

This paper presents a comprehensive study of the application of AI-driven inpainting techniques to the restoration of historical photographs of the Czech city Most, with a focus on restoration and reconstructing the lost architectural heritage. The project combines state-of-the-art methods, including generative adversarial networks (GANs), patch-based inpainting, and manual retouching, to restore and enhance severely degraded images. The reconstructed/restored photographs of the city Most offer an invaluable visual representation of a city that was largely destroyed for industrial purposes in the 20th century. Through a series of blind and informed user tests, we assess the subjective quality of the restored images and examine how knowledge of edited areas influences user perception. Additionally, this study addresses the technical challenges of inpainting, including computational demands, interpretability, and bias in AI models. Ethical considerations, particularly regarding historical authenticity and speculative reconstruction, are also discussed. The findings demonstrate that AI techniques can significantly contribute to the preservation of cultural heritage, but must be applied with careful oversight to maintain transparency and cultural integrity. Future work will focus on improving the interpretability and efficiency of these methods, while ensuring that reconstructions remain historically and culturally sensitive.

Visual place recognition is one of the core modern computer vision tasks concerned with identifying location based on the image taken there. Modern state-of-the-art approaches heavily rely on RGB images which are largely affected by changes in the same scene such as varying daytime, illumination, seasonal changes, and presence of dynamic objects (people, vehicles). This results into a large difference between the images in the training dataset and the ones taken by a person in real life at the same place as a part of some application, rendering modern approaches less effective. To deal with this problem, we propose a novel approach that uses only geometrical information (shapes of buildings, terrains, trees, and their relevant positions) obtained from depth and semantic maps inpainted to remove dynamic objects. In this paper, we study two versions of the pipeline: the first one uses direct inpainting, and the second utilizes synthetic data to improve the inpainting process. Our most efficient model achieved 60.6% correct answers with synthetic refinement. With direct inpainting, it kept metrics high at 51.1%. With these compelling results, our approach offers a novel and effective alternative to known algorithms, making it an exciting avenue for future research in visual place recognition.

The issue of missing radar data has long presented a challenge for meteorologists. Meteorological radars often suffer from data gaps and poor quality due to various factors, such as random partial beam blockage (PBB), instrument malfunctions, or irregular blind spots in detection locations. Therefore, it is essential that low-quality weather radar observations be remedied. Traditional deep learning methods are not well suited to this task, as they are incompatible with pixel-level remote sensing data and struggle to identify discrete features, leading to inaccurate repairs in areas of high reflectivity. In light of the aforementioned challenges, in this article, we propose a method for radar data restoration using a noise-guided diffusion probability model driven by known information. The aim of this approach is to effectively address missing and poor-quality radar data in various scenarios. We performed experiments in the central region of the United States using seamlessly blended scanning reflectivity radar data provided by the National Oceanic and Atmospheric Administration. We simulated two data gap types: random PBB and short radar outage periods. We tested our method on data from different years and compared its performance to those of other deep learning models under the same missing data conditions. The experimental results indicate that, compared with other methods, our method improves the inpainting accuracy index by approximately 30% and the inpainting quality index by approximately 10% on the missing inpainting of outage. Moreover, it shows excellent inpainting effects on beam blockage, highlighting its considerable potential to improve quantitative applications and various tasks under inclement weather conditions.

In the process of gradually upgrading aquaculture to the information and intelligence industries, it is usually necessary to collect images of underwater fish. In practical work, the quality of underwater images is often affected by water clarity and light refraction, resulting in most fish images not fully displaying the entire fish body. Image inpainting helps infer the occluded fish image information based on known images, thereby better identifying and analyzing fish populations. When using existing image inpainting methods for underwater fish images, limited by the small datasets available for training, the results were not satisfactory. Lightweight Multi-scales Feature Diffusion (LMF-Diffusion) is proposed to achieve results closer to real images when dealing with image inpainting tasks from small datasets. LMF-Diffusion is based on guided diffusion and flexibly extracts features from images at different scales, effectively capturing remote dependencies among pixels, and it is more lightweight, making it more suitable for practical deployment. Experimental results show that our architecture uses only 48.7% of the parameter of the guided diffusion model and produces inpainting results closer to real images in our dataset. Experimental results show that LMF-Diffusion enables the Repaint method to exhibit better performance in underwater fish image inpainting. Underwater fish image inpainting results obtained using our LMF-Diffusion model outperform those produced by current popular image inpainting methods.

In contemporary times, the preservation of scientific and creative endeavours often relies on the utilization of film and image archives, hence emphasizing the significance of image processing as a critical undertaking. Image inpainting refers to the process of digitally altering an image in a manner that renders the adjustments imperceptible to a viewer lacking knowledge of the original image. Image inpainting is a technique mostly employed to restore damaged regions within an image by utilizing information obtained from matching characteristics in relevant images. This process involves filling in the damaged areas and removing undesired objects. The SURF (Speeded Up Robust Feature) algorithm under consideration is partitioned into three primary phases. Firstly, the essential characteristics of the impaired image and the pertinent image are identified. In the second stage, the relationship between the damaged image and the relevant image is determined in terms of translation, scaling, and rotation. Ultimately, the destroyed area is reconstructed through the application of the inverse transformation. The quality assessment of inpainted images can be evaluated using metrics such as Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Mean Squared Error (MSE). The experimental findings provide evidence that the suggested inpainting technique is effective in terms of both speed and quality.

Computer vision has advanced much in recent years. Several tasks, such as image recognition, classification, or image restoration, are regularly solved with applications using artificial intelligence techniques. Image restoration comprises different use cases such as style transferring, improvement of quality resolution, or completing missing parts. The latter is also known as image inpainting, virtual image inpainting in this case, which consists of reconstructing missing regions or elements. This paper explores how to evaluate the performance of a deep learning method to do virtual image inpainting to reconstruct missing architectonical elements in images of ruined Greek temples to measure the performance of different activation functions. Unlike a previous study related to this work, a direct reconstruction process without segmented images was used. Then, two evaluation methods are presented: the objective one (mathematical metrics) and an expert (visual perception) evaluation to measure the performance of the different approaches. Results conclude that ReLU outperforms other activation functions, while Mish and Leaky ReLU perform poorly, and Swish’s professional evaluations highlight a gap between mathematical metrics and human visual perception.

Clouds and other data artefacts frequently limit the retrieval of key variables from remotely sensed Earth observations. We train a natural language processing (NLP)-inspired algorithm with high-fidelity ocean simulations to accurately reconstruct masked or missing data in sea surface temperature (SST) fields—one of 54 essential climate variables identified by the Global Climate Observing System. We demonstrate that the resulting model, referred to as Enki, repeatedly outperforms previously adopted inpainting techniques by up to an order of magnitude in reconstruction error, while displaying exceptional performance even in circumstances where the majority of pixels are masked. Furthermore, experiments on real infrared sensor data with masked percentages of at least 40% show reconstruction errors of less than the known uncertainty of this sensor (root mean square error (RMSE) ≲0.1 K). We attribute Enki’s success to the attentive nature of NLP combined with realistic SST model outputs—an approach that could be extended to other remotely sensed variables. This study demonstrates that systems built upon Enki—or other advanced systems like it—may therefore yield the optimal solution to mitigating masked pixels in in climate-critical ocean datasets sampling a rapidly changing Earth.

Digitized methodologies in the recent era contribute to various fields of automation that used to hold different interests and meanings of human life. Buildings with historical significance, cultural values, and beliefs are becoming an interdisciplinary field of interest, engaging more computer scientists nowadays. Such structures need more attention towards reconstructing their values using a flavor of computerized tools instead of brickwork directly. Due to the wear of time, the tiles and engravings of most of the historical monuments are on the verge of ruin, endangering significant historical values. In this survey, we rebuild the values by delving deep into the device and methodologies by providing a comprehensive understanding of emerging fields and some experimental decisions. We discuss heritage restoration from some essential papers on 3D reconstruction, image inpainting, IoT-based methods, genetic algorithms, and image processing. The survey explains Machine Learning, Deep Learning, and Computer Vision-based methods for various restoration tasks in the related field. We divide this into certain parts contributing to different fields that restore cultural heritage. Moreover, we infer that the techniques will be faster, cheaper, and more beneficial to the context of image reconstruction in the near future.

Computed tomography (CT) is usually accompanied by a long scanning time and substantial patient radiation exposure. Sinograms are the basis for constructing CT scans; however, continuous sinograms may highly overlap, resulting in extra radiation exposure. This paper proposes a deep learning model to inpaint a sparse-view sinogram sequence. Because a sinogram sequence around the human body is circular in nature, we propose a circular LSTM (CirLSTM) architecture that feeds position-relevant information to our model. To evaluate the performance of our proposed method, we compared the results of our inpainted sinograms with ground truth sinograms using evaluation metrics, including the peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM). The SSIM values for both our proposed method and the state-of-the-art method range from 0.998 to 0.999, indicating that the prediction of structures is not challenging for either method. Our proposed CirLSTM achieves PSNR values ranging from 49 to 52, outperforming all the other compared methods. These results demonstrate the feasibility of using only interleaved sinograms to construct a complete sinogram sequence and to generate high-quality CT images. Furthermore, we validated the proposed model across different body portions and CT machine models. The results show that CirLSTM outperforms all other methods in both the across-body segment validation and across-machine validation scenarios.

Abstract Background Positron emission tomography (PET) in combination with magnetic resonance imaging (MRI) could allow inflammatory complications near total knee arthroplasty (TKA) to be studied early in their development. However, attenuation of the PET signal by the metal TKA implants imparts substantial error into measurements of tracer activity, and conventional MR-based attenuation correction (AC) methods have large signal voids in the vicinity of metal implants. Purpose To evaluate a segmentation-based AC approach to measure tracer uptake from PET/MRI scans near TKA implants. Methods A TKA implant (Triathlon, Stryker, Mahwah, USA) was implanted into a cadaver. Four vials were filled with [18F]fluorodeoxyglucose with known activity concentration (4.68 MBq total, 0.76 MBq/mL) and inserted into the knee. Images of the knee were acquired using a 3T PET/MRI system (Biograph mMR, Siemens Healthcare, Erlangen, Germany). Models of the implant components were registered to the MR data using rigid-body transformations and the other tissue classes were manually segmented. These segments were used to create the segmentation-based map and complete the AC. Percentage error of the resulting measured activities was calculated by comparing the measured and known amounts of activity in each vial. Results The original AC resulted in a percentage error of 64.1% from the known total activity. Errors in the individual vial activities ranged from 40.2 to 82.7%. Using the new segmentation-based AC, the percentage error of the total activity decreased to 3.55%. Errors in the individual vials were less than 15%. Conclusions The segmentation-based AC technique dramatically reduced the error in activity measurements that result from PET signal attenuation by the metal TKA implant. This approach may be useful to enhance the reliability of PET/MRI measurements for numerous applications.

Visible watermarking offers direct copyright protection to the naked eye using visible pattern superimposed in the host image. However, due to their exposure, the watermark pattern becomes an easy target to remove by malicious entities that generates copyright-free images. Until now, in the scientific literature several semi-automatic visible watermark removal attacks have been proposed, showing enough visual quality of the recovered host image. In this paper, we propose a watermark design based on modified Seam Carving technique to provide robustness against removal attacks including inpainting-based attacks, Independent Component Analysis (ICA)-based attacks, and image matting-based attacks. Robustness against the attacks is measured using common image quality assessments and a watermark visibility assessment based on the Human Visual System (HVS). Experimental results show robustness of the proposed method against above mentioned watermark removal attacks under the most severe situation, in which adversary has the logotype used as a watermark pattern because it is public in Internet. The proposed method can be applied to any visible watermarking algorithms to make them robust against illegal removal attempts, without sacrificing their original functionality, such as reversibility for authorized users.

In this paper, we investigate the inpainting of normal maps that were captured from a lightstage. Occlusion of parts of the face during performance capture can be caused by the movement of, e.g., arms, hair, or props. Inpainting is the process of interpolating missing areas of an image with plausible data. We build on previous works about general image inpainting that use generative adversarial networks (GANs). We extend our previous work on normal map inpainting to use a U-Net structured generator network. Our method takes into account the nature of the normal map data and so requires modification of the loss function. We use a cosine loss rather than the more common mean squared error loss when training the generator. Due to the small amount of training data available, even when using synthetic datasets, we require significant augmentation, which also needs to take account of the particular nature of the input data. Image flipping and inplane rotations need to properly flip and rotate the normal vectors. During training, we monitor key performance metrics including the average loss, structural similarity index measure (SSIM), and peak signal-to-noise ratio (PSNR) of the generator, alongside the average loss and accuracy of the discriminator. Our analysis reveals that the proposed model generates high-quality, realistic inpainted normal maps, demonstrating the potential for application to performance capture. The results of this investigation provide a baseline on which future researchers can build with more advanced networks and comparison with inpainting of the source images used to generate the normal maps.

Remote sensing involves actions to obtain information about an area located on Earth. In the Amazon region, the presence of clouds is a common occurrence, and the visualization of important terrestrial information in the image, like vegetation and temperature, can be difficult. In order to estimate land surface temperature (LST) and the normalized difference vegetation index (NDVI) from satellite images with cloud coverage, the inpainting approach will be applied to remove clouds and restore the image of the removed region. This paper proposes the use of the neural network LaMa (large mask inpainting) and the scalable model named Big LaMa for the automatic reconstruction process in satellite images. Experiments are conducted on Landsat-8 satellite images of the Amazon rainforest in the state of Acre, Brazil. To evaluate the architecture’s accuracy, the RMSE (root mean squared error), SSIM (structural similarity index) and PSNR (peak signal-to-noise ratio) metrics were used. The LST and NDVI of the reconstructed image were calculated and compared qualitatively and quantitatively, using scatter plots and the chosen metrics, respectively. The experimental results show that the Big LaMa architecture performs more effectively and robustly in restoring images in terms of visual quality. And the LaMa network shows minimal superiority for the measured metrics when addressing medium marked areas. When comparing the results achieved in NDVI and LST of the reconstructed images with real cloud coverage, great visual results were obtained with Big LaMa.

Multi-focus image fusion aims at combining source information from differently focused images. Fusion of multi-focus images has great applications in machine vision. The paper focuses on removal of fence occlusions in multi-focus images. The proposed model extracts fence occlusion map using salient image features, and refined by morphological operators. Binary operators, and inpainting methods are used for fence removal and restoration. The proposed model estimates the fence area using statistical characteristics in focused regions. Similarly, binary filtering is used to perform thinning of enlarged areas for optimised restoration. The proposed model employs guided filtering for consistency verification. Fusion and restoration results are compared using several (reference and no-reference based) image quality metrics. Simulations show that proposed scheme achieves better results (visually and quantitatively) as compared to existing state-of-the-art techniques.

Local image descriptors play a crucial role in many image processing tasks, such as object tracking, object recognition, panorama stitching, and image retrieval. In this paper, we focus on learning local image descriptors in an unsupervised way, using autoencoders and variational autoencoders. We perform a thorough comparative analysis of these two approaches along with an in-depth analysis of the most relevant hyperparameters to guide their optimal selection. In addition to this analysis, we give insights into the difficulties and the importance of selecting right evaluation techniques during the unsupervised learning of the local image descriptors. We explore the extent to which a simple perceptual metric during training can predict the performance on tasks such as patch matching, retrieval and verification. Finally, we propose an improvement to the encoder architecture that yields significant savings in memory complexity, especially in single-image tasks. As a proof of concept, we integrate our descriptor into an inpainting algorithm and illustrate its results when applied to the virtual restoration of master paintings. The source code required to reproduce the presented results has been made available as a repository on GitHub (https://github.com/nimpy/local-img-descr-ae).

The cornea is an important refractive structure in the human eye. The corneal segmentation technique provides valuable information for clinical diagnoses, such as corneal thickness. Non-contact anterior segment optical coherence tomography (AS-OCT) is a prevalent ophthalmic imaging technique that can visualize the anterior and posterior surfaces of the cornea. Nonetheless, during the imaging process, saturation artifacts are commonly generated due to the tangent of the corneal surface at that point, which is normal to the incident light source. This stripe-shaped saturation artifact covers the corneal surface, causing blurring of the corneal edge, reducing the accuracy of corneal segmentation. To settle this matter, an inpainting method that introduces structural similarity and frequency loss is proposed to remove the saturation artifact in AS-OCT images. Specifically, the structural similarity loss reconstructs the corneal structure and restores corneal textural details. The frequency loss combines the spatial domain with the frequency domain to ensure the overall consistency of the image in both domains. Furthermore, the performance of the proposed method in corneal segmentation tasks is evaluated, and the results indicate a significant benefit for subsequent clinical analysis.

Conditional image generation plays a vital role in medical image analysis as it is effective in tasks such as super-resolution, denoising, and inpainting, among others. Diffusion models have been shown to perform at a state-of-the-art level in natural image generation, but they have not been thoroughly studied in medical image generation with specific conditions. Moreover, current medical image generation models have their own problems, limiting their usage in various medical image generation tasks. In this paper, we introduce the use of conditional Denoising Diffusion Probabilistic Models (cDDPMs) for medical image generation, which achieve state-of-the-art performance on several medical image generation tasks.

This paper presents an extensive evaluation of the Deep Image Prior (DIP) technique for image inpainting on Synthetic Aperture Radar (SAR) images. SAR images are gaining popularity in various applications, but there may be a need to conceal certain regions of them. Image inpainting provides a solution for this. However, not all inpainting techniques are designed to work on SAR images. Some are intended for use on photographs, while others have to be specifically trained on top of a huge set of images. In this work, we evaluate the performance of the DIP technique that is capable of addressing these challenges: it can adapt to the image under analysis including SAR imagery; it does not require any training. Our results demonstrate that the DIP method achieves great performance in terms of objective and semantic metrics. This indicates that the DIP method is a promising approach for inpainting SAR images, and can provide high-quality results that meet the requirements of various applications.

Melanoma and non-melanoma skin cancers have shown a rapidly increasing incidence rate, pointing to skin cancer as a major problem for public health. When analyzing these lesions in dermoscopic images, the hairs and their shadows on the skin may occlude relevant information about the lesion at the time of diagnosis, reducing the ability of automated classification and diagnosis systems. In this work, we present a new approach for the task of hair removal on dermoscopic images based on deep learning techniques. Our proposed model relies on an encoder-decoder architecture, with convolutional neural networks, for the detection and posterior restoration of hair's pixels from the images. Moreover, we introduce a new combined loss function in the network's training phase that combines the L1 distance, the total variation loss, and a loss function based on the structural similarity index metric. Currently, there are no datasets that contain the same images with and without hair, which is necessary to quantitatively evaluate our model. Thus, we simulate the presence of hair in hairless images extracted from publicly known datasets. We compare our results with six state-of-the-art algorithms based on traditional computer vision techniques by means of similarity measures that compare the reference hairless image and the one with simulated hair. Finally, the Wilcoxon signed-rank test is used to compare the methods. The results, both qualitatively and quantitatively, demonstrate the effectiveness of our model and how our loss function improves the restoration ability of the proposed model.

Ring artifacts are typical defects of computed tomography (CT) that degrade the quality of a 3D reconstructed image. Existing techniques for a ring reduction have various shortcomings and limitations, in particular, a lot of them are unable to process arbitrary fragments of the image and blur artifact-free regions. We propose an algorithm for ring artifacts segmentation and reduction by deep convolutional neural networks that correct 3D fragments of CT image by inpainting. We compare 2D and 3D architectures of networks. For creation a dataset with a big number of ring artifacts, we propose a procedure that is able to transfers an artifact from one image to an arbitrary place of another image. The appearance of the transferred artifact changes. For ring artifact segmentation and correction in images of sandstones and sand, the proposed networks demonstrate good visual results and outperform existing methods. The proposed technique concentrates on the Digital Rock workflow, but the networks can be adjusted for the processing of other CT images as well.

Developing an efficient short-term prediction framework for public transportation systems is of fundamental importance. This paper proposes a new image-processing-oriented methodology for the short-term prediction of train loads. First, we introduce a novel approach for representing the metro traffic by generating an image, exhibiting the spatial information of the trains running on a metro line while taking into account the irregular temporal sampling of the train loads. Second, we propose a prediction framework using deep learning methods. In particular, we build a U-net convolutional neural network, consisting of Inpainting and image-to-image translation mechanisms. We construct an image of the load predictions for different trains and stations. The framework performs a multi-step forecasting task for each station at any given time. The proposed prediction model is capable of making a global prediction for several departures on a whole metro line. Third, we benchmark our model against other prediction models using real load data collected over ten months on a Paris metro line. The comparison shows that the proposed framework is efficient compared to standard methods in image-processing prediction models. Finally, we evaluate the performance of the model in atypical operating situations (e.g., strike, incident). The results show that the performance of the model remains at acceptable levels of prediction errors in the event of metro traffic disruptions.

Saturation artifacts in optical coherence tomography (OCT) occur when received signal exceeds the dynamic range of spectrometer. Saturation artifact shows a streaking pattern and could impact the quality of OCT images, leading to inaccurate medical diagnosis. In this paper, we automatically localize saturation artifacts and propose an artifact correction method via inpainting. We adopt a dictionary-based sparse representation scheme for inpainting. Experimental results demonstrate that, in both case of synthetic artifacts and real artifacts, our method outperforms interpolation method and Euler's elastica method in both qualitative and quantitative results. The generic dictionary offers similar image quality when applied to tissue samples which are excluded from dictionary training. This method may have the potential to be widely used in a variety of OCT images for the localization and inpainting of the saturation artifacts.

Many modern computer vision systems include several modules that perform different processing operations packaged as a single pipeline architecture. This generally introduces a challenge in the evaluation process since most datasets provide evaluation data for just one of the operations. In this paper, we present an RGB-D dataset that was designed from first principles to cater for applications that involve salient object detection, segmentation, inpainting and blending techniques. This addresses a gap in the evaluation of image inpainting and blending applications that generally rely on subjective evaluation due to the lack of availability of comparative data. A set of experiments were carried out to demonstrate how the COTS dataset can be used to evaluate these different applications. This dataset includes a variety of scenes, where each scene is captured multiple times, each time adding a new object to the previous scene. This allows for a comparative analysis at pixel level in image inpainting and blending applications. Moreover, all objects were manually labeled in order to offer the possibility of salient object detection even in scenes that contain multiple objects. An online test with 1267 participants was also carried out, and this dataset also includes the click coordinates of users' selection for every image, introducing a user interaction dimension in the same RGB-D dataset. This dataset was also validated using state of the art techniques, obtaining an $F_\beta $ of 0.957 in salient object detection and a mean (Intersection over Union) IoU of 0.942 in Segmentation. Results demonstrate that the COTS dataset introduces novel possibilities for the evaluation of computer vision applications.

Ring artifacts are inevitable in microtomographic images. In Digital Rock workflow such defects might affect subsequent segmentation and flow simulation. We propose correction of ring artifacts in reconstructed microtomographic image by inpainting. Our blind inpainting method uses 3D convolutional network U-net. For the creation of training and validation datasets, we suggest an algorithm for transferring real ring artifacts to an arbitrary place of the undistorted slices of 8 big images of sandstones and sand. Parameters of the deep neural network and loss functions are analyzed. Loss function based on multi-scale structural similarity index (MS-SSIM) allows to achieve the best performance. Developed solution corrects ring artifacts perfectly from point of view visual assessment and outperforms existing inpainting methods according to quality metrics based on MS-SSIM and mean absolute error (MAE).

The RealSense camera frequently includes hole regions-unfilled regions where depth information are missing in the depth image. The holes cause serious problems in applications that use depth sensors and often straddle the background and object simultaneously. Accordingly, a hole must be filled by considering the boundary of the object. Fortunately, the object boundary inside the hole can be estimated by the RealSense color image. This paper proposes a hole-filling method in which filling converges to the object boundary from both border pixels. However, the color edge elements of the object boundary are often missing. Because the missing edge highly influences the hole-filling procedure, the boundary of the object close to the camera should be filled as completely as possible. Consequently, we introduce a simple method of filling the gap between two edge endpoints. Furthermore, we can address the situation in which there are one or more edge pixels between two border pixels. We repeatedly change the filling direction by alternating between the horizontal and vertical directions until no hole remains. Our experiment reveals that the proposed method outperforms other filling methods when numerically comparing the hole-filled depth image with the ground truth image for four methods: root-mean-square error (RMSE), peak signal-to-noise ratio (PSNR), structural similarity index map (SSIM), and correlation coefficient (CC). The proposed method also produces a superior visual when we observe the 3D stereo image pair constructed using an original color image and a left-view color image generated with depth-image-based rendering (DIBR).

Since the publication of the “Teoria del restauro” by Cesare Brandi in 1963, the question of retouching still raises theoretical debates among the stakeholders. Deliberately focusing on the practical context in which retouching is performed, and relaying on personal experience as a professional restorer and professor, here are some useful tools to best accomplish this phase of intervention: the setting up of the work space, the choice of light, the care of clothing, the implementation of procedures for the protection of health and personal and environmental safety.

Sparse demixing aims to separate signals that are sparse in some general dictionary, which has wide applications in signal and image processing, such as in super-resolution, image inpainting, robust sparse recovery, source separation, interference cancellation, saturation, and clipping restoration. For sparsity promotion in sparse demixing, the convex ℓ1 norm is of the most popular but it has a bias problem. In comparison, nonconvex regularization can mitigate the bias problem and can be expected to yield significantly better performance. In this paper, we employ the nonconvex ℓq-norm (0 ≤ q q-minimization formulation for the sparse demixing problem. Since the ℓq-minimization formulation is nonconvex and nonsmoothing, the standard alternative direction method of multipliers (ADMM) often fails to converge. To address this problem, we develop an iteratively reweighted ADMM algorithm which solves convex subproblems in each iteration and is convergent. Further, for the application of color image inpainting, we extend the new algorithm for multi-channel (RGB) joint recovery. The experimental results showed that the new algorithms can achieve significantly better performance than the ℓ1 algorithm.

Supply of missing data, also known as inpainting, is an important application of image processing.Wavelets are commonly used for inpainting algorithms. Shearlet transform which is an affinetransformation is the improvement of the wavelet transform. An asymptotic analysis may help to evaluatethe performance of an algorithm. In this article we compare the asymptotical analysis for wavelet andshearlet transforms in the case of inpainting where the missing data is shaped like a rectangle.

Machine learning models have been successfully applied for analysis of skin images. However, due to the black box nature of such deep learning models, it is difficult to understand their underlying reasoning. This prevents a human from validating whether the model is right for the right reasons. Spurious correlations and other biases in data can cause a model to base its predictions on such artefacts rather than on the true relevant information. These learned shortcuts can in turn cause incorrect performance estimates and can result in unexpected outcomes when the model is applied in clinical practice. This study presents a method to detect and quantify this shortcut learning in trained classifiers for skin cancer diagnosis, since it is known that dermoscopy images can contain artefacts. Specifically, we train a standard VGG16-based skin cancer classifier on the public ISIC dataset, for which colour calibration charts (elliptical, coloured patches) occur only in benign images and not in malignant ones. Our methodology artificially inserts those patches and uses inpainting to automatically remove patches from images to assess the changes in predictions. We find that our standard classifier partly bases its predictions of benign images on the presence of such a coloured patch. More importantly, by artificially inserting coloured patches into malignant images, we show that shortcut learning results in a significant increase in misdiagnoses, making the classifier unreliable when used in clinical practice. With our results, we, therefore, want to increase awareness of the risks of using black box machine learning models trained on potentially biased datasets. Finally, we present a model-agnostic method to neutralise shortcut learning by removing the bias in the training dataset by exchanging coloured patches with benign skin tissue using image inpainting and re-training the classifier on this de-biased dataset.

This paper proposes a novel approach to image inpainting that optimizes the shape of masked regions given by users. In image inpainting, which removes and restores unwanted regions in images, users draw masks to specify the regions. However, it is widely known that the users typically need to adjust the masked region by trial and error until they obtain the desired natural inpainting result, because inpainting quality is significantly affected by even a slight change in the mask. This manual masking takes a great deal of users' working time and requires considerable input. To reduce the human labor required, we propose a method for masked region optimization so that good inpainting results can be automatically obtained. To this end, our approach estimates “naturalness of inpainting” for all super pixels in inpainted images and reforms an original mask on a super-pixel basis, so that the naturalness of the inpainting result is improved. The efficacy of this approach does not depend on inpainting algorithms, thus it can be applied for every inpainting method as a plug-in. To demonstrate the effectiveness of our approach, we test our algorithm with varied images and show that it outperforms the existing inpainting methods without masked region reformation.

X-ray scattering images contain numerous gaps and defects arising from detector limitations and experimental configuration. We present a method to heal X-ray scattering images, filling gaps in the data and removing defects in a physically meaningful manner. Unlike generic inpainting methods, this method is closely tuned to the expected structure of reciprocal-space data. In particular, we exploit statistical tests and symmetry analysis to identify the structure of an image; we then copy, average and interpolate measured data into gaps in a way that respects the identified structure and symmetry. Importantly, the underlying analysis methods provide useful characterization of structures present in the image, including the identification of diffuse versus sharp features, anisotropy and symmetry. The presented method leverages known characteristics of reciprocal space, enabling physically reasonable reconstruction even with large image gaps. The method will correspondingly fail for images that violate these underlying assumptions. The method assumes point symmetry and is thus applicable to small-angle X-ray scattering (SAXS) data, but only to a subset of wide-angle data. Our method succeeds in filling gaps and healing defects in experimental images, including extending data beyond the original detector borders.

In an image processing field the digital image recovery is termed as inpainting. Efficient retrieval of an image, especially having large objects with high curvature and complex texture is an immensely challenging problem for image inpainting researchers and practitioner. This enthused researchers and emerge various inpainting algorithms and many are in progress. Generally inpainting techniques approaches the available area source of given image(s) to restore the unavailable area target by the information available at the target edge. This paper represents a novel approach BSDD (Blended Spatial and Dimensional Distances) by sampling patches at each pixel of the source region. From the given sample, selection of local edge patch is gradient based without priority computation overhead as previous techniques. These local patches are searched globally by linear distance in which both spatial and dimensional distances are considered with regularization factor. The main motive of this method consists in achieving the efficiency, curvature and textural challenges of inpainting without compromising the quality of inpainted image. We have tested the proposed method in real as well as synthetic images with high curvature and complex textures in all cases results are comparable with other well-known techniques. In view of quality and optical the proposed algorithm exhibits better results.

Mathematical methods of image inpainting often involve the discretization of a given continuous model. Typically, this is done by a pointwise discretization. We present a method that avoids this by modeling known variational approaches using a finite dimensional spline space. The way we build our algorithm we are able to prove that the basis of the spline space is stable. Further, due to the compact supports and structure of the basis the model involves a sparse system matrix allowing a fast and memory efficient implementation. Besides the analysis of the resulting model, we present a numerical implementation based on the alternating method of multipliers. We compare the results numerically with classical TV inpainting and give examples of applications, such as text removal, restoration, and noise removal.

The virtual inpainting of artworks provides a nondestructive mode of hypothesis visualization, and it is especially attractive when physical restoration raises too many methodological and ethical concerns. At the same time, in Cultural Heritage applications, the level of details in virtual reconstruction and their accuracy are crucial. We propose an inpainting algorithm that is based on generative adversarial network, with two generators: one for edges and another one for colors. The color generator rebalances chromatically the result by enforcing a loss in the discretized gamut space of the dataset. This way, our method follows the modus operandi of an artist: edges first, then color palette, and, at last, color tones. Moreover, we simulate the stochasticity of the lacunae in artworks with morphological variations of a random walk mask that recreate various degradations, including craquelure. We showcase the performance of our model on a dataset of digital images of wall paintings from the Dunhuang UNESCO heritage site. Our proposals of restored images are visually satisfactory and they are quantitatively comparable to state-of-the-art approaches.

Video inpainting or completion is a vital video improvement technique used to repair or edit digital videos. This paper describes a framework for temporally consistent video completion. The proposed method allows to remove dynamic objects or restore missing or tainted regions presented in a video sequence by utilizing spatial and temporal information from neighboring scenes. Masking algorithm is used for detection of scratches or damaged portions in video frames. The algorithm iteratively performs the following operations: achieve frame; update the scene model; update positions of moving objects; replace parts of the frame occupied by the objects marked for remove by using a background model. In this paper, we extend an image inpainting algorithm based texture and structure reconstruction by incorporating an improved strategy for video. Our algorithm is able to deal with a variety of challenging situations which naturally arise in video inpainting, such as the correct reconstruction of dynamic textures, multiple moving objects and moving background. Experimental comparisons to state-of-the-art video completion methods demonstrate the effectiveness of the proposed approach. It is shown that the proposed spatio-temporal image inpainting method allows restoring a missing blocks and removing a text from the scenes on videos.

This paper addresses the recovery and demixing problem of signals that are sparse in some general dictionary. Involved applications include source separation, image inpainting, super-resolution, and restoration of signals corrupted by clipping, saturation, impulsive noise, or narrowband interference. We employ the ℓq-norm (0 ≤ q q-minimization formulation for the recovery and demixing problem. This nonconvex formulation is approximately solved by two efficient first-order algorithms based on proximal coordinate descent and alternative direction method of multipliers (ADMM), respectively. The new algorithms are convergent in the nonconvex case under some mild conditions and scale well for high-dimensional problems. A convergence condition of the new ADMM algorithm has been derived. Furthermore, the extension of the two algorithms for multichannels joint recovery has been presented, which can further exploit the joint sparsity pattern among multichannel signals. Various numerical experiments showed that the new algorithms can achieve considerable performance gain over the ℓ1-regularized algorithms.

Underwater optical cameras are widely used for security monitoring in ocean, such as earthquake prediction and tsunami alarming. Optical cameras recognize objects for autonomous underwater vehicles and provide security protection for sea-floor networks. However, there are many issues for underwater optical imaging, such as forward and backward scattering, light absorption, and sea snow. Many underwater image processing techniques have been proposed to overcome these issues. Among these techniques, the depth map gives important information for many applications of the post-processing. In this paper, we propose a Kinect-based underwater depth map estimation method that uses a captured coarse depth map by Kinect with the loss of depth information. To overcome the drawbacks of low accuracy of coarse depth maps, we propose a corresponding reconstruction architecture that uses the underwater dual channels prior dehazing model, weighted enhanced image mode filtering, and inpainting. Our proposed method considers the influence of mud sediments in water and performs better than the traditional methods. The experimental results demonstrated that, after inpainting, dehazing, and interpolation, our proposed method can create high-accuracy depth maps.

Image inpainting, a technique of completing missing or corrupted image regions in undetected form, is an open problem in digital image processing. Inpainting of large regions using Deep Convolutional Generative Adversarial Nets (DCGAN) is a new and powerful approach. In described approaches the size of generated image and size of input image should be the same. In this paper we propose a new method where the size of input image with corrupted region can be up to 4 times larger than generated image.

In this paper we introduce a novel machine learning-based fully automatic approach for the semantic analysis and documentation of masonry wall images, performing in parallel automatic detection and virtual completion of occluded or damaged wall regions, and brick segmentation leading to an accurate model of the wall structure. For this purpose, we propose a four-stage algorithm which comprises three interacting deep neural networks and a watershed transform-based brick outline extraction step. At the beginning, a U-Net-based sub-network performs initial wall segmentation into brick, mortar and occluded regions, which is followed by a two-stage adversarial inpainting model. The first adversarial network predicts the schematic mortar-brick pattern of the occluded areas based on the observed wall structure, providing in itself valuable structural information for archeological and architectural applications. The second adversarial network predicts the pixels’ color values yielding a realistic visual experience for the observer. Finally, using the neural network outputs as markers in a watershed-based segmentation process, we generate the accurate contours of the individual bricks, both in the originally visible and in the artificially inpainted wall regions. Note that while the first three stages implement a sequential pipeline, they interact through dependencies of their loss functions admitting the consideration of hidden feature dependencies between the different network components. For training and testing the network a new dataset has been created, and an extensive qualitative and quantitative evaluation versus the state-of-the-art is given. The experiments confirmed that the proposed method outperforms the reference techniques both in terms of wall structure estimation and regarding the visual quality of the inpainting step, moreover it can be robustly used for various different masonry wall types.

Depth sensors are important in several fields to recognize real space. However, there are cases where most depth values in a depth image captured by a sensor are constrained because the depths of distal objects are not always captured. This often occurs when a low-cost depth sensor or structured-light depth sensor is used. This also occurs frequently in applications where depth sensors are used to replicate human vision, e.g., when using the sensors in head-mounted displays (HMDs). One ideal inpainting (repair or restoration) approach for depth images with large missing areas, such as partial foreground depths, is to inpaint only the foreground; however, conventional inpainting studies have attempted to inpaint entire images. Thus, under the assumption of an HMD-mounted depth sensor, we propose a method to inpaint partially and reconstruct an RGB-D depth image to preserve foreground shapes. The proposed method is comprised of a smoothing process for noise reduction, filling defects in the foreground area, and refining the filled depths. Experimental results demonstrate that the inpainted results produced using the proposed method preserve object shapes in the foreground area with accurate results of the inpainted area with respect to the real depth with the peak signal-to-noise ratio metric.

Three-dimensional (3D) reconstruction methods generate a 3D textured model from the combination of data from several captures. As such, the geometrical transformations between these captures are required. The process of computing or refining these transformations is referred to as alignment. It is often a difficult problem to handle, in particular due to a lack of accuracy in the matching of features. We propose an optimization framework that takes advantage of fiducial markers placed in the scene. Since these markers are robustly detected, the problem of incorrect matching of features is overcome. The proposed procedure is capable of enhancing the 3D models created using consumer level RGB-D hand-held cameras, reducing visual artefacts caused by misalignments. One problem inherent to this solution is that the scene is polluted by the markers. Therefore, a tool was developed to allow their removal from the texture of the scene. Results show that our optimization framework is able to significantly reduce alignment errors between captures, which results in visually appealing reconstructions. Furthermore, the markers used to enhance the alignment are seamlessly removed from the final model texture.

This paper proposes the use of spectral unmixing and sparse reconstruction methods to restore a simulated dataset for the Environmental Mapping and Analysis Program (EnMAP), the forthcoming German spaceborne hyperspectral mission. The described method independently decomposes each image element into a set of representative spectra, which come directly from the image and have previously undergone a low-pass filtering in noisy bands. The residual vector from the unmixing process is considered as mostly composed of noise and ignored in the reconstruction process. The first assessment of the results is encouraging, as the original bands taken into account are reconstructed with a high signal-to-noise ratio and low overall distortions. Furthermore, the same method could be applied for the inpainting of dead pixels, which could affect EnMAP data, especially at the end of the satellite’s life cycle.

An image inpainting scheme has been proposed that utilizes the spatial contextual information approach for image completion. The domain to be inpainted is smooth for texture images. It can be inpainted using exemplar and variational methods. In proposed method, the regions to be removed from the image are segmented and pull out of the image where it is tenure as hole. As the hole in the image to be inpainted is an unsupervised approach, we are computing the pixel in the holes using spatial contextual correlations. The method’s efficacy is embodied by real images.

Depth image-based rendering (DIBR) plays an important role in 3D video and free viewpoint video synthesis. However, artifacts might occur in the synthesized view due to viewpoint changes and stereo depth estimation errors. Holes are usually out-of-field regions and disocclusions, and filling them appropriately becomes a challenge. In this paper, a virtual view synthesis approach based on asymmetric bidirectional DIBR is proposed. A depth image preprocessing method is applied to detect and correct unreliable depth values around the foreground edges. For the primary view, all pixels are warped to the virtual view by the modified DIBR method. For the auxiliary view, only the selected regions are warped, which contain the contents that are not visible in the primary view. This approach reduces the computational cost and prevents irrelevant foreground pixels from being warped to the holes. During the merging process, a color correction approach is introduced to make the result appear more natural. In addition, a depth-guided inpainting method is proposed to handle the remaining holes in the merged image. Experimental results show that, compared with bidirectional DIBR, the proposed rendering method can reduce about 37% rendering time and achieve 97% hole reduction. In terms of visual quality and objective evaluation, our approach performs better than the previous methods.

In this article, the covariance of wavelet transform coefficients is used to obtain a texture inpainted image. The covariance is obtained by the maximum likelihood estimate. The integration of wavelet decomposition and maximum likelihood estimate of a group of pixels (texture) captures the best-fitting texture used to fill in the inpainting area. The image is decomposed into four wavelet coefficient images by using wavelet transform. These wavelet coefficient images are divided into small square patches. The damaged region in each coefficient image is filled by searching a similar square patch around the damaged region in that particular wavelet coefficient image by using covariance.

A speedy recovery of damaged digitized photographs based on orientation driven multi structure morphology is proposed. The recovery order plays an important factor for human visualization and hence it is guided by the orientation of edges at the surrounding known regions of the missing domain. The image is edge detected by thresholding the image gradient along the eight possible orientations. These eight edge images are represented as eight edge planes. The edge-plane-sliced information is used twice manifold for reconstructing the regions within the missing part, as well as for guiding the integration that follows. The damaged regions are morphologically eroded using the structuring elements of corresponding orientations dictated by the edge-planes. The resultant filled image is obtained using local isotopic driven integration. The novelty of our approach is to explicitly specify the direction of filling herby ensuring ease in convergence in different orientations and then streamlining the process to guarantee complete and natural look. By implementing region-filling through morphological erosion, several pixels instead of one can be restored at every inpainting step, making the method faster than many traditional texture synthesis inpainting algorithms and successfully recovers images with better Peak Signal to Noise ratios even for massive damages.