Search

Over the recent years, a number of deep learning approaches are successfully introduced to tackle the problem of image in-painting for achieving better perceptual effects. However, there still exist obvious hole-edge artifacts in these deep learning-based approaches, which need to be rectified before they become useful for practical applications. In this article, we propose an iteration-driven in-painting approach, which combines the deep context model with the backpropagation mechanism to fine-tune the learning-based in-painting process and hence, achieves further improvement over the existing state of the arts. Our iterative approach fine tunes the image generated by a pretrained deep context model via backpropagation using a weighted context loss. Extensive experiments on public available test sets, including the CelebA, Paris Streets, and PASCAL VOC 2012 dataset, show that our proposed method achieves better visual perceptual quality in terms of hole-edge artifacts compared with the state-of-the-art in-painting methods using various context models.

Recent developments of image super-resolution often utilize the deep convolutional neural network (CNN) and residual learning to relate the observed low-resolution pixels and unknown high-resolution pixels. However, image interpolation assumes that the observed image was directly down-sampled without low-pass filtering, such that the aliased down-sampled low-resolution image exhibits jags and chaos that cannot be easily modeled by conventional residual learning in super-resolution. In this paper, we propose a new framework to exploit the residual dense network using hierarchical levels of recursive residual learning and densely connected convolutional layers for image interpolation. The proposed deep recursive network iteratively reconstructs hierarchical levels of image details for aliased and discontinuous residual of interpolated pixels. Experimental results on popular Set16, Set18, and Urban12 image datasets show that the proposed method outperforms state-of-the-art image interpolation methods using local and nonlocal autoregressive models, random forests and deep CNN, in terms of PSNR (0.27–1.57 dB gain), SSIM and subjective evaluations. More importantly, model parameters of the proposed method are significantly less than that of existing deep CNN for image interpolation.

In general, existing research on single image super-resolution does not consider the practical application that, when image transmission is over noisy channels, the effect of any possible geometric transformations could incur significant quality loss and distortions. To address this problem, we present a new and robust super-resolution method in this paper, where a robust spatially-transformed deep learning framework is established to simultaneously perform both the geometric transformation and the single image super-resolution. The proposed seamlessly integrates deep residual learning based spatial transform module with a very deep super-resolution module to achieve a robust and improved single image super-resolution. In comparison with the existing state of the arts, our proposed robust single image super-resolution has a number of novel features, including 1) content-characterized deep features are extracted out of the input LR images to identify the incurred geometric transformations, and hence transformation parameters can be optimized to influence and control the super-resolution process; 2) the effects of any geometric transformations can be automatically corrected at the output without compromise on the quality of final super-resolved images; and 3) compared with the existing research reported in the literature, our proposed achieves the advantage that HR images can be recovered from those down-sampled LR images corrupted by a number of different geometric transformations. The extensive experiments, measured by both the peak-signal-to-noise-ratio and the similar structure index measurement, show that our proposed method achieves a high level of robustness against a number of geometric transformations, including scaling, translations, and rotations. Benchmarked by the existing state-of-the-arts SR methods, our proposed delivers superior performances on a wide range of datasets which are publicly available and widely adopted across relevant research communities.

For many practical applications, it is essential to address both geometric corrections and missing information reconstruction of face images and natural images. However, it is unfavorable to separate the problem into two sub-tasks due to error accumulations of sequential tasks. In this paper, we propose a novel robust missing information reconstruction framework via deep feature transformations to simultaneously address both geometric corrections and image completion. Specifically, our proposed framework realizes multiple channel spatial transformations to tackle geometric corrections, and address image completion through non-linear features projections. The flow of our framework includes deep feature extraction, feature enhancement, feature projection, and feature refinement, where deep features are extracted and learnt to achieve robust image completion. Experimental results show the superior performance of our framework for both face images and natural images in various databases. Compared with the conventional approaches approach to split the problem into two sub-tasks, including image inpainting and spatial transformation, our proposed framework achieves a number of advantages, including i) an unified framework to automatically correct the geometric distortions and to reconstruct the missing information simultaneously and ii) achieving much better visual quality for those recovered images.

Recently, there have been significant advances in image super-resolution based on generative adversarial networks (GANs) to achieve breakthroughs in generating more images with high subjective quality. However, there are remaining challenges needs to be met, such as simultaneously recovering the finer texture details for large upscaling factors and mitigating the geometric transformation effects. In this paper, we propose a novel robust super-resolution GAN (i.e. namely RSR-GAN) which can simultaneously perform both the geometric transformation and recovering the finer texture details. Specifically, since the performance of the generator depends on the discreminator, we propose a novel discriminator design by incorporating the spatial transformer module with residual learning to improve the discrimination of fake and true images through removing the geometric noise, in order to enhance the super-resolution of geometric corrected images. Finally, to further improve the perceptual quality, we introduce an additional DCT loss term into the existing loss function. Extensive experiments, measured by both PSNR and SSIM measurements, show that our proposed method achieves a high level of robustness against a number of geometric transformations, including rotation, translation, a combination of rotation and scaling effects, and a cobmination of rotaion, transalation and scaling effects. Benchmarked by the existing state-of-the-arts SR methods, our proposed delivers superior performances on a wide range of datasets which are publicly available and widely adopted across research communities.

Video super-resolution aims to increase the resolution of videos by exploiting the intra-frame and inter-frame dependencies of the low-resolution video sequences. There are usually two dependent steps in the video super-resolution: the motion compensation and the super-resolution reconstruction. In this paper, we propose a new deep learning framework without the explicit motion estimation by utilizing the self-attention model to exploit the full receptive field of the input video frames. In other words, the proposed deep neural network extracts the local features at all spatial-temporal locations for combining into global features using the self-attention networks in order to reconstruct the high-resolution video frame. The proposed global-aware network outperforms the state-of-the-art deep learning-based image and video super-resolution algorithms in terms of subjective and objective quality with less computational operations, as verified by extensive experiments on public image and video datasets, including Set5, Set14, B100, Urban100, and Vid4.

Recent researches show that the high-frequency discrete cosine transform (DCT) coefficients can be estimated from low-frequency DCT coefficients by exploiting the spatial correlations. Hence, images coded by DCT such as JPEG/MJPEG/H.264, etc., can be down-sampled in DCT domain, where the high-frequency information can be accurately restored through image up-sampling. In this letter, we propose a novel deep neural network using the cascaded fully connected layers and convolution layers in dual domains (DCT and spatial domains), in order to restore high-frequency DCT coefficients from observed low-frequency DCT coefficients by exploiting the DCT inter-block and spatial correlations. In the proposed network, many recent techniques are adopted, including residual network in dual domains, batch normalization, denseNet, etc. Experimental results show that the proposed cascaded networks in dual domains significantly outperforms the state-of-the-art DCT up-sampling methods in terms of PSNR (0.63–2.57 dB gain), SSIM values, and subjective evaluations on standard image datasets Set5 and Set14.

Recent developments of virtual reality applications have accelerated the usage of cameras with wide-angle and telephoto/macro lens, which produce nonlinear radial lens distortions, such as barrel distortion and pincushion distortion. However, due to many reasons, the resolution of images with nonlinear lens distortions is often limited. In this paper, we address the image super-resolution (SR) for images with nonlinear lens distortions through the deep convolutional neutral network with residual learning, which can significantly improve the image quality before and after the camera calibration. The proposed deep learning network was trained using hundreds of simulated images and tested on real cameras with fisheye and macro lens. Experimental results show that the proposed image SR method outperforms state-of-the-art SR methods for various degrees of radial-based barrel and pincushion distortions.

In this paper, we propose two approaches to generate a high resolution (HR) image from a low resolution (LR) version. It is commonly referred as single-image super-resolution(SISR). Our approaches are inspired by the spatial transformer (ST) module and Very Deep Convolutional Network (VDSR). The spatial transformer module is the neural network originally used for geometric transformations of images, while the VDSR is for image super-resolution. In the first approach, we propose to add the ST module with the VDSR to generate HR images. The use of the spatial transform with VDSR makes the network more robust to the geometric transformations. We propose the second approach to replace the convolutional neural network (CNN) used in the ST module with VDSR network. The replacement of CNN leads to improvements in the performance of ST module. The simulation results confirm that the feasibility of combing ST module and VDSR for super-resolution reconstruction, where the performance of the combination of ST module and VDSR is comparable to the VDSR alone in terms of Peak signal-to-noise ratio (PSNR) and structural Similarity Index Measurement (SSIM). Hence, the revised spatial transformer network can be used in the future for simultaneous geometric transformation and image super-resolution, which solve the practical applications of image super-resolution in real life.

Learning-based image interpolation using precise and robust k-NN searching for an accurate AR modeling.Robustness to insufficient k-NN matches and adaptation to relevant k-NN matches during online searching.Online coherent soft-decision estimation of both local AR parameters and high-resolution pixels.Highly competitive performance compared with the state-of-the-art approaches in terms of PSNR and SSIM. Image interpolation is to convert a low-resolution (LR) image into a high-resolution (HR) image through mathematical modeling. An accurate model usually leads to a better reconstruction quality, and the autoregressive (AR) model is a widely adopted model for image interpolation. Although a large amount of works have been done on AR models for image interpolation, there are plenty of rooms for improvements. In this work, we propose a robust and precise k-nearest neighbors (k-NN) searching scheme to form an accurate AR model of the local statistic. We make use of both LR and HR information obtained from a large amount of training data, in order to form a coherent soft-decision estimation of both AR parameters and high-resolution pixels. Experimental results show that the proposed learning-based AR interpolation algorithm has a very competitive performance compared with the state-of-the-art image interpolation algorithms in terms of PSNR and SSIM values.

Down-sampling in the discrete cosine transform (DCT) domain is preferable for images coded by DCT transform, such as JPEG/MJPEG/H.264, etc. Recent researches show that the truncated high-frequency DCT coefficients during the DCT down-sampling process can be estimated by learning the correlations between low-frequency and high-frequency DCT coefficients. In this paper, we propose to utilize the powerful super-resolution framework using sparse dictionaries with anchored neighborhood regression to significantly improve the accuracy of the estimated high-frequency DCT coefficients. Experimental results show that the proposed framework outperforms the state-of-the-art DCT-based up-sampling methods in terms of PSNR (0.3–1.63dB) and SSIM values for standard image datasets Set5 and Set14, while the computational time of the proposed method is 23× times faster than the state-of-the-art learning-based method using k-NN MMSE due to pre-computation of the ridge regression during the training process.

Camera lens distortions are widely observed in different applications for achieving specific optical effects, such as wide angle captures. Moreover, the image with lens distortion is often limited in resolution due to the cost of camera, limited bandwidth, etc. In this paper, we present a learning-based image super-resolution method for improving the resolution of images captured by cameras with barrel lens distortions. The key to the significant improvement of the resolution loss due to lens distortions is to learn a sparse dictionary with a post-processing step. During the training stage, the training images are used to learn the sparse dictionary and projection matrixes. During the testing stage, the observed low-resolution image uses the projection matrixes for two step super-resolution reconstructions of the final high-resolution image. Experimental results show that the proposed method outperforms the conventional learning-based super-resolution methods in terms of PSNR and SSIM values using the same set of training images for algorithm trainings.

In the camera manufacturing, there exist dusts, fingerprints, and water spots on the image sensor and lens. Hence, the resultant effects of darker region is called blemish, which causes a significant reduction in camera quality. The shapes of blemishes are diverse and irregular. Traditional method detects blemishes using image median filtering in a single direction which leads to false alarm and mis-detection for images with high level of noises. Thus, we present a novel filtering method for blemish detection, which utilizes four directional filters in the 0, 45, 90, 135 degrees directions. Compared to the conventional single direction filter, the multidirectional filters take into account more spatial information to more accurately detect blemishes for both weak and strong blemishes. Moreover, the proposed method uses a new adaptive threshold to better accommodate different image noise levels automatically. Experimental results on two batches of production samples (600 images) show the effectiveness of the proposed method over the conventional method.

As deep learning neural networks (DNNs) advance and increase in computational complexity, particularly in terms of memory cost, it becomes difficult to implement DNNs in fixed-point memory-sparse environments (e.g. integrated circuits in consumer electronics). Thus, the training of DNNs must be reformulated to balance the hardware costs needed to represent the often millions of parameter weights in such machine learning models. This paper proposes a novel optimization approach that simultaneously minimizes complexity (total memory) and maximizes accuracy. Specifically, a bit-depth complexity penalty is induced to urge the DNN model towards a state of lower memory using a numerical gradient in optimization iterations. Experimental results on the MNIST (Mixed National Institute of Standards and Technology) handwritten digit classification benchmark demonstrate a minimal loss (1%) in DNN accuracy with a significant reduction (37%) of memory cost on average.

DCT-Wiener image up-sampling scheme is highly desirable since it makes use of the advantages of the information in both spatial and DCT domains. The idea is to combine the observed low-frequency DCT coefficients with the estimated high-frequency DCT coefficients obtained by the Wiener filters in the spatial domain. However, the available 1-D and 2-D Wiener filters that were proposed for high-frequency DCT coefficients estimation are block non-adaptive, mainly due to the limited information from the observed image. In this paper, we propose a block-adaptive Wiener filter by utilizing the information from external training data. During the online estimation, for each image block, the k-nearest relevant DCT LR-HR block pairs are searched from the training data, in order to estimate the coefficients of the Wiener filter. Experimental results show that the proposed block-adaptive Wiener filter improves the PSNR value of the DCT-Wiener scheme by 1.5 dB compared with that using non-adaptive 1-D Wiener filter.

The hybrid DCT-Wiener-based (DCT-WB) interpolation scheme provides a powerful framework to interpolate an image by utilizing the information in both spatial and DCT domain. In this paper, we investigate the bottleneck of this hybrid scheme and propose a 2D non-separable block-based Wiener filter for the hybrid scheme. The Wiener filter is learnt using training image pairs through the minimum mean squares error estimation. The proposed Wiener filter resolves the quarter-pixel shift issue and provides much better performance over the original 1D 6-tap pixel-based Wiener filter. Experimental results show that incorporating the proposed Wiener filter into the hybrid scheme improves the PSNR (0.44 dB), SSIM and subjective quality for our extensive experimental work on testing images with various contents.

In novel view synthesis using the depth image based rendering, there exist some unknown pixel intensities (holes) due to the unexposed area. In this paper, we propose a depth-assisted nonlocal means algorithm to fill the holes using the information in the current frame and other frames in the synthesized video. The nonlocal means has been successfully applied to the video super-resolution applications. In hole filling, the challenge is the irregular interpolation. However, there can be a relatively reliable depth map, such that we are able combine the irregular intensity information and the reliable depth information, and make use of the formulation of nonlocal means to fill the holes. Experimental results show that the proposed algorithm outperforms the conventional spatial and temporal approaches in terms of visual quality.

In this paper, we propose an iterative Wiener filter which can simultaneously perform interpolation and restoration by using non-local means to directly model the correlation between the desired high-resolution image and observed low-resolution image. A novel mechanism is proposed to control the decay speed of the correlation function while iteratively updating both estimated correlation and high-resolution image. During the iterations, the image is decomposed into patches with similar intensities at initial iterations and the patches are connected naturally with good convergence. Experimental results show that the proposed algorithm is able to produce natural image structures, and provides better PSNR and visual quality than the state-of-the-art algorithms using the sparse representation and natural image priors.

In this paper, we propose a novel image interpolation technique using the small-kernel bilateral filter. The range distance of the bilateral filter is estimated using a novel maximum a posterior (MAP) estimation, in order to consider both the diagonal and vertical-horizontal correlations in image. The proposed method requires a few arithmetic operations, such that it is suitable for real time image zoom, video up-scaling and video transcoding applications in HDTV systems. Subjective evaluation shows that the proposed approach outperforms the conventional method in terms of edge sharpness and edge smoothness.

Recently, the probabilistic motion field was proposed for super-resolution reconstruction (SRR). In the interpolation step of SRR, a missing pixel can be estimated by the weighted average of neighboring pixels, which are weighted by the errors with the missing pixel. However, the errors are far from true values due to the approximated missing pixel in calculating the errors. Hence, in this paper, we propose a linear motion model to better approximate the errors, which results in a better interpolation quality. Experimental results show that a gain of 0.6 dB in PSNR is achievable using this linear motion model, and only a small number of neighboring pixels have to be used for fast interpolation/upsampling.

A typical dynamic reconstruction-based super-resolution video involves three independent processes: registration, fusion and restoration. Fast video super-resolution systems apply translational motion compensation model for registration with low computational cost. Traditional motion compensation model assumes that the whole spectrum of pixels is consistent between frames. In reality, the low frequency component of pixels often varies significantly. We propose a translational motion compensation model via frequency classification for video super-resolution systems. A novel idea to implement motion compensation by combining the up-sampled current frame and the high frequency part of the previous frame through the SAD framework is presented. Experimental results show that the new motion compensation model via frequency classification has an advantage of 2dB gain on average over that of the traditional motion compensation model. The SR quality has 0.25dB gain on average after the fusion process which is to minimize error by making use of the new motion compensated frame.