Search

In general, existing research on single image super-resolution does not consider the practical application that, when image transmission is over noisy channels, the effect of any possible geometric transformations could incur significant quality loss and distortions. To address this problem, we present a new and robust super-resolution method in this paper, where a robust spatially-transformed deep learning framework is established to simultaneously perform both the geometric transformation and the single image super-resolution. The proposed seamlessly integrates deep residual learning based spatial transform module with a very deep super-resolution module to achieve a robust and improved single image super-resolution. In comparison with the existing state of the arts, our proposed robust single image super-resolution has a number of novel features, including 1) content-characterized deep features are extracted out of the input LR images to identify the incurred geometric transformations, and hence transformation parameters can be optimized to influence and control the super-resolution process; 2) the effects of any geometric transformations can be automatically corrected at the output without compromise on the quality of final super-resolved images; and 3) compared with the existing research reported in the literature, our proposed achieves the advantage that HR images can be recovered from those down-sampled LR images corrupted by a number of different geometric transformations. The extensive experiments, measured by both the peak-signal-to-noise-ratio and the similar structure index measurement, show that our proposed method achieves a high level of robustness against a number of geometric transformations, including scaling, translations, and rotations. Benchmarked by the existing state-of-the-arts SR methods, our proposed delivers superior performances on a wide range of datasets which are publicly available and widely adopted across relevant research communities.

Video super-resolution aims to increase the resolution of videos by exploiting the intra-frame and inter-frame dependencies of the low-resolution video sequences. There are usually two dependent steps in the video super-resolution: the motion compensation and the super-resolution reconstruction. In this paper, we propose a new deep learning framework without the explicit motion estimation by utilizing the self-attention model to exploit the full receptive field of the input video frames. In other words, the proposed deep neural network extracts the local features at all spatial-temporal locations for combining into global features using the self-attention networks in order to reconstruct the high-resolution video frame. The proposed global-aware network outperforms the state-of-the-art deep learning-based image and video super-resolution algorithms in terms of subjective and objective quality with less computational operations, as verified by extensive experiments on public image and video datasets, including Set5, Set14, B100, Urban100, and Vid4.

In recent years, deep convolutional neural networks (CNNs) have been widely used for image super-resolution (SR) to achieve a range of sophisticated performances. Despite the significant advancement made in CNNs, it is still difficult to apply CNNs to practical SR applications due to enormous computations of deep convolutions. In this paper, we propose two lightweight deep neural networks using depthwise separable convolution for the real-time image SR. Specifically, depthwise separable convolution divides the standard convolution into depthwise convolution and pointwise convolution to significantly reduce the number of model parameters and multiplication operations. Moreover, recursive learning is adopted to increase the depth and receptive field of the network in order to improve the SR quality without increasing the model parameters. Finally, we propose a novel technique called Super-Sampling (SS) to learn more abundant high-resolution information by over-sampling the output image followed by adaptive down-sampling. The proposed two models, named SSNet-M and SSNet, outperform the existing state-of-the-art real-time image SR networks, including SRCNN, FSRCNN, ESPCN, and VDSR, in terms of model complexity, and subjective and PSNR/SSIM evaluations on Set5, Set14, B100, Urban100, and Manga109.

Recently, there have been significant advances in image super-resolution based on generative adversarial networks (GANs) to achieve breakthroughs in generating more images with high subjective quality. However, there are remaining challenges needs to be met, such as simultaneously recovering the finer texture details for large upscaling factors and mitigating the geometric transformation effects. In this paper, we propose a novel robust super-resolution GAN (i.e. namely RSR-GAN) which can simultaneously perform both the geometric transformation and recovering the finer texture details. Specifically, since the performance of the generator depends on the discreminator, we propose a novel discriminator design by incorporating the spatial transformer module with residual learning to improve the discrimination of fake and true images through removing the geometric noise, in order to enhance the super-resolution of geometric corrected images. Finally, to further improve the perceptual quality, we introduce an additional DCT loss term into the existing loss function. Extensive experiments, measured by both PSNR and SSIM measurements, show that our proposed method achieves a high level of robustness against a number of geometric transformations, including rotation, translation, a combination of rotation and scaling effects, and a cobmination of rotaion, transalation and scaling effects. Benchmarked by the existing state-of-the-arts SR methods, our proposed delivers superior performances on a wide range of datasets which are publicly available and widely adopted across research communities.

For many practical applications, it is essential to address both geometric corrections and missing information reconstruction of face images and natural images. However, it is unfavorable to separate the problem into two sub-tasks due to error accumulations of sequential tasks. In this paper, we propose a novel robust missing information reconstruction framework via deep feature transformations to simultaneously address both geometric corrections and image completion. Specifically, our proposed framework realizes multiple channel spatial transformations to tackle geometric corrections, and address image completion through non-linear features projections. The flow of our framework includes deep feature extraction, feature enhancement, feature projection, and feature refinement, where deep features are extracted and learnt to achieve robust image completion. Experimental results show the superior performance of our framework for both face images and natural images in various databases. Compared with the conventional approaches approach to split the problem into two sub-tasks, including image inpainting and spatial transformation, our proposed framework achieves a number of advantages, including i) an unified framework to automatically correct the geometric distortions and to reconstruct the missing information simultaneously and ii) achieving much better visual quality for those recovered images.