Search

With the help of convolutional neural networks (CNN), the single image super-resolution problem has been widely studied. Most of these CNN based methods focus on learning a model to map a low-resolution (LR) image to a high-resolution (HR) image, where the LR image is downsampled from the HR image with a known model. However, in a more general case when the process of the down-sampling is unknown and the LR input is degraded by noises and blurring, it is difficult to acquire the LR and HR image pairs for traditional supervised learning. Inspired by the recent unsupervised image-style translation applications using unpaired data, we propose a multiple Cycle-in-Cycle network structure to deal with the more general case using multiple generative adversarial networks (GAN) as the basis components. The first network cycle aims at mapping the noisy and blurry LR input to a noise-free LR space, then a new cycle with a well-trained $\times 2$ network model is orderly introduced to super-resolve the intermediate output of the former cycle. The number of total cycles depends on the different up-sampling factors ($\times 2$ , $\times 4$ , $\times 8$). Finally, all modules are trained in an end-to-end manner to get the desired HR output. Quantitative indexes and qualitative results show that our proposed method achieves comparable performance with the state-of-the-art supervised models. [ABSTRACT FROM AUTHOR]

Frame rate up-conversion (FRUC) usually suffers from unreliable motion vectors due to the absence of the current frame to be interpolated. In addition, since the majority of video sequences are usually compressed by various coding standards to reduce the data volume, the quality of the generated frames in the FRUC will be further impaired. To address this problem, we proposed two FRUC algorithms based on deep residual network. We first present a deep residual network for the FRUC (DRNFRUC), which consists of feature extraction, feature recursive analysis, and image restoration parts with a skip connection between the input and the output of the network. The proposed DRNFRUC takes the result of an arbitrary existing FRUC method as the input and is able to significantly reduce the edge blurring and blocking artifacts when the motion of the block is violent. In addition, we proposed a deep residual network with weighted convolutional motion compensation (DRNWCMC) for the FRUC, where the convolution operations can be embedded into the motion compensation interpolation (MCI) in any existing MCI-based FRUC method. In DRNWCMC, we first devise two convolutional neural networks corresponding to the forward and backward motion compensated frames, respectively. And then, the adaptive interpolation coefficients for motion compensation are designed as two $1\times1$ convolutional kernels. Finally, the interpolation result of WCMC is fed into another convolutional neural network to further improve the performance. All the parameters involved in the DRNWCMC are trained simultaneously under the same cost function. The experimental results show that the two proposed algorithms can remarkably improve both the objective and subjective quality of the interpolated frames. [ABSTRACT FROM AUTHOR]

Motion-compensated frame interpolation (MCFI) usually finds the most matched blocks by minimizing pixel intensity discrepancies between neighboring frames along the motion trajectory. However, the quality of interpolated frames is susceptible to inaccurate motion vectors possibly for regions with complex texture patterns, irregularly shaped objects, repeated patterns, motion blurring or aliasing, and so on. It is believed that pixel intensity across adjacent frames varies gradually and smoothly, and therefore can be modeled mathematically by a continuous and differentiable function. Thus, the pixel intensity within one frame can be expressed as either a forward polynomial approximation (FWPA) or a backward polynomial approximation (BWPA) by the Taylor expansion in this paper. The discrepancy between the FWPA and BWPA is employed to find the best motion vector. In addition, a motion-aligned partial derivative is proposed to calculate the Taylor expansion along the motion trajectory. The proposed method is applicable to any existing MCFI schemes and achieve superior performance by consuming relatively more buffer memories and computational resources. Extensive experimentation with comparison with previous techniques validates our method in terms of both objective and subjective criteria. [ABSTRACT FROM PUBLISHER]

Recent development of depth acquiring technique has accelerated the progress of 3D video in the market. Utilizing the acquired depth, arbitrary view frames can be generated based on depth image based rendering (DIBR) technique in free viewpoint video system. Different from texture video, depth sequence is mainly utilized for virtual view generation rather than viewing. Inspired by this, a depth frame interpolation scheme using texture information is proposed in this paper. The proposed scheme consists of a texture aided motion estimation (TAME) and texture aided motion compensation (TAMC) to fully explore the correlation between depth and the accompanying textures. The optimal motion vectors in TAME and the best interpolation weights in TAMC are respectively selected taking the geometric mapping relationship between depth and the accompanying texture frames into consideration. The proposed scheme is able to not only maintain the temporal consistency among interpolated depth sequence but also improve the quality of virtual frames generated by interpolated depth. Besides, it can be easily applied to arbitrary motion compensation based frame interpolation scheme. Experimental results demonstrate that the proposed depth frame interpolation scheme is able to improve the quality of virtual view texture frames in both subjective and objective criterions compared with existing schemes. [ABSTRACT FROM AUTHOR]

Stereo interleaving video coding, in which both left and right view frames are subsampled into half size and multiplexed into one single frame before being encoded by a traditional 2-D video encoder, is an efficient encoding scenario for stereoscopic video. Many existing stereo interleaving video coding methods subsample each frame by utilizing fixed subsampling filter coefficients. Such methods are easy to implement; however, the varying property of the frame signal is ignored. By jointly considering the influences of subsampling and compression, a rate and distortion analysis about stereo interleaving video coding is proposed. The final distortion in stereo interleaving video coding is the summation of errors caused by subsampling (causing distortion between subsampling-interpolated image and the original full resolution one) and by quantization during compression. Based on the provided rate distortion analysis, a content adaptive image subsampling (CAIS) is also proposed. In CAIS, the half-size frames are generated by the optimal subsampling filters, which are calculated based on frame contents and the targeted interpolation coefficients. Experimental results demonstrate that the proposed CAIS is able to greatly improve compression efficiency of stereo interleaving video coding. [ABSTRACT FROM AUTHOR]

Abstract: Reducing the frame rate of a sequence results in the information loss of interpolated frames that cannot be recovered by temporal up-sampling. The most prevalent frame rate reduction method is the direct down-sampling, which maintains the original frames of the input sequence with a constant interval. It achieves wide applications due to simplicity. However, it neglects the influences on interpolated frames brought by temporal up-sampling. To maintain more information during down-sampling, which will be used for interpolation, this paper proposes an up-sampling oriented frame rate reduction by hinging the temporal up-sampling and down-sampling. It is designed with the goal of improving the quality of up-sampled frames while keeping the down-sampled sequence faithful to the input one. For a particular temporal up-sampling method, e.g., motion compensation frame interpolation, the optimal down-sampled frame is the one that not only minimizes the difference between the original and the down-sampled frames but also minimizes the difference between the original and the corresponding up-sampled frames. Users can make a tradeoff between the two difference terms by selecting a proper lagrange factor depending on the applications. Furthermore, to decrease the space and computational complexity, a block-wise implementation is also devised in this paper. Experimental results conducted on various sequences demonstrate that the proposed frame rate reduction is able to significantly improve the quality of up-sampled frames compared with other down-sampling methods. [Copyright &y& Elsevier]

In this paper, auto regressive (AR) model is applied to error concealment for block-based packet video coding. In the proposed error concealment scheme, the motion vector for each corrupted block is first derived by any kind of recovery algorithms. Then each pixel within the corrupted block is replenished as the weighted summation of pixels within a square centered at the pixel indicated by the derived motion vector in a regression manner. Two block-dependent AR coefficient derivation algorithms under spatial and temporal continuity constraints are proposed respectively. The first one derives the AR coefficients via minimizing the summation of the weighted square errors within all the available neighboring blocks under the spatial continuity constraint. The confidence weight of each pixel sample within the available neighboring blocks is inversely proportional to the distance between the sample and the corrupted block. The second one derives the AR coefficients by minimizing the summation of the weighted square errors within an extended block in the previous frame along the motion trajectory under the temporal continuity constraint. The confidence weight of each extended sample is inversely proportional to the distance toward the corresponding motion aligned block whereas the confidence weight of each sample within the motion aligned block is set to be one. The regression results generated by the two algorithms are then merged to form the ultimate restorations. Various experimental results demonstrate that the proposed error concealment strategy is able to improve both the objective and subjective quality of the replenished blocks compared to other methods. [ABSTRACT FROM PUBLISHER]

Traditional methods for image downsampling commit to remove the aliasing artifacts. However, the influences on the quality of the image interpolated from the downsampled one are usually neglected. To tackle this problem, in this paper, we propose an interpolation-dependent image downsampling (IDID), where interpolation is hinged to downsampling. Given an interpolation method, the goal of IDID is to obtain a downsampled image that minimizes the sum of square errors between the input image and the one interpolated from the corresponding downsampled image. Utilizing a least squares algorithm, the solution of IDID is derived as the inverse operator of upsampling. We also devise a content-dependent IDID for the interpolation methods with varying interpolation coefficients. Numerous experimental results demonstrate the viability and efficiency of the proposed IDID. [ABSTRACT FROM PUBLISHER]