Search

Generating non-existing frames from a consecutive video sequence has been an interesting and challenging problem in the video processing field. Typical kernel-based interpolation methods predict pixels with a single convolution process that convolves source frames with spatially adaptive local kernels, which circumvents the time-consuming, explicit motion estimation in the form of optical flow. However, when scene motion is larger than the pre-defined kernel size, these methods are prone to yield less plausible results. In addition, they cannot directly generate a frame at an arbitrary temporal position because the learned kernels are tied to the midpoint in time between the input frames. In this paper, we try to solve these problems and propose a novel non-flow kernel-based approach that we refer to as enhanced deformable separable convolution (EDSC) to estimate not only adaptive kernels, but also offsets, masks and biases to make the network obtain information from non-local neighborhood. During the learning process, different intermediate time step can be involved as a control variable by means of an extension of coord-conv trick, allowing the estimated components to vary with different input temporal information. This makes our method capable to produce multiple in-between frames. Furthermore, we investigate the relationships between our method and other typical kernel- and flow-based methods. Experimental results show that our method performs favorably against the state-of-the-art methods across a broad range of datasets. Code will be publicly available on URL: https://github.com/Xianhang/EDSC-pytorch. [ABSTRACT FROM AUTHOR]

Given a video sequence, video frame interpolation aims to synthesize an in-between frame of two consecutive frames. In this paper, we propose a multi-scale position feature transform (MS-PFT) network for video frame interpolation where two parallel prediction networks and one optimization network are designed to predict the features of target frame and generate the final interpolation result, respectively. To increase the fidelity of the synthesised frames, we propose to apply a position feature transform (PFT) layer in the residual blocks of the prediction networks to estimate scaling factors which help evaluate different degrees of the importance of deep features around a target pixel. A PFT layer utilizes optical flow to extract and generate position features and then adjusts the learning process of our model. We further extend our model into a multi-scale structure in which each scale of the network shares the same parameters to maximise the efficiency of our network with model size unchanged. The experiments show that our method can handle the challenging scenarios like occlusion and large motion effectively and the proposed method outperforms those state-of-the-art approaches on different datasets. [ABSTRACT FROM AUTHOR]