Search

The objective of this paper is to develop an effective photographic style transfer method while preserving the semantic correspondence between the style and content images for both scenery and portrait images. A semantic correspondence guided photographic style transfer algorithm is developed, which is to ensure that the semantic structure of the content image has not been changed while the color of the style images is being migrated. The semantic correspondence is constructed in large scale regions based on image segmentation and also in local scale patches using Nearest-neighbor Field Search in the deep feature domain. Based on the semantic correspondence, a matting optimization is utilized to optimize the style transfer result to ensure the semantic accuracy and transfer faithfulness. The proposed style transfer method is further extended to automatically retrieve the style images from a database to make style transfer more-friendly. The experimental results show that our method could successfully conduct the style transfer while preserving semantic correspondence between diversity of scenes. A user study also shows that our method outperforms state-of-the-art photographic style transfer methods.

Image inpainting is a challenging task due to the loss of the image information. Recently, GAN-based approaches have shown promising performance in the field of image inpainting. For this task, a superior similarity measurement of extracted patches from known and missing regions is important. Existing approaches usually adopt cosine distance to measure this similarity for missing region reconstruction. However, from the semantic-level perspective, these methods often generate content with inconsistent color and disorder structure due to the ignorance of the magnitude distance of the attended patches. To resolve this problem, we propose a Hybrid Contextual Attention Network (HCA-Net) with a novel attention module called hybrid contextual attention module (HCAM). HCAM takes account of both cosine distance and Euclidean distance as the measurement of the extracted patches and gives a better prediction of missing features. Besides, a Spectral-Normalization patch discriminator and the cosine loss are added into the model for patch-level and pixel-level consistency enhancement. Extensive results on three public datasets (Paris Street View, Celeba-HQ, and Places2), have both validated that our approach significantly outperforms the state-of-the-art approaches.

Recent developments of video super-resolution reconstruction often exploit spatial and temporal contexts from input frame sequence by making use of explicit motion estimation, e.g., optical flow, which may introduce accumulated errors and requires huge computations to obtain an accurate estimation. In this paper, we propose a novel multi-branch dilated convolution module for real-time frame alignment without explicit motion estimation, which is incorporated with the depthwise separable up-sampling module to formulate a sophisticated real-time video super-resolution network. Specifically, the proposed video super-resolution framework can efficiently acquire a larger receptive field and learn spatial–temporal features of multiple scales with minimal computational operations and memory requirements. Extensive experiments show that the proposed super-resolution network outperforms current state-of-the-art real-time video super-resolution networks, e.g., VESPCN and 3DVSRnet, in terms of PSNR values (0.49 dB and 0.17 dB) on average in various datasets, but requires less multiplication operations.

In recent years, convolutional neural networks (CNNs) have accelerated the developments of video super resolution (SR) for achieving higher image quality. However, the computational cost of existing CNN-based video super-resolution is too heavy for real-time applications. In this paper, we propose a new video super-resolution framework using lightweight frame alignment module and well-designed up-sampling module for real-time processing. Specifically, our framework, which is called as Lightweight Shuffle Video Super-Resolution Network (LSVSR), combines channel shuffling, depthwise convolution and pointwise group convolution to significantly reduce the computational burden during frame alignment and high-resolution frame reconstruction. On the public benchmark datasets, our proposed network outperforms the state-of-the-art lightweight video SR networks in terms of objective (PSNR and SSIM) and subjective evaluations, number of network parameters and floating-point operations. Our network can achieve real-time 540P to 2160P 4 × super-resolution for more than 60fps using desktop GPUs or mobile phones with neural processing unit.

The objective of this paper is to develop an effective photographic transfer method while preserving the semantic correspondence between the style and content images. A semantic correspondence guided deep photo style transfer algorithm is developed, which is to ensure that the semantic structure of the content image has not been changed while the color of the style images is being migrated. The semantic correspondence is constructed in large scale regions based on image segmentation and also in local scale patches using deep image analogy. Based on the semantic correspondence, a matting optimization is utilized to optimize the style transfer result to ensure the semantic accuracy and transfer faithfulness. The proposed style transfer method is further extended to automatically retrieve the style images from a database to make style transfer more-friendly. The experimental results show that our method could successfully conduct the style transfer while preserving semantic correspondence between diversity of scenes. A user study also shows that our method outperforms state-of-the-art photographic style transfer methods.

Automatic fine-art painting classification is an important task to assist the analysis of fine-art paintings. In this paper, we propose a novel two-channel deep residual network to classify fine-art painting images. In detail, we take the advantage of the ImageNet to pre-train the deep residual network. Our two channels include the RGB channel and the brush stroke information channel. The gray-level co-occurrence matrix is used to detect the brush stroke information, which has never been considered in the task of fine-art painting classification. Experiments demonstrate that the proposed model achieves better classification performance than other models. Moreover, each stage of our model is effective for the image classification.

The large-scale video data on the web contain a lot of semantics, which are an important part of semantic web. Video descriptors can usually represent somewhat the semantics. Thus, they play a very important role in web multimedia content analysis, such as Scale-invariant feature transform SIFT feature. In this paper, we proposed a new video descriptor, called a temporal-compress and shorter SIFTTC-S-SIFT which can efficiently and effectively represent the semantics of web videos. By omitting the least discriminability orientation in three stages of standard SIFT on every representative frame, the dimensions of the shorter SIFT are reduced from 128-dimension to 96-dimension to save space storage. Then, the SIFT can be compressed by tracing SIFT features on video temporal domain, which highly compress the quantity of local features to reduce visual redundancy, and keep basically the robustness and discrimination. Experimental results show our method can yield comparable accuracy and compact storage size.