Search

Due to hardware limitations, existing hyperspectral (HS) camera often suffer from low spatial/temporal resolution. Recently, it has been prevalent to super‐resolve a low resolution (LR) HS image into a high resolution (HR) HS image with a HR RGB (or multispectral) image guidance. Previous approaches for this guided super‐resolution task often model the intrinsic characteristic of the desired HR HS image using hand‐crafted priors. Recently, researchers pay more attention to deep learning methods with direct supervised or unsupervised learning, which exploit deep prior only from training dataset or testing data. In this article, an efficient convolutional neural network‐based method is presented to progressively super‐resolve HS image with RGB image guidance. Specifically, a progressive HS image super‐resolution network is proposed, which progressively super‐resolve the LR HS image with pixel shuffled HR RGB image guidance. Then, the super‐resolution network is progressively trained with supervised pre‐training and unsupervised adaption, where supervised pre‐training learns the general prior on training data and unsupervised adaptation generalises the general prior to specific prior for variant testing scenes. The proposed method can effectively exploit prior from training dataset and testing HS and RGB images with spectral‐spatial constraint. It has a good generalisation capability, especially for blind HS image super‐resolution. Comprehensive experimental results show that the proposed deep progressive learning method outperforms the existing state‐of‐the‐art methods for HS image super‐resolution in non‐blind and blind cases. [ABSTRACT FROM AUTHOR]

Fifth generation mobile networks (5G) is coming into our life and it will not only provide an increase of 1000 times in Internet traffic in the next decade but will also offer "smarter" user experience. With the commercial uses of 5G, online social networks and smart phones will spring up again. Then cross-modality data will play a more important role in the daily information dissemination. As an effective way of content analysis, topic detection has attracted much research interest, but conventional topic analysis is undergoing the limitations from the cross-modality heterogenous data. This paper proposes a sentimental image dominant graph topic model, that can detect the topic from the heterogenous data and mine the sentiment of each topic. In details, we design a topic model to transfer both the low-level visual modality and the high-level text modality into a semantic manifold, and improve the discriminative power of CNN feature by jointly optimizing the output of both convolutional layer and fully-connected layer. Furthermore, since the sentimental impact is very significant for understanding the intrinsic meaning of topics, we introduce a semantic score of subjective sentences to calculate the sentiment on the base of the contextual sentence structure. The comparison experiments on the public cross-modality benchmark show the promising performance of our model. So our method using AI technology will facilitate the intellectualization of 5G. [ABSTRACT FROM AUTHOR]

Frame rate up-conversion (FRUC) usually suffers from unreliable motion vectors due to the absence of the current frame to be interpolated. In addition, since the majority of video sequences are usually compressed by various coding standards to reduce the data volume, the quality of the generated frames in the FRUC will be further impaired. To address this problem, we proposed two FRUC algorithms based on deep residual network. We first present a deep residual network for the FRUC (DRNFRUC), which consists of feature extraction, feature recursive analysis, and image restoration parts with a skip connection between the input and the output of the network. The proposed DRNFRUC takes the result of an arbitrary existing FRUC method as the input and is able to significantly reduce the edge blurring and blocking artifacts when the motion of the block is violent. In addition, we proposed a deep residual network with weighted convolutional motion compensation (DRNWCMC) for the FRUC, where the convolution operations can be embedded into the motion compensation interpolation (MCI) in any existing MCI-based FRUC method. In DRNWCMC, we first devise two convolutional neural networks corresponding to the forward and backward motion compensated frames, respectively. And then, the adaptive interpolation coefficients for motion compensation are designed as two $1\times1$ convolutional kernels. Finally, the interpolation result of WCMC is fed into another convolutional neural network to further improve the performance. All the parameters involved in the DRNWCMC are trained simultaneously under the same cost function. The experimental results show that the two proposed algorithms can remarkably improve both the objective and subjective quality of the interpolated frames. [ABSTRACT FROM AUTHOR]

Image restoration methods based on convolutional neural networks have shown great success in the literature. However, since most of networks are not deep enough, there is still some room for the performance improvement. On the other hand, though some models are deep and introduce shortcuts for easy training, they ignore the importance of location and scaling of different inputs within the shortcuts. As a result, existing networks can only handle one specific image restoration application. To address such problems, we propose a novel adaptive residual network (ARN) for high-quality image restoration in this paper. Our ARN is a deep residual network, which is composed of convolutional layers, parametric rectified linear unit layers, and some adaptive shortcuts. We assign different scaling parameters to different inputs of the shortcuts, where the scaling is considered as part parameters of the ARN and trained adaptively according to different applications. Due to the special construction of ARN, it can solve many image restoration problems and have superior performance. We demonstrate its capabilities with three representative applications, including Gaussian image denoising, single image super resolution, and JPEG image deblocking. Experimental results prove that our model greatly outperforms numerous state-of-the-art restoration methods in terms of both peak signal-to-noise ratio and structure similarity index metrics, e.g., it achieves 0.2–0.3 dB gain in average compared with the second best method at a wide range of situations. [ABSTRACT FROM AUTHOR]

Given the tremendous growth of sport fans, the 'Intelligent Arena', which can greatly improve the fun of traditional sports, becomes one of the new-emerging applications and research topics. The development of multimedia computing and artificial intelligence technologies support intelligent sport video analysis to add live video broadcast, score detection, highlight video generation, and online sharing functions to the intelligent arena applications. In this paper, we have proposed a deep learning based video analysis scheme for intelligent basketball arena applications. First of all, with multiple cameras or mobile devices capturing the activities in arena, the proposed scheme can automatically select the camera to give high-quality broadcast in real-time. Furthermore, with basketball energy image based deep conventional neural network, we can detect the scoring clips as the highlight video reels to support the wonderful actions replay and online sharing functions. Finally, evaluations on a built real-world basketball match dataset demonstrate that the proposed system can obtain 94.59 % accuracy with only less than 45 m s processing time (i.e., 10 m s broadcast camera selection, and 35 m s for scoring detection) for each frame. As the outstanding performance, the proposed deep learning based basketball video analysis scheme is implemented into a commercial intelligent basketball arena application named 'Standz Basketball'. Although the application had been only released for one month, it achieves the 85 t h day download ranking place in the sport category of Chinese iTunes market. [ABSTRACT FROM AUTHOR]

Aerial-view geo-localization tends to determine an unknown position through matching the drone-view image with the geo-tagged satellite-view image. This task is mostly regarded as an image retrieval problem. The key underpinning this task is to design a series of deep neural networks to learn discriminative image descriptors. However, existing methods meet large performance drops under realistic weather, such as rain and fog, since they do not take the domain shift between the training data and multiple test environments into consideration. To minor this domain gap, we propose a Multiple-environment Self-adaptive Network (MuSe-Net) to dynamically adjust the domain shift caused by environmental changing. In particular, MuSe-Net employs a two-branch neural network containing one multiple-environment style extraction network and one self-adaptive feature extraction network. As the name implies, the multiple-environment style extraction network is to extract the environment-related style information, while the self-adaptive feature extraction network utilizes an adaptive modulation module to dynamically minimize the environment-related style gap. Extensive experiments on three widely-used benchmarks, i.e. , University-1652, SUES-200, and CVUSA, demonstrate that the proposed MuSe-Net achieves a competitive result for geo-localization in multiple environments. Furthermore, we observe that the proposed method also shows great potential to the unseen extreme weather, such as mixing the fog, rain and snow. • Identifying one key challenge in visual geo-localization: weather and illumination changes. • Presenting MuSe-Net to alleviate the interference caused by environmental changes. • Designing Residual SPADE for efficient training and feature discrimination boosting. • Results on three geo-localization benchmarks confirm the superiority of our MuSe-Net. [ABSTRACT FROM AUTHOR]

The assessment of image is a critical task in computer vision, particularly in a no-referenced context, which presents numerous challenges. Though Deep neural networks have exhibited excellent performance on no-reference quality assessment (NR-IQA), the distortion-aware NR-IQA methods are limited to specific distortion type and struggle to handle realistic distortion scenarios. In this work, we propose a feature rectification and enhancement convolutional neural network namely FREIQA for NR-IQA. Our approach targets to fully utilize the multi-stage semantic features to produce a well fused and rectified score vector for effective image quality regression. Specifically, the multi-stage semantic features extracted by a pre-trained backbone would be rectified with a multi-level channel attention module before integrating to a global score vector. The fused score vector then will be gradually optimized through the perception feature provided by the multi-stage semantic feature, and finally sent to quality regression network to obtain the image quality score. Experimental results on public benchmarks including LIVE, TID2013, CSIQ, LIVEC, KADID-10k, KonIQ-10k and Waterloo exploration database show that our FREIQA achieve the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

This paper introduces a novel video saliency model for salient object detection in videos. Firstly, we generate multi-level deep features via a symmetrical convolutional neural network, in which the inputs are the current frame and the optical flow image. Then, the multi-level deep features are integrated in a hierarchical manner using a fusion network, which deploys attention module to make a selection for deep features. Lastly, the integrated deep feature is combined with the boundary information originated from shallow layer of the feature extraction networks, and the saliency map is generated by the saliency prediction step. The key advantages of our model lie on the attention module, the hierarchical integration and the boundary information, in which the former one acts as weight filter and is used to select the most salient regions in deep features, the middle one gives an effective integration manner for features from different layers and the last one provides well-defined boundaries for saliency map. Extensive experiments are performed on two challenging video dataset, and the experimental results show that our model consistently outperforms the state-of-the-art saliency models in a large margin. [ABSTRACT FROM AUTHOR]