Search

Abstract The demand for high‐resolution videos has been consistently rising across various domains, propelled by continuous advancements in societal. Nonetheless, limitations in imaging and economic factors often result in obtaining low‐resolution images. The currently available space‐time video super‐resolution methods often fail to fully exploit the information existing within the spatio‐temporal domain. To address this problem, the issue is tackled by conceptualizing the input low‐resolution video as a cuboid structure. An innovative methodology called “Cuboid‐Net”, which incorporates a multi‐branch convolutional neural network, is introduced. Cuboid‐Net is designed to collectively enhance the spatial and temporal resolutions of videos, enabling the extraction of rich and meaningful information across both spatial and temporal dimensions. Specifically, the input video is taken as a cuboid to generate different directional slices as input for different branches of the network. The proposed network contains four modules, that is, a multi‐branch‐based hybrid feature extraction module, a multi‐branch‐based reconstruction module, a first‐stage quality enhancement module, and a second‐stage cross frame quality enhancement module for interpolated frames only. Experimental results demonstrate that the proposed method is not only effective for spatial and temporal super‐resolution of video but also for spatial and angular super‐resolution of light field.

Abstract Aiming at accelerating the inter coding of versatile video coding (VVC), the existing deep‐learning‐based methods utilize a single convolutional neural network (CNN) to directly predict the quadtree plus multi‐type tree (QTMT)‐based partition of the whole coding tree unit (CTU). However, these methods adopt one prediction network for unevenly distributed CTUs and ignore that the different CTUs have different partition prediction difficulties, leading to performance degradation and computation waste. To overcome these limitations, a classification‐prediction joint framework is proposed to accelerate inter coding of VVC in this letter, which combines classification and prediction to process different CTUs through different networks with appropriate capacities. To achieve effective partition prediction of the whole CTU, the QTMT‐based partition is first modeled as the partition homogeneity map (PHM), which is a value map reflecting the partition of each 8× 8 unit. Second, the classification module classifies the CTUs into different classes according to their partition prediction difficulty, and then different prediction sub‐networks with appropriate capacities are utilized to predict the PHM for the corresponding CTU class. Finally, the decision tree (DT) is adopted to determine the optimal split modes based on the predicted PHM. Experimental results show that the approach achieves 44.5% time saving with 1.94% BD‐BR increase, outperforming state‐of‐the‐art approaches.

Abstract To meet the challenge of compressing the vast light field data, many prediction‐based algorithms achieve satisfactory results. Although these algorithms remove most redundancy through angular prediction, they rarely develop the suitable de‐correlation solutions for the prediction residue, leading to a large gap for higher compression performance. Therefore, this letter proposes a hybrid residue transform (HRT) mechanism to fully exploit the correlations in prediction residue for a three‐layer hierarchical light field compression. Specifically, the prediction residue is hierarchically generated from the authors' three‐layer structure. Besides, considering the high‐dimensional feature of light field residue, the high‐dimensional transforms, including 3D and 4D discrete cosine transforms, are alternately utilized to make the energy of the residue more compact. Experimental results demonstrate that the proposed method outperforms the state‐of‐the‐art methods.

Recently, stereoscopic image quality assessment has attracted a lot attention. However, compared with 2D image quality assessment, it is much more difficult to assess the quality of stereoscopic images due to the lack of understanding of 3D visual perception. This paper proposes a novel no-reference quality assessment metric for stereoscopic images using natural scene statistics with consideration of both the quality of the cyclopean image and 3D visual perceptual information (binocular fusion and binocular rivalry). In the proposed method, not only is the quality of the cyclopean image considered, but binocular rivalry and other 3D visual intrinsic properties are also exploited. Specifically, in order to improve the objective quality of the cyclopean image, features of the cyclopean images in both the spatial domain and transformed domain are extracted based on the natural scene statistics (NSS) model. Furthermore, to better comprehend intrinsic properties of the stereoscopic image, in our method, the binocular rivalry effect and other 3D visual properties are also considered in the process of feature extraction. Following adaptive feature pruning using principle component analysis, improved metric accuracy can be found in our proposed method. The experimental results show that the proposed metric can achieve a good and consistent alignment with subjective assessment of stereoscopic images in comparison with existing methods, with the highest SROCC (0.952) and PLCC (0.962) scores being acquired on the LIVE 3D database Phase I.

In order to adapt to various real-time applications, fast coding algorithms for high efficiency video coding (HEVC) standard maintain a hot research topic in recent years. In this paper, a complexity reduction algorithm based on hierarchical classification for HEVC inter coding is proposed. It consists of five fast algorithms which is accomplished by hierarchical classification trees at coding unit (CU) level, prediction unit (PU) level and transformation unit (TU) level respectively. At the beginning of proposed algorithm, intra features and inter features which describe the texture and context properties of CU, PU and TU are extracted from the training set. Then the classification trees for CU, PU and TU are generated by carefully selecting features and designing the classification criteria. By analyzing the spatiotemporal correlation, two strategies including early termination and early split are applied to fast coding by referring to these classification trees. The objective evaluation demonstrates that the proposed algorithm can significantly reduce coding complexity with little compression loss. Particularly the subjective evaluation based on similarity measurement for color histogram approves that decoded video quality between the original HM16.9 algorithm and the proposed algorithm is nearly identical.

Event detection is an important task in the field of natural language processing, which aims to detect trigger words in a sentence and classify them into specific event types. Event detection tasks suffer from data sparsity and event instances imbalance problems in small-scale datasets. For this reason, the correlation information of event types can be used to alleviate the above problems. In this paper, we design a Hierarchical Attention Neural Network for Event Types (HANN-ET). Specifically, we select Long Short-Term Memory (LSTM) as the semantic encoder and utilize dynamic multi-pooling and the Graph Attention Network (GAT) to enrich the sentence feature. Meanwhile, we build several upper-level event type modules and employ a weighted attention aggregation mechanism to integrate these modules to obtain the correlation event type information. Each upper-level module is completed by a Neural Module Network (NMNs), event types within the same upper-level module can share information, and an attention aggregation mechanism can provide effective bias scores for the trigger word classifier. We conduct extensive experiments on the ACE2005 and the MAVEN datasets, and the results show that our approach outperforms previous state-of-the-art methods and achieves the competitive F1 scores of 78.9% on the ACE2005 dataset and 68.8% on the MAVEN dataset.

Since a large proportion of the information that is received daily is in the form of images, a highly effective no-reference stereo image quality assessment (SIQA) method is desired. This paper proposes an improved method that covers wide quality-aware features, including the structure, color, luminance, phase, and human visual system (HVS). To be specific, since human eyes are highly sensitive to the structure of images, the gradient magnitude (GM) and gradient orientation (GO) are extracted from the left and right views of the stereo image. Considering the influence of color distortions, the images are decomposed into the RGB channels to be processed, and the local gradient of the color image is obtained by adding up the RGB gradient vectors. In addition, according to the study of the two main visual channels, especially the cyclopean and disparity maps, the binocular-related images of position and phase congruency are generated. Correspondingly, two special dictionaries for the gradient and phase are trained to parse the high-dimensional sample sets. The experimental results show that the proposed metric always achieves high consistency with human subjective assessments for both symmetric and asymmetric distortions.

Evaluation of light field image (LFI), especially micro-lens camera light field (LF), is a new and challenging work. The development of image quality assessment (IQA) metric of LFIs relies on the subjective quality assessment database. In this paper, we establish a perceptual quality assessment dataset consisting of 240 distorted images from 8 source images with five distortion types. Furthermore, a no-reference IQA metric is proposed by combining 2D and 3D characteristics of LFI with the Support Vector Regression (SVR) model. The performance of the proposed metric is demonstrated by comparing with some classical full reference IQA metrics both on the presented dataset and a third-party dataset. The experiment results show that our method has a better performance than others.

Because of the capacity of capturing both the spatial and angular information of the light rays simultaneously, light field images (LFIs) contain richer scene information compared with conventional images, but at the cost of huge volume. This paper proposes a novel LFI sparse compression framework driven by convolutional neural network (CNN). The epipolar plane image (EPI) super-resolution is for compensating the information loss caused by sparse sampling and the decoder-side sub-aperture images (SAIs) quality enhancement is for compensating the information loss caused by lossy compression. Specifically, we choose those SAIs both in odd rows and odd columns as our key SAIs and compress them using standard video encoder. For those non-key SAIs, we predict them using decompressed key SAIs by taking advantage of the special structure of EPI. The low-resolution EPIs generated from the sparse SAIs are super-resolved by a CNN and the outputs, high-resolution EPIs, are used to rebuild the dense SAIs. Moreover, in order to improve the quality of the predicted SAI, we add decoder-side quality enhancement before prediction. We propose a multi-scale dense residual network (MSDRN) to implement both EPI super-resolution and quality enhancement. Transfer learning strategies are used to improve the training performance of quality enhancement. The experimental results show the superior performance of the proposed framework over existing methods in terms of rate-distortion performance.

Perceptual stereo image quality assessment (SIQA) aims to design computational models to measure the stereo image quality in accordance with human opinions. In this paper, a novel reduced-reference (RR) SIQA is proposed by characterizing the statistical and perceptual properties of the stereo image in both the spatial and gradient domains. To be specific, in the spatial domain, we extract the parameters of the generalized Gaussian distribution fits of luminance wavelet coefficients to form the underlying features. In the gradient domain, the modified gradient magnitudes maps are generated by jointly considering human visual system's contrast sensitivity and neighborhood gradient information to weight the gradient magnitudes in a locally adaptive manner. Afterward, perceptual features are extracted based on the entropy of discrete wavelet transform coefficients of modified gradient magnitudes. Furthermore, we consolidate the left and right features into a single set of features per stereo image pair. Finally, the qualities of both the spatial and gradient domains are combined to obtain the overall quality of stereo image. Extensive experiments performed on popular data sets demonstrate that the proposed RR-SIQA method achieves highly competitive performance as compared with the state-of-the-art RR-SIQA models as well as full-reference ones for both symmetric and asymmetric distortions.

Massive recent investigations from both industry and academia have been poured into autostereoscopic display. One of the emerging techniques, light field image (LFI), can provide more immersive perception by increasing the number of views and the spatial resolution. However, these advantages restrict the storage and transmission due to such dense-view image simultaneously. To solve this problem, we propose to compress the LFI using multiview video plus depth (MVD) coding architecture. In this paper, we preliminarily estimate the depth based on the concept of epipolar plane image. To achieve a depth value, we design an optimal slope decision algorithm to determine the best slope with the minimal cost. While this rough estimation produces some error points within initial depth map, therefore, we present a depth optimization algorithm using the characteristic of the associated texture image. Ultimately, a small number of selected viewpoint images are encoded with their corresponding depth maps using the MVD framework, and then, the unselected viewpoint images are synthesized by depth image-based rendering technique. To verify the validity of the proposed LFI compression scheme, extensive experiments are conducted. The simulated results demonstrate that the proposed depth map estimation algorithm is superior to other state-ofthe-art methods for the LFI. Meanwhile, our LFI compression method outperforms other LFI compression algorithms significantly.

The latest Joint Video Exploration Team employs quad-tree plus binary-tree (QTBT) block partitioning structure, which can improve coding performance significantly than High Efficiency Video Coding with hugely increased encoding complexity. To address this issue, we propose a novel fast QTBT partition method through a convolutional neural network (CNN). Specifically, the proposed algorithm uses CNN to predict the QTBT partition depth range of 32 × 32 block directly according to the inherent texture richness of the image, rather than to judge split or not at each depth level. For training optimization, we introduce a misclassification penalty term combined with L2 HingeLoss function, which can further boost the classification accuracy. Experimental results demonstrate the effectiveness of our proposed method; our rate-distortion maintaining setting can achieve 42.33% complexity reduction with just 0.69% bitrate increase. Our low complexity setting can achieve 62.08% complexity reduction with 2.04% bitrate increase.

The research on autonomous driving based on deep reinforcement learning algorithms is a research hotspot. Traditional autonomous driving requires human involvement, and the autonomous driving algorithms based on supervised learning must be trained in advance using human experience. To deal with autonomous driving problems, this paper proposes an improved end-to-end deep deterministic policy gradient (DDPG) algorithm based on the convolutional block attention mechanism, and it is called multi-input attention prioritized deep deterministic policy gradient algorithm (MAPDDPG). Both the actor network and the critic network of the model have the same structure with symmetry. Meanwhile, the attention mechanism is introduced to help the vehicles focus on useful environmental information. The experiments are conducted in the open racing car simulator (TORCS)and the results of five experiment runs on the test tracks are averaged to obtain the final result. Compared with the state-of-the-art algorithm, the maximum reward increases from 62,207 to 116,347, and the average speed increases from 135 km/h to 193 km/h, while the number of success episodes to complete a circle increases from 96 to 147. Also, the variance of the distance from the vehicle to the center of the road is compared, and the result indicates that the variance of the DDPG is 0.6 m while that of the MAPDDPG is only 0.2 m. The above results indicate that the proposed MAPDDPG achieves excellent performance.

The 3D High Efficiency Video Coding (3D-HEVC) standard aims to code 3D videos that usually contain multi-view texture videos and its corresponding depth information. It inherits the same quadtree prediction structure of HEVC to code both texture videos and depth maps. Each coding unit (CU) allows recursively splitting into four equal sub-CUs. At each CU depth level, it enables 10 types of inter modes and 35 types of intra modes in inter frames. Furthermore, the inter-view prediction tools are applied to each view in the test model of 3D-HEVC (HTM), which uses variable size disparity-compensated prediction to exploit inter-view correlation within neighbor views. It also exploits redundancies between a texture video and its associated depth using inter-component coding tools. These achieve the highest coding efficiency to code 3D videos but require a very high computational complexity. In this paper, we propose a context-adaptive based fast CU processing algorithm to jointly optimize the most complex components of HTM including CU depth level decision, mode decision, motion estimation (ME) and disparity estimation (DE) processes. It is based on the hypothesis that the optimal CU depth level, prediction mode and motion vector of a CU are correlated with those from spatiotemporal, inter-view and inter-component neighboring CUs. We analyze the video content based on coding information from neighboring CUs and early predict each CU into one of five categories i.e., DE-omitted CU, ME-DE-omitted CU, SPLIT CU, Non-SPLIT CU and normal CU, and then each type of CU adaptively adopts different processing strategies. Experimental results show that the proposed algorithm saves 70% encoder runtime on average with only a 0.1% BD-rate increase on coded views and 0.8% BD-rate increase on synthesized views. Our algorithm outperforms the state-of-the-art algorithms in terms of coding time saving or with better RD performance.