Search

This paper presents a revised rate control scheme based on an improved frame complexity measure. Rate control adopted by both MPEG-4 VM18 and H.264/AVC use a quadratic rate–distortion (R–D) model that determines quantization parameters (QPs). Classical quadratic R–D model is suitable for MPEG-4 but it performs poorly for H.264/AVC because one of the important parameters, mean absolute difference (MAD), is predicted through a linear model, whereas the MAD used in MPEG-4 VM18 is the actual MAD. Inaccurately predicted MAD results in wrong QP and consequently degrades rate–distortion optimization (RDO) performance in H.264. To overcome the limitation of the existing rate control schemes, we introduce an enhanced linear model for predicting MAD, utilizing some knowledge of current frame complexity. Moreover, we propose a more accurate frame complexity measure, namely, normalized MAD, to replace the current use of MAD parameter. Normalized MAD has a stronger correlation with optimally allocated bits than that of the predicted MAD. To minimize video quality variations, we also propose a novel long-term QP limiter (LTQPL). Finally, a dynamic bit allocation scheme among basic units is implemented. Extensive simulation results show that our method, with inexpensive computational complexity added, improves the average peak signal-to-noise ratio (PSNR) and reduces video quality variations considerably. [Copyright &y& Elsevier]

In this paper, we presents a PSNR-based frame complexity estimation to improve H.264/AVC rate control. Our scheme is based on adding PSNR-based complexity estimation to the existing mean absolute difference based (MAD-based) complexity measure to form combined frame complexity estimation. The aim is to allocate bits more accurately, especially for frames with scene changes and high motions. Bit allocation to each frame is not just computed by encoder buffer status but also by a combined frame complexity measure. The quadratic rate-quantization (R-Q) model is used to determine the quantization parameter (QP) for each frame and the computed QP is further adjusted in two special cases. We also propose a frame skipping decision scheme to improve the performance at high frame rates. Simulation results show that the H.264 coder, using our proposed scheme, achieves better visual performances with similar or smaller PSNR deviations when compared to that of the recent JM8.6 rate control. [ABSTRACT FROM AUTHOR]

This paper presents a method for automatic optimization of FAPs (Facial Animation Parameters) based on a face video sequence. An FAP optimizer unit compares a rendered face with the original face in the video sequence. The difference is iteratively minimized to produce an optimal set of FAPs. Unnecessary FAPs are eliminated and the search is partitioned to speed up the optimization process. Three different search techniques, Steepest Descent Method, Linear Search Method, and Cyclic Coordinates Method are applied to derive an optimum. The resulting FAPs are very accurate and the animation based on the FAPs is lifelike and truthful to the original video. [ABSTRACT FROM AUTHOR]

There are still some challenges in the task of single image rain removal, such as artefact remnant, background over‐smooth, and increasingly complex and heavy‐weight network architecture. Especially too heavy‐weight network to fit outdoor detection devices or mobile devices. To address the above challenges, we propose a lightweight single image Deraining algorithm incorporating visual attention saliency mechanisms (LDVS). The proposed network consists of five blocks and two convolution operations, where each block consists of a dilation convolution module and a convolutional block attention module (CBAM). Specifically, visual saliency module CBAM is used for accurate localization of rain streak, and further the combinations of dilated convolution with CBAM is used to extract feature maps of rain streaks faithfully, which is able to remove artefact remnant while maintaining background details. A good tradeoff is presented between the network's weight size and effect of rain removal. Specifically, with only 48,268 parameters, the proposed model can achieve a guaranteed performance. Extensive experiments on a few typical rainy scenarios on synthetic and real‐world datasets have demonstrated that to achieve the same level of performance, the proposed method has far smaller size than most of the baselines under both qualitative and quantitative analyses. [ABSTRACT FROM AUTHOR]

Presents information on a study which developed a method for re-encoding data from MPEG video decoder before sending them to dynamic random-access memory. Encoding algorithm; Decoding algorithm; Simulation results; Conclusion.

Presents information on a video decoder with an efficient block-level pipeline controller scheme for MPEG-2. Features of the video decoder; External DRAM scheme; Simulation results.

Presents an approach to increase the amount of vector quantization (VQ) codebook patterns to improve image quality without a large overhead on the codebook memory and search time. Features of VQ system; Two main technique, pattern characterization and operation of a VQ system; Codebook structure and search strategy; Experimental results.

Presents a model and a simulator to aid Motion Picture Experts Group (MPEG) decoder design by providing useful statistics related to bus utilization and waiting cycles. MPEG decoder architecture and data nature; Results on the analysis of bus bandwidth and determining proper sizes of decoder input/output (I/O) buffers connected to the bus; Conclusions.

Although the in-loop filtering incorporated in High Efficiency Video Coding (HEVC) standard improves the subjective quality of reconstructed pictures and increases the compression efficiency, it still cannot satisfy the demand for higher quality in the rapid growth of video usage. In this paper, we propose recursive residual convolution neural network (RRCNN)-based in-loop filtering to further improve the quality of reconstructed intra frames while reducing the bitrates. Specifically, RRCNN estimates the residual images between the compressed distorted images and original noncompressed ones, and there are shortcut connections that skip a few stacked layers in the structure of RRCNN to ease the training difficulty. By applying the same set of weights recursively, RRCNN achieves excellent performance while utilizing far fewer parameters. For concise in-loop filtering, we train a single model capable of handling various bitrate settings. Different networks for the filtering of luma and chroma components are designed respectively to better learn the filtering characteristics of different channels. Moreover, to fully adapt the various input videos and boost the performance, a coding tree unit (CTU) control flag is signaled to indicate the filtering method from the sense of rate-distortion optimization (RDO). Extensive experimental results show that our scheme achieves significant bitrate savings compared to HEVC, leading to on average 8.7% BD-rate reduction, with up to a 15.1% BD-rate reduction for luma, and more than 20% BD-rate reductions for chroma on average. [ABSTRACT FROM AUTHOR]

Arbitrary shape text detection is a challenging task in scene text recognition. Driven by deep learning and large‐scale data sets, the detection method based on connected component (CC) has increasingly gained popularity. However, there are still problems of unclear separation of text instances and incorrect component links. Thus, in this paper, the authors propose a novel component connection method, that is, Fragmented Affinity Reasoning Network of Text Instances (FARNet), for arbitrary shape text detection. The network consists of a Weighted Feature Fusion Pyramid Network (WFFPN), Text Fragments Subgraph (TFS), and Dense Graph Attention Network (DGAT), which can be trained end‐to‐end. The WFFPN is used to generate text fragments, TFS and DGAT jointly construct an affinity reasoning network. Since the neighbouring boundaries between text instances may blend them into a single instance, the core idea is to use the WFFPN to divide the text instance into a series of rectangular fragments, the affinity reasoning network infers the affinity between fragments and then links them to rebuild text instances. Extensive experiments on seven challenging datasets (ICDAR2015, MSRA‐TD500, Totaltext, CTW‐1500, ICDAR 2019MLT, ICDAR2019 ArT, and DAST‐1500) demonstrate that the proposed text detector achieves state‐of‐the‐art performance in both on polygon datasets and quadrilateral datasets. The code is available at https://github.com/giganticpower/FARNet. [ABSTRACT FROM AUTHOR]

The arbitrary-shape scene text detection faces enormous challenges in accuracy and speed with increasing demand in the industrial sector. Although most studies have introduced segmentation and achieved remarkable performance in text detection tasks, researchers ignored the importance of detection efficiency. This paper proposes a new Joint Multi-Space Perception Network (JMNET) for efficient scene text detection to address this issue. Based on a lightweight feature extraction backbone, we put forward two novel modules, i.e., Scale Spatial Perception Module (SSPM) and Attention Spatial Perception Module (ASPM) to enhance the expression ability of text features with low computational complexity. Moreover, we propose an Unsupervised Embedding Spatial Perception Loss function (UnESP Loss) by introducing the Euclidean distance measurement between the embeddings to overcome the ambiguity of text instance boundary, such as a small line spacing. In this way, text embedding learning is not restricted by specific shapes, and detection robustness can be improved. Extensive experiments on four benchmarks, including the ICDAR2015, MSRA-TD500, CTW1500 and Totaltext, show that the proposed JMNET achieves competitive performance in terms of both accuracy and speed over the state-of-the-art methods. Particularly, our method can reach a remarkable F-measure of 85.5% at 52 FPS on Totaltext. Code is available at: https://github.com/sakura-910/JMNET. [ABSTRACT FROM AUTHOR]

Deep neural networks have achieved great performance on blind Image Quality Assessment (IQA), but it is still challenging for using one network to accurately predict the quality of images with different distortions. In this paper, a Distortion-Aware Convolutional Neural Network (DACNN) is proposed for blind IQA, which works effectively for not only synthetically distorted images but also authentically distorted images. The proposed DACNN consists of a distortion aware module, a distortion fusion module, and a quality prediction module. In the distortion aware module, a Siamese network-based pretraining strategy is proposed to design a synthetic distortion-aware network for full learning the synthetic distortions, and an authentic distortion-aware network is used for extracting the authentic distortions. To efficiently fuse the learned distortion features, and make the network pay more attention to the essential features, a weight-adaptive fusion network is proposed to adaptively adjust the weight of each distortion. Finally, the quality prediction module is adopted to map the fused features to a quality score. Extensive experiments on four authentic IQA databases and four synthetic IQA databases have proved the effectiveness of the proposed DACNN. [ABSTRACT FROM AUTHOR]

With the exploding development of edge intelligence and smart industry, deep learning-based intelligent industrial solutions are promptly applied in the manufacturing process. Many intelligent industrial solutions such as automatic manufacturing inspection are computer vision based and require fast and efficient video encoding techniques so that video streams can be processed as quickly as possible either at the edge cluster or over the cloud. As one of the most popular video coding standards, the high efficiency video coding (HEVC) standard has been applied to various industrial scenes. However, HEVC brings not only a higher compression rate but also a significant increase in encoding complexity, which hinders its practical application in industrial scenarios. Fortunately, a large amount of video coding data makes it possible to accelerate the encoding process in the industry. To speed up the video coding process in some industrial scenes, this paper proposes a data-driven fast approach for coding tree unit (CTU) partitioning in HEVC intracoding. First, we propose a method to represent the partition result of a CTU as a column vector of length 21. Then, we employ lots of encoding data produced in normal industry scenes to train transformer models used to predict the partitioning vector of the CTU. Finally, the final partitioning structure of the CTU is generated from the partitioning vector after a postprocessing operation and used by an industrial encoder. Compared with the original HEVC encoder used by some industrial applications, experiment results show that our approach achieves 58.77% encoding time reduction with 3.9% bit rate loss, which indicates that our data-driven approach for video coding has great capacity working in industrial applications. [ABSTRACT FROM AUTHOR]

In the three-dimensional video system, the depth image-based rendering is a key technique for generating synthesized views, which provides audiences with depth perception and interactivity. However, the inaccuracy of depth information leads to geometrical rendering position errors, and the compression distortion of texture and depth videos degrades the quality of the synthesized views. Although existing quality enhancement methods can eliminate the distortions in the synthesized views, their huge computational complexity hinders their applications in real-time multimedia systems. To this end, a residual distillation enhanced network (RDEN)-guided lightweight synthesized view quality enhancement (SVQE) method is proposed to minimize holes and compression distortions in the synthesized views while reducing the model complexity. First, a rethinking on the deep-learning-based SVQE methods is performed. Then, a feature distillation attention block is proposed to effectively reduce the distortions in the synthesized views and make the model fulfill more real-time tasks, which is a lightweight and flexible feature extraction block using an information distillation mechanism and a lightweight multi-scale spatial attention mechanism. Third, a residual feature fusion block is proposed to improve the enhancement performance by using the feature fusion mechanism, which efficiently improves the feature extraction capability without introducing any additional parameters. Experimental results prove that the proposed RDEN efficiently improves the SVQE performance while consuming few computational complexities compared with the state-of-the-art SVQE methods. [ABSTRACT FROM AUTHOR]

The depth video contains large smooth contents with sharp edges. Since the deep learning-based color video orientated intra prediction methods pay no attention to the characteristics of depth video, they are unsuitable for optimizing the coding efficiency of depth video. In this paper, a multiple resolution prediction method with deep up-sampling is proposed to promote the coding efficiency of depth video. To efficiently encode the depth blocks of different complexity, the depth block is selectively encoded at different resolutions, including $\times 1$ , $\times 1$ /2, and $\times 1$ /4 resolutions. If the block is encoded with a low-resolution (LR), the resolution of reconstructed LR depth block is recovered by an up-sampling network. To constrain the quality of both reconstructed high-resolution depth block and its synthesized view, a view synthesis distortion guidance mechanism is proposed for the up-sampling network. In addition, a distillation-based lightweight up-sampling network is proposed to reduce the computational complexity. Experimental results demonstrate that the proposed multiple resolution prediction method obtains an average of 10.84% BD-rate saving in comparison with 3D-HEVC. [ABSTRACT FROM AUTHOR]

This paper proposes a fast mode decision algorithm for 3D High Efficiency Video Coding (3D-HEVC) depth intra coding. In the current 3D-HEVC design, it is observed that for most of the cases, full rate-distortion (RD) cost search of Bi-Partition mode could be skipped since most coding units (CUs) of depth map are very flat or smooth while Bi-Partition modes are designed for CUs with edge or sharp transition. Using the rough RD cost value calculated by HEVC Rough Mode Decision as a selection threshold, we propose a fast Bi-Partition modes selection algorithm to speed up the encoding process. The test result for the proposed fast algorithm reports a 34.4% encoding time saving with a 0.3% bitrate increase on synthesized views for the All-Intra test case and negligible impact under the random access test case. Moreover, by simply varying the selection threshold, we can make a tradeoff between encoding time saving and bitrate loss based on the requirement of different applications. [ABSTRACT FROM AUTHOR]

Multiview video coding (MVC) aims to compress the multiview video through the elimination of video redundancies, where the quality of the reference frame directly affects the compression efficiency. In this paper, we propose a deep virtual reference frame generation method based on a disparity-aware reference frame generation network (DAG-Net) to transform the disparity relationship between different viewpoints and generate a more reliable reference frame. The proposed DAG-Net consists of a multi-level receptive field module, a disparity-aware alignment module, and a fusion reconstruction module. First, a multi-level receptive field module is designed to enlarge the receptive field, and extract the multi-scale deep features of the temporal and inter-view reference frames. Then, a disparity-aware alignment module is proposed to learn the disparity relationship, and perform disparity shift on the inter-view reference frame to align it with the temporal reference frame. Finally, a fusion reconstruction module is utilized to fuse the complementary information and generate a more reliable virtual reference frame. Experiments demonstrate that the proposed reference frame generation method achieves superior performance for multiview video coding. [ABSTRACT FROM AUTHOR]

In video coding, it is a challenge to deal with scenes with complex motions, such as rotation and zooming. Although affine motion compensation (AMC) is employed in Versatile Video Coding (VVC), it is still difficult to handle non-translational motions due to the adopted hand-craft block-based motion compensation. In this paper, we propose a deep affine motion compensation network (DAMC-Net) for inter prediction in video coding to effectively improve the prediction accuracy. To the best of our knowledge, our work is the first attempt to deal with the deformable motion compensation based on CNN in VVC. Specifically, a deformable motion-compensated prediction (DMCP) module is proposed to compensate the current encoding block through a learnable way to estimate accurate motion fields. Meanwhile, the spatial neighboring information and the temporal reference block as well as the initial motion field are fully exploited. By effectively fusing the multi-channel feature maps from DMCP, an attention-based fusion and reconstruction (AFR) module is designed to reconstruct the output block. The proposed DAMC-Net is integrated into VVC and the experimental results demonstrate that the proposed method considerably enhances the coding performance. [ABSTRACT FROM AUTHOR]

In three-dimensional video system, the texture and depth videos are jointly encoded, and then the Depth Image Based Rendering (DIBR) is utilized to realize view synthesis. However, the compression distortion of texture and depth videos, as well as the disocclusion problem in DIBR degrade the visual quality of the synthesized view. To address this problem, a Two-stream Attention Network (TSAN)-based synthesized view quality enhancement method is proposed for 3D-High Efficiency Video Coding (3D-HEVC) in this article. First, the shortcomings of the view synthesis technique and traditional convolutional neural networks are analyzed. Then, based on these analyses, a TSAN with two information extraction streams is proposed for enhancing the quality of the synthesized view, in which the global information extraction stream learns the contextual information, and the local information extraction stream extracts the texture information from the rendered image. Third, a Multi-Scale Residual Attention Block (MSRAB) is proposed, which can efficiently detect features in different scales, and adaptively refine features by considering interdependencies among spatial dimensions. Extensive experimental results show that the proposed synthesized view quality enhancement method achieves significantly better performance than the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Stereoscopic image retargeting aims at converting stereoscopic images to the target resolution adaptively. Different from 2D image retargeting, stereoscopic image retargeting needs to preserve both the shape structure of salient objects and depth consistency of 3D scenes. In this paper, we present a stereoscopic image retargeting method based on deep convolutional neural network to obtain high-quality retargeted images with both object shape preservation and scene depth preservation. First, a cross-attention extraction mechanism is constructed to generate attention map, which contains the valuable attention features of the left and right images and the common attention features between them. Second, since the disparity map can provide accurate depth information of objects in 3D scenes, a disparity-assisted 3D significance map generation module is utilized to further preserve the valuable depth information of stereoscopic images. Finally, in order to predict the retargeted stereoscopic images accurately, an image consistency loss is developed to preserve the geometric structure of salient objects, and a disparity consistency loss is introduced to eliminate depth distortions. Experimental results demonstrate that the proposed deep convolutional neural network can provide favorable stereoscopic image retargeting results. [ABSTRACT FROM AUTHOR]

Three-dimensional (3D) video contains plentiful multi-domain correlations, including spatial, temporal, and inter-view correlations. In this paper, a deep multi-domain prediction method is proposed for 3D video coding. Different from previous methods, our proposed method utilizes not only spatial and temporal correlations but also inter-view correlation to obtain a more accurate prediction, and adopts deep convolutional neural networks to effectively fuse multi-domain references. More specifically, a hierarchical prediction mechanism, which includes a spatial-temporal prediction network and a multi-domain prediction network, is designed to overcome the fusion difficulty of multi-domain reference information. Furthermore, a progressive spatial-temporal prediction network and a multi-scale multi-domain prediction network are designed to obtain the spatial-temporal prediction result and multi-domain prediction result respectively. Experimental results show that the proposed method achieves considerable bitrate saving compared with 3D-HEVC. [ABSTRACT FROM AUTHOR]

Typical high efficiency video coding (HEVC) encoders use rate-distortion optimization (RDO) to select the best coding parameters among numerous candidates to achieve high compression efficiency. However, calculating the rate-distortion cost for each coding parameter entails high computational complexity, which limits the real-time application of these encoders. In this study, time-consuming processes, such as reconstruction and entropy coding, are removed during the calculation of the full RD cost to reduce the complexity of the encoder. Instead, RD costs are estimated for transformed coefficients by using the RD cost model, which is made up of a distortion model based on the distribution features of the transformed coefficients, a residual rate model based on the characteristics of the entropy coding module, and a simple and convenient header rate model. In addition to the full RD cost, typical encoders also adopt fast RD cost, which is composed of the sum of absolute transform differences (SATD) and header bit. To further reduce encoding complexity, three simplified SATD (SSATD) methods based on DCT and DST features are proposed to serve as substitutes for SATD in fast RD cost, thus reducing the computational complexity of SATD calculation. Experimental results show that 35.53%, 14.57%, and 14.37% of coding time can be reduced respectively under AI, RA and LDP configurations with negligible RD performance loss when applying the proposed RD cost model and the SSATD methods to HM16.7. Besides, the proposed methods also can reduce 4%–9% of encoding time and exhibits the same RD performance as state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Stereoscopic image retargeting aims to manipulate the stereoscopic images to fit various devices with different resolutions and prescribed aspect ratios. With the development of various types of three-dimensional (3D) displays, stereoscopic image retargeting becomes increasingly popular in the field of computer graphics. In this paper, we propose an unsupervised stereoscopic image retargeting network (USIR-Net) to address the problem of stereoscopic image retargeting without label information. By exploring the inter-view correlation and disparity relationship of stereoscopic images, two unsupervised losses are developed to guide the learning of stereoscopic image retargeting model. First, in view of the inter-view correlation, a view synthesis loss is proposed to guarantee the generation of high-quality stereoscopic images with accurate inter-view relationship. Second, by exploiting the consistency of stereoscopic images before and after the retargeting, a stereo cycle consistency loss, which consists of a content consistency term and a disparity consistency term, is developed to preserve the structure information and prevent binocular disparity inconsistency. Quantitative and qualitative experimental results demonstrate that the proposed method achieves superior performance compared with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

• The current 3D human body reconstruction technology requires a large number of 3D scanning equipment and a complex dynamic capture process, which significantly limits its rapid adoption in these application domains. The purpose of this article is to simplify the process of 3D reconstruction. • By combining deep learning methods, an end-to-end 3D human body reconstruction framework is proposed to achieve the goal of reconstructing a high-precision 3D human body model based on RGB images. [Display omitted] In recent years, with the improvement of artificial intelligence technology, it has become possible to reconstruct high-precision 3D human body models based on ordinary RGB images. The current 3D human body reconstruction technology requires complex external equipment to scan all angles of the human body, which is complicated to be implemented and cannot be popularized. In order to solve this problem, this paper applies deep learning models on reconstructing 3D human body based on monocular images. First of all, this paper uses Stacked Hourglass network to perform convolution operations on monocular images collected from different views. Then Multi-Layer Perceptrons (MLPs) are used to decode the encoded high-level images. The feature codes in the two views(main and side) are fused, and the interior and exterior points are classified by the fusion features, so as to obtain the corresponding 3D occupancy field. At last, the Marching Cube algorithm is used for 3D reconstruction with a specific threshold and then we use Laplace smoothing algorithm to remove artifacts. This paper proposes a dense sampling strategy based on the important joint points of the human body, which has a certain optimization effect on the realization of high-precision 3D reconstruction. The performance of the proposed scheme has been validated on the open source datasets, MGN dataset and the THuman dataset, provided by Tsinghua University. The proposed scheme can reconstruct features such as clothing folds, color textures, and facial details,and has great potential to be applied in different applications. [ABSTRACT FROM AUTHOR]

Video surveillance and its applications have become increasingly ubiquitous in modern daily life. In video surveillance system, video coding as a critical enabling technology determines the effective transmission and storage of surveillance videos. In order to meet the real-time or time-critical transmission requirements of video surveillance systems, the low-delay (LD) configuration of the advanced high efficiency video coding (HEVC) standard is usually used to encode surveillance videos. The coding efficiency of the LD configuration is closely related to the quantization parameter (QP) cascading technique which selects or determines the QPs for encoding. However, the quantization parameter cascading (QPC) technique currently adopted for the LD configuration in HEVC test model (i.e., HM) is not optimized since it has not taken full account of the reference dependency in coding. In this paper, an efficient QPC technique for surveillance video coding, referred to as QPC-SV, is proposed, considering all inter reference frames under the LD configuration. Experimental results demonstrate the efficacy of the proposed QPC-SV. Compared with the default configuration of QPC in the HM, the QPC-SV achieves significant rate-distortion performance gain with average BD-rates of −9.35% and −9.76% for the LDP and LDB configurations, respectively. [ABSTRACT FROM AUTHOR]

Hyperspectral image (HSI) clustering is a challenging task due to the complex characteristics in HSI data, such as spatial-spectral structure, high-dimension, and large spectral variability. In this paper, we propose a novel deep spatial-spectral subspace clustering network (DS3C-Net), which explores spatial-spectral information via the multi-scale auto-encoder and collaborative constraint. Considering the structure correlations of HSI, the multi-scale auto-encoder is first designed to extract spatial-spectral features with different-scale pixel blocks which are selected as the inputs. Then, the collaborative constrained self-expressive layers are introduced between the encoder and decoder, to capture the self-expressive subspace structures. By designing a self-expressiveness similarity constraint, the proposed network is trained collaboratively, and the affinity matrices of the feature representation are learned in an end-to-end manner. Based on the affinity matrices, the spectral clustering algorithm is utilized to obtain the final HSI clustering result. Experimental results on three widely used hyperspectral image datasets demonstrate that the proposed method outperforms state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Discrete cosine transform (DCT) is an indispensable module in video codecs and is a major part in many video coding standards including the latest high efficiency video coding (HEVC). As the video resolution increases, both transform sizes and the number of transforms increase continuously which poses challenges to the reusability design especially in hardware implementation. This paper presents reconfigurable transform architecture to flexibly support the reusability of different transform sizes. The proposed architecture maximally reuses the hardware resources by rearranging the order of input data for different transform sizes while still exploiting the butterfly property. Furthermore, this architecture supports reconfigurable throughput according to different hardware resource requirements. By applying the proposed architecture to the field-programmable gate array (FPGA) design of HEVC core transform matrices, the synthesis results show much lower consumption of hardware resources comparing to existing methods in the literature. The implementation in Altera’s Stratix III FPGA can operate at 139 MHz and supports real-time processing of $3840\times 2160$ ultra-high definition video at a minimum of 45 f/s and up to 359 f/s for different DCT sizes. [ABSTRACT FROM AUTHOR]

With proliferation of social media, image has become ubiquitous giving rise to the demand and importance of image semantic analysis and retrieval to access information quickly on social media. However, even with humongous information available, there are certain categories of images which are important for certain applications but are very scarce. Convolutional neural network is an effective method to extract high-level semantic features for image database retrieval. To overcome the problem of over-fitting when the number of training samples in dataset is limited, this paper proposes an image database retrieval algorithm based on the framework of transfer learning and feature fusion. Based on the fine-tuning of the pre-trained Convolutional Neural Network (CNN), the proposed algorithm first extracts the semantic features of the images. Principal Component Analysis (PCA) is then applied for dimension reduction and to reduce the computational complexity. Last, the semantic feature extracted from the CNN is fused with traditional low-level visual feature to improve the retrieval accuracy further. Experimental results demonstrated the effectiveness of the proposed method for image database retrieval. [ABSTRACT FROM AUTHOR]

In this paper, a novel rate-control algorithm based on the region adaptive R- $\lambda $ model is proposed for depth maps coding. First, in order to obtain an accurate rate control for depth maps coding, a modified frame level bit allocation method based on coding bits statistical distribution of depth maps is proposed. Second, considering that different areas in a depth map have an imparity effect on virtual view rendering, the blocks of the depth map are divided into two types, namely, interested blocks for virtual view rending (IBV) and noninterested blocks for virtual view rending (NIBV). Then, two different R- $\lambda $ models are derived for IBV and NIBV, respectively. The optimal bitrates for IBV and NIBV are determined by solving an optimization problem. After that, based on the regional R- $\lambda $ models, the optimal Lagrange multipliers are calculated for both IBV and NIBV. Finally, the largest coding unit (LCU) level rate control is performed by adaptively adjusting the Lagrange multiplier to avoid blocking artifacts and smooth the quality of coding. Experimental results demonstrate that the proposed method can achieve considerable BD-PSNR gains compared with the unified rate-quantization model and conventional R- $\lambda $ model-based algorithms in terms of rendered virtual views quality. [ABSTRACT FROM AUTHOR]

The 3D extension of High Efficiency Video Coding significantly improves the coding efficiency of 3D video at the expense of computational complexity. This paper presents a novel fast mode decision algorithm for depth map coding based on the grayscale similarity and inter-view correlation. First, depth map grayscale similarity is adopted to judge whether the reference frame could assist the coding of the current frame. When the difference in the average grayscale between the co-located coding unit (CU) and the current CU is smaller than the similarity threshold, the depth level of the current CU will be restricted by that of the coded reference CU. Second, the grayscale similarity and inter-view correlation are jointly used for dependent views to achieve early decision on the best prediction unit (PU) mode. The mode decision procedure will be determined early when the co-located CU, which has a grayscale similarity with the current CU, selects Merge or Inter $2N \times 2N$ as the best prediction mode. Moreover, when the corresponding CU in the independent view selects Merge or Inter $2N \times 2N$ as the best prediction mode, the current CU will skip other PU modes checking based on the strong inter-view correlation. Finally, different strategies are proposed for the P-frames and B-frames of dependent views in view of the characteristics of different prediction structures. For B frames, the PU mode information of the coded independent view is utilized as reference to skip the unnecessary mode decision processes. For P frames, the spatial–temporal correlation is considered in the process of early mode decision to determine whether to choose the Merge mode or Inter $2N \times 2N$ as the best mode. Experimental results show that our proposed scheme achieves considerable time saving with negligible degradation of coding performance. [ABSTRACT FROM PUBLISHER]

In this paper, we put forward an effective and efficient no reference image blurriness assessment metric on the basis of local binary pattern (LBP) features. In this proposal, we reveal that part of the LBP histogram bins present monotonously with the degree of blurriness. The proposed method contains the following steps. Firstly, the LBP maps of an input image are extracted with multiple radiuses. And then, the frequency of pattern histogram is analyzed before part of bins are chosen as the features. In addition, we also take the entropy of these bins as another feature. Finally, we learn the extracted features to predict the image blurriness score. Validation of the proposed method is conducted on the blurred images of LIVE-II, CSIQ, TID2008, TID2013, LIVE3D IQA Phase I and LIVE3D IQA Phase II. Experimental results demonstrate that compared with the state-of-the-art image quality assessment (IQA) methods, the proposed algorithm has notable advantage in correlation with subjective perception and computational complexity. [ABSTRACT FROM AUTHOR]

Traditional image editing techniques cannot be directly used to process stereoscopic media, as extra constraints are required to ensure consistent changes between the left and right images. In this paper, we propose a hybrid warping model for stereoscopic image stitching by combining projective and content-preserving warping. First, a uniform homography algorithm is proposed to prewarp the left and right images, and thus ensure consistent changes. Second, a content-preserving warping is introduced to locally refine alignment and reduce vertical disparities. Finally, a seam-cutting-based algorithm is used to find a blending seam, and the multiband blending algorithm is used to produce the final stitched image. Experimental results show that the proposed method can effectively stitch stereoscopic images, which not only avoids local distortions, but also reduces vertical disparities reasonably. [ABSTRACT FROM PUBLISHER]

Accurate and high-quality depth maps are required in lots of 3D applications, such as multi-view rendering, 3D reconstruction and 3DTV. However, the resolution of captured depth image is much lower than that of its corresponding color image, which affects its application performance. In this paper, we propose a novel depth map super-resolution (SR) method by taking view synthesis quality into account. The proposed approach mainly includes two technical contributions. First, since the captured low-resolution (LR) depth map may be corrupted by noise and occlusion, we propose a credibility based multi-view depth maps fusion strategy, which considers the view synthesis quality and interview correlation, to refine the LR depth map. Second, we propose a view synthesis quality based trilateral depth-map up-sampling method, which considers depth smoothness, texture similarity and view synthesis quality in the up-sampling filter. Experimental results demonstrate that the proposed method outperforms state-of-the-art depth SR methods for both super-resolved depth maps and synthesized views. Furthermore, the proposed method is robust to noise and achieves promising results under noise-corruption conditions. [ABSTRACT FROM AUTHOR]

This paper presents a novel depth video estimation method, which improves estimation property and refines the temporal consistency and spatial accuracy at the same time. The main idea is to incorporate more useful motion and structure information into depth estimation. First, an adaptive weight is calculated based on the motion information of adjacent frames and attached to a temporal term to update the energy function, thus reducing the matching errors and improving temporal consistency. Second, the depth continuity/discontinuity in spatial domain is properly judged by combining the edges of initial depth map and color segmentation, and used in the refinement strategy. Finally, we evaluate the performance of our algorithm with several public stereoscopic video sequences. Experimental results show that the qualities of depth video and synthesized virtual view are all improved. [ABSTRACT FROM PUBLISHER]

The field of video coding has been exploring the compact representation of video data, where perceptual redundancies in addition to signal redundancies are removed for higher compression. Many research efforts have been dedicated to modeling the human visual system's characteristics. The resulting models have been integrated into video coding frameworks in different ways. Among them, coding enhancements with the just noticeable distortion (JND) model have drawn much attention in recent years due to its significant gains. A common application of the JND model is the adjustment of quantization by a multiplying factor corresponding to the JND threshold. In this paper, we propose an alternative perceptual video coding method to improve upon the current H.264/advanced video control (AVC) framework based on an independent JND-directed suppression tool. This new tool is capable of finely tuning the quantization using a JND-normalized error model. To make full use of this new rate distortion adjustment component the Lagrange multiplier for rate distortion optimization is derived in terms of the equivalent distortion. Because the H.264/AVC integer discrete cosine transform (DCT) is different from classic DCT, on which state-of-the-art JND models are computed, we analytically derive a JND mapping formula between the integer DCT domain and the classic DCT domain which permits us to reuse the JND models in a more natural way. In addition, the JND threshold can be refined by adopting a saliency algorithm in the coding framework and we reduce the complexity of the JND computation by reusing the motion estimation of the encoder. Another benefit of the proposed scheme is that it remains fully compliant with the existing H.264/AVC standard. Subjective experimental results show that significant bit saving can be obtained using our method while maintaining a similar visual quality to the traditional H.264/AVC coded video. [ABSTRACT FROM PUBLISHER]

• A long-term quality model for rate control in concurrent MVT is proposed. • Lyapunov-based dynamic programming algorithm is developed to solve our problem. • Long-term video quality is improved without reducing the delay performance. Concurrent multipath transmission provides an effective solution for streaming high-quality mobile videos in heterogeneous wireless networks. Rate control is commonly adopted in multimedia communication systems to fully utilize the available network bandwidth. This paper proposes a novel rate control for concurrent multipath video transmission. The existing rate control algorithms mainly adapt bit rate in the short-term pattern, i.e., without considering the long-term video transmission quality. We propose a long-term rate control scheme that takes into account the status of both the transmission buffer and video frames. First, a mathematical model is developed to formulate the non-convex problem of long-term quality maximization. Second, we develop a dynamic programming solution for online encoding bit rate control based on buffer status. The performance evaluation is conducted in a real test bed over LTE and Wi-Fi networks. Experimental results demonstrate that the proposed long-term rate control scheme achieves appreciable improvements over the short-term rate control schemes in terms of video quality and delay performance. [ABSTRACT FROM AUTHOR]

• To balance reconstruction quality with computational complexity, we introduce a structural group sparsity model for use in the initial reconstruction phase and propose a weight-based group sparse optimization algorithm acting in joint domains. • Then, a coarse-to-fine optical flow estimation model with successive approximation is introduced for use in the interframe prediction stage to recover non-key frames through alternating optical flow estimation and residual sparse reconstruction. • The experimental results demonstrate that the proposed algorithm outperforms the state-of-the-art methods in both objective and subjective reconstruction quality. The existing video compressed sensing (CS) algorithms for inconsistent sampling ignore the joint correlations of video signals in space and time, and their reconstruction quality and speed need further improvement. To balance reconstruction quality with computational complexity, we introduce a structural group sparsity model for use in the initial reconstruction phase and propose a weight-based group sparse optimization algorithm acting in joint domains. Then, a coarse-to-fine optical flow estimation model with successive approximation is introduced for use in the interframe prediction stage to recover non-key frames through alternating optical flow estimation and residual sparse reconstruction. Experimental results show that, compared with the existing algorithms, the proposed algorithm achieves a peak signal-to-noise ratio gain of 1–3 dB and a multi-scale structural similarity gain of 0.01–0.03 at a low time complexity, and the reconstructed frames not only have good edge contours but also retain textural details. [ABSTRACT FROM AUTHOR]