Search

Rate control typically involves two steps: bit allocation and bitrate control. The bit allocation step can be implemented in various fashions depending on how many levels of allocation are desired and whether or not an optimal rate–distortion (R-D) performance is pursued. The bitrate control step has a simple aim in achieving the target bitrate as precisely as possible. In our recent research, we have developed a \lambda -domain rate control algorithm that is capable of controlling the bitrate precisely for High Efficiency Video Coding (HEVC). The initial research showed that the bitrate control in the \lambda -domain can be more precise than the conventional schemes. However, the simple bit allocation scheme adopted in this initial research is unable to achieve an optimal R-D performance reflecting the inherent R-D characteristics governed by the video content. In order to achieve an optimal R-D performance, the bit allocation algorithms need to be developed taking into account the video content of a given sequence. The key issue in deriving the video-content-guided optimal bit allocation algorithm is to build a suitable R-D model to characterize the R-D behavior of the video content. In this paper, to complement the R- \lambda model developed in our initial work, a D- \lambda model is properly constructed to complete a comprehensive framework of \lambda -domain R-D analysis. Based on this comprehensive \lambda -domain R-D analysis framework, a suite of optimal bit allocation algorithms are developed. In particular, we design both picture-level and basic-unit-level bit allocation algorithms based on the fundamental R-D optimization theory to take full advantage of the content-guided principles. The proposed algorithms are implemented in HEVC reference software, and the experimental results demonstrate that they can achieve an obvious R-D performance improvement with a smaller bitrate control error. The proposed bit allocation algorithms have already been adopted by the Joint Collaborative Team on Video Coding and integrated into the HEVC reference software. [ABSTRACT FROM PUBLISHER]

Cloud gaming has emerged as a promising application to enable high-end game playing with thin clients. Transmission control protocol (TCP) is pervasively adopted as the transport-layer protocol in the mainstream cloud gaming systems for video communication. However, streaming mobile cloud gaming video using the TCP is challenged with several key technical barriers: 1) the performance limitations of wireless networks in bandwidth and reliability; 2) the high throughput demand and stringent delay constraint imposed by high-quality gaming video transmission; and 3) the deadline violations and throughput fluctuations caused by the packet retransmission and congestion control mechanisms in the TCP. To address these critical problems, this paper proposes an application-layer source–forward error correction (FEC) coding framework dubbed adaptive source–FEC coding over TCP (ESCOT). First, we analytically formulate the optimization problem of joint source–FEC coding to minimize the end-to-end distortion of real-time video communication over TCP. Second, we develop a heuristic solution for effective loss rate approximation, source rate control, and FEC coding adaptation. ESCOT is distinct from existing source–FEC coding schemes in proactively analyzing and leveraging the TCP characteristics. The proposed solution is able to effectively mitigate both consecutive and sporadic video frame drops caused by congestion and random packet losses. We conduct the performance evaluation through extensive emulations in the Exata platform using real-time gaming video encoded by the H.264 codec. Experimental results show that the ESCOT advances the state of the art with noticeable improvements in video peak signal-to-noise ratio, end-to-end delay, goodput, and frame success rate. [ABSTRACT FROM PUBLISHER]

Light field (LF) technology has been popularly adopted by a wide range of conventional industries. However, one problem when dealing with LFs is the sheer size of data volume. There have been many multi-view video coding (MVC)-based LF video coding methods reported in the literature, aiming at finding the best prediction structure for LF video coding. It is clear that the number of possible prediction structures is unlimited, and it is also observed that the coding bit-rate can be reduced by increasing the number of bi-directionally encoded views in the prediction structure. However, none work has been conducted to analyze the relationship of the prediction structure with its coding performance. In light of this observation, we first design a new LF-MVC prediction structure by extending the inter-view prediction into a two-directional parallel structure. Analytical models for source coding rate and encoding time are developed to analyze their relationships with the prediction structure, and are proven to be well-matched to our experimental results. Experimental evaluation of two LF video sequences demonstrates that the proposed LF-MVC prediction structure can achieve a factor of 26% bit-rate reduction against the conventional MVC prediction structure for an LF video with $5\times 5$ views, and a further 34% bit-rate reduction for an LF video with a larger $10\times 10$ views. Compared with the state-of-the-art MVC-based LF video coding prediction structures in the literature, LF-MVC can achieve the best coding performance, and with its high encoding efficiency, is well suited for deployment in practical LF-based 3D systems. [ABSTRACT FROM PUBLISHER]

High-frame-rate (HFR) video technology is becoming widely implemented in popular multimedia applications (e.g., Youtube and cloud gaming) to provide a smooth viewing experience, while transmission control protocol (TCP) is pervasively adopted as the transport-layer solution for video communications to achieve firewall traversal and network friendliness. However, it is severely challenging to effectively deliver real-time HFR video over TCP: 1) HFR video streaming features high transmission rate, enhanced frame density, and stringent delay constraint and 2) the packet retransmission and congestion control mechanisms in TCP may cause frequent throughput fluctuations and deadline violations. Motivated by addressing these critical issues, this research presents an application-layer forward error correction (FEC) framework dubbed Raptor coded HFR video over TCP (ROCHET). First, we develop a mathematical model to analyze the frame-level distortion of systematic Raptor code-based HFR video communication over TCP in wireless networks. Second, we propose a joint approximate distortion estimation and Raptor coding adaption solution to minimize the sum of total distortion. The proposed ROCHET is able to effectively leverage unequal error protection and TCP state analysis to enhance streaming video quality. We conduct the performance evaluation through extensive emulations in Exata involving real-time HFR video encoded with H.264 codec. Compared with the existing FEC coding schemes, ROCHET achieves appreciable improvements in terms of video peak signal-to-noise ratio, goodput, and frame success rate. Thus, ROCHET is recommended for TCP-based HFR video transmission over wireless networks. [ABSTRACT FROM PUBLISHER]

The state-of-the-art High Efficiency Video Coding (HEVC) standard adopts a hierarchical coding structure to improve its coding efficiency. This allows for the quantization parameter cascading (QPC) scheme that assigns quantization parameters (Qps) to different hierarchical layers in order to further improve the rate-distortion (RD) performance. However, only static QPC schemes have been suggested in HEVC test model, which are unable to fully explore the potentials of QPC. In this paper, we propose an adaptive QPC scheme for an HEVC hierarchical structure to code natural video sequences characterized by diversified textures, motions, and encoder configurations. We formulate the adaptive QPC scheme as a non-linear programming problem and solve it in a scientifically sound way with a manageable low computational overhead. The proposed model addresses a generic Qp assignment problem of video coding. Therefore, it also applies to group-of-picture-level, frame-level and coding unit-level Qp assignments. Comprehensive experiments have demonstrated that the proposed QPC scheme is able to adapt quickly to different video contents and coding configurations while achieving noticeable RD performance enhancement over all static and adaptive QPC schemes under comparison as well as HEVC default frame-level rate control. We have also made valuable observations on the distributions of adaptive QPC sets in the videos of different types of contents, which provide useful insights on how to further improve static QPC schemes. [ABSTRACT FROM PUBLISHER]

We propose an adaptive hybrid digital–analog video transmission (A-HDAVT) scheme for robust video streaming in mobile networks with realistic fading channels. This scheme is fundamentally different from recent research in hybrid digital–analog video transmission in which wireless channels are unrealistically assumed to be Gaussian. For fading channels, it is critical to take full advantage of diversity in both video contents and multiuser channels. Like all hybrid approaches, A-HDAVT is designed to exploit the benefits from both digital and analog systems. To achieve this goal, each group of pictures is first transformed into one low-pass frame and several high-pass frames with motion-compensated temporal filtering. The critical low-pass frame is reliably transmitted as base layer in a digital mode, while high-pass frames are transmitted as enhancement layers in an analog mode to achieve desired graceful degradation performance. In analog transmission, we introduce a channel prediction-based adaptive power-distortion optimization (P-APDO) scheme to combat channel fading in mobile networks. The basic idea behind P-APDO is to perform power allocation based on the video content as well as the predicted channel status. Furthermore, we also investigate the multiuser scenarios in which the content diversity and channel diversity among users are appropriately exploited. Extensive simulations have been carried out to evaluate the performance of A-HDAVT under various degrees of channel fading. The results show that A-HDAVT achieves significant performance gains over competing schemes Robust Uncoded Video Transmission, Parcast, and Softcast in both single-user and multiuser scenarios. [ABSTRACT FROM PUBLISHER]

In contemporary multiuser multiple-input multiple-output systems, it is common for the reception devices to have a varying number of antennas. When multicast is performed, the number of concurrent spatial streams is limited by the device with the least number of antennas, which prevents more capable devices from getting higher rates. In this paper, we address the antenna heterogeneity in wireless video multicast by the innovative design of multiple similar description (MSD) video coding and multiplexed space–time block coding (M-STBC). MSD coding generates multiple descriptions of a video and features that any linear combinations of the descriptions are decodable. The descriptions comprising of real numbers are further processed by transform and power allocation steps for efficient transmission in a power-constrained system. M-STBC puts symbols in similar descriptions to corresponding space–time positions and ensures decodability under any antenna settings and channel conditions. As a result, we build up a scalable video multicast system, named AirScale, which allows receivers with a various number of antennas to decode from a single transmission, and the reconstructed video quality improves with the number of equipped antennas. Evaluations on Sora shows that, in a \1,2,3,4\ \times 4 system, AirScale provides baseline quality for one-antenna receiver and a much higher quality for multiantenna receivers. The gain over SoftCast is up to 3.5, 3.9, and 4.1 dB for two-, three-, and four-antenna receivers, respectively. [ABSTRACT FROM PUBLISHER]

High frame rate (HFR) video is emerging as a new paradigm in popular multimedia applications (e.g., cloud gaming) to achieve smooth viewing experience perceived by end-users. In the context of HFR streaming video, end-to-end distortion and sending frame rate are equally important to the perceptual quality. This study presents a modeling-based approach to optimizing the HFR video transmission over wireless networks. First, we develop an analytical model dubbed FRIED (Frame Rate versus vIdEo Distortion) to characterize the tradeoff between sending frame rate and end-to-end video distortion. Second, we propose a Joint frAme Selection and FEC (Forward Error Correction) cOding (JASCO) approach based on the FRIED model to optimize the transmission performance. The efficacy of the proposed JASCO is evaluated through extensive semi-physical emulations in Exata involving H.264 video streaming. Experimental results show that JASCO outperforms the reference approaches in improving video peak signal-to-noise ratio (PSNR) at the same frame rate. Or conversely, JASCO is able to achieve higher received frame rate while guaranteeing the same video PSNR. [ABSTRACT FROM PUBLISHER]

High-frame-rate (HFR) video is emerging in popular gaming applications to enhance the smooth experience perceived by end users. However, it is challenging to guarantee the delivery quality of HFR video in mobile cloud gaming scenarios because of the high transmission rate and limited wireless resources. To address this critical problem, we develop a novel transmission scheduling framework dubbed AdaPtive HFR vIdeo Streaming (APHIS). The term adaptive indicates this scheme’s capability in dynamically adjusting the video traffic load and forward error correction (FEC) coding. First, we propose an online video frame selection algorithm to minimize the total distortion based on the network status, input video data, and delay constraint. Second, we introduce an unequal FEC coding scheme to provide differentiated protection for Intra (I) and Predicted (P) frames with low-latency cost. The proposed APHIS framework is able to appropriately filter video frames and adjust data protection levels to optimize the quality of HFR video streaming. We conduct extensive emulations in Exata involving HFR video encoded with H.264 codec. Experimental results show that APHIS outperforms the reference transmission schemes in terms of video peak signal-to-noise ratio, end-to-end delay, and goodput. Therefore, we recommend APHIS for delivering HFR video streaming in mobile cloud gaming systems. [ABSTRACT FROM AUTHOR]

We propose an improved coarse-grain scalable video coding (SVC) approach based on the structural similarity (SSIM) index as the visual quality criterion, aiming at maximizing the overall coding performance constrained by user-defined quality weightings for all scalable layers. First, we develop an interlayer rate-SSIM dependency model, by investigating bit rate and SSIM relationships between different layers. Second, a reduced-reference SSIM-Q model and a Laplacian R-Q model are introduced for SVC, by incorporating the characteristics of hierarchical prediction structure in each layer. Third, based on the user-defined weightings and the proposed models, we design a rate-distortion optimization approach to adaptively adjust Lagrange multipliers for all layers to maximize the overall rate-SSIM performance of the scalable encoder. Experiments with multiple layers, different layer weightings, and various videos demonstrate that the proposed framework can achieve better rate-SSIM performance than single layer optimization method, and provide better coding efficiency as compared to the conventional SVC scheme. Subjective tests further demonstrate the benefits of the proposed scheme. [ABSTRACT FROM AUTHOR]

Video transmission using Scalable Video Coding (SVC) enables the functionalities of adaptation for both channel bandwidth and terminal device. However, link adaptation, which requires the video bitstream to provide error resilient scalability (ERS), is currently not supported in SVC. In this paper, we propose a robust SVC bitstream transmission framework, which integrates both error resilient (ER) coding and error concealment (EC) to enable the desired link adaptability. First, at the encoder, we design an ER coding scheme for SVC using redundant pictures, in which redundant picture information (RPI) is generated under rate-distortion criteria and transmitted to the media gateway together with the SVC bitstream. Then, ERS can be effectively achieved at the media gateway by selectively adding or removing coded pictures of different video coding layers according to the RPI and current network link status. Finally, at the decoder, we take advantage of the unique characteristics of SVC to develop an EC scheme based on both the proposed Virtual-BLSkip method and Wiener filtering to recover the lost or removed enhancement layer pictures. The proposed framework has negligible impact on the coding efficiency because it transmits RPIs rather than directly adding redundancy into the original bitstream. More importantly, it successfully strikes a balance between coding efficiency, error resiliency, and operation complexity. It is capable of providing the desired ERS with extremely low complexity, and maintaining virtually the same bit rate as input video bitstream. The experimental results show that the proposed framework achieves significant performance gains over existing ER transmission schemes. [ABSTRACT FROM PUBLISHER]

This research addresses the problem of robust linear video transmission over the Rayleigh fading channel, where only statistical channel state information (CSI) is available to the sender. We observe that discarding low-priority (LP) data and saving the channel uses for high-priority (HP) data can significantly improve the quality of the received video. We formulate an optimization problem that aims to minimize the total squared error of a multi-variant Gaussian random vector under the given bandwidth and power resources. To tame the complexity of this NP-hard problem, we analyze two sub-problems, namely power allocation and bandwidth allocation, and propose an iterative algorithm to approximate the solution. Subsequently, we propose a one-pass two-step fast algorithm that further reduces both algorithmic and computational complexity. A linear video transmission system is implemented based on the proposed algorithm. Simulations show that our system significantly outperforms SoftCast, and the PSNR gain at 5th percentile of 1000 test runs is between 4.0 dB and 7.5 dB under varying noise levels. [ABSTRACT FROM PUBLISHER]

Within the existing reconstruction process of distributed video coding (DVC), there are two major approaches: the maximum probability reconstruction and the minimum mean square error (MMSE) reconstruction. Both of them assume that each node, a pixel in pixel domain DVC or a coefficient in transform domain DVC, is i.i.d., and reconstruct the value of each node independently by only exploiting statistical correlation between source and side-information. These kinds of models produce considerable amount of artifacts in decoded Wyner–Ziv (WZ) frames and degrade the objective performance. In this paper, we propose a context-adaptive Markov random field (MRF) reconstruction algorithm which exploits both the statistical correlation and the spatio-temporal consistency by modeling the corresponding MRF of a generic DVC architecture, and solve the inference by finding its MRF-based maximum a posteriori (MAP) estimate. The energy function of the MRF model consists of two terms: a data term measuring the statistical correlation, and a geometric regularity term enforcing local spatio-temporal structure consistency which is modeled by optical flow estimation with regard to the critical parameters under a wide variety of DVC scenarios. In case the unreliability of the derived local structure, a confidence parameter is introduced to prevent inappropriate penalizing. To find the reconstructed patch assignment with the largest expected probability in the context-adaptive MRF, the energy minimization for the MRF-based MAP estimate of the WZ frames is solved by global optimization and greedy strategies. Compared to the existing maximum probability and MMSE reconstruction with i.i.d. model, a better subjective and objective performance is validated by extensive experiments. [ABSTRACT FROM AUTHOR]

With the development of 3D display and interactive multimedia systems, new 3D video applications, such as 3DTV and Free Viewpoint Video, are attracting significant interests. In order to enable these new applications, new data formats including captured 2D video sequences and corresponding depth maps have been proposed. Compared to conventional video frames, depth maps have very different characteristics. First, they typically consist of homogeneous areas partitioned by sharp edges representing depth discontinuities, while the sharp discontinuities play very important roles in view rendering. Second, there exists structure similarity between depth map and corresponding video, in which the edges in depth exhibit quite similar behaviors as the edges in the corresponding video. In conventional video coding techniques with transforms followed by quantization, there usually exist large artifacts along sharp edges and it costs significant more bits to represent the edges with higher accuracy. In order to suppress the coding artifacts while preserving edges, and to better compress depth information, we propose in this paper two new depth compression techniques: Trilateral Filter and Sparse Dyadic Mode. Both techniques utilize the structure similarity between depth and corresponding video while focusing on different aspects in depth compression. As a new in-loop filter, Trilateral Filter is designed to filter depth with coding artifacts based on the proximity of pixel positions, the similarity among depth samples, as well as the similarity among the collocated pixels in the video frame. While Sparse Dyadic Mode is used as an intra mode to reconstruct depth map with sparse representations of depth blocks and effective reference of edge information from video frames. With these two new coding tools, we can achieve up to about 1.5 dB gain on rendering quality as compared to depth sequences coded using MVC under the same coding rate. [ABSTRACT FROM AUTHOR]

In this paper, we propose a generic error resilient video coding (ERVC) scheme using embedded Wyner–Ziv (WZ) description. At the encoder side, a joint source-channel R-D optimized mode selection (JSC-RDO-MS) algorithm with WZ-coded anchor frames is statistically studied and developed. Given a stationary first-order Markov Gaussian source, the proposed mode optimization is justified by an analysis of the RD impact on the WZ bit-rate. JSC-RDO-MS involves in the estimation of expected rate and distortion of WZ coding with the unavailable side information, and the WZ bit-rate of each coding mode is determined based on the error correction capability of the specific WZ codec. At the decoder side, an online correlation noise model between the source and the side-information is proposed with a mixture of Laplacians whose parameters are attained to reflect the coherence of the motion field of successive frames and the energy of prediction residual. Each mixture component represents the statistical distribution of prediction residuals, and the mixing coefficients represent the amount of errors in motion compensation. The proposed scheme achieves the so-called classification gain by exploiting the spatially non-stationary characteristics of the motion field and texture. Extensive experimental results show that the proposed WZ-ERVC scheme achieves a better overall RD performance than existing ERVC schemes, and the proposed modeling algorithm also significantly outperforms the conventional Laplacian model by up to 2 dB. [ABSTRACT FROM PUBLISHER]

This paper studies a joint coding/routing optimization between network lifetime and video distortion by applying information theory to wireless visual sensor networks for correlated sources. Arbitrary coding [distributed video coding and network coding (NC)] from both combinatorial optimization and information theory could make significant progress toward the performance limit and tractable. Also, multipath routing can spread energy utilization across nodes within the entire network to keep a potentially longer lifetime, and solve the wireless contention issues by the splitting traffic. The objective function not only keeps the total energy consumption of encoding power, transmission power, and reception power minimized, but ensures the information received by sink nodes to approximately reconstruct the visual field. Also, a generalized power consumption model for distributed video sources is developed, in which the coding complexity of Key frames and Wyner-Ziv frames is measured by translating specific coding behavior into energy consumption. On the basis of the distributed multiview video coding and NC-based multipath routing, the balance problem between lifetime (costs) and distortion (capacity) is modeled as an optimization formulation with a fully distributed solution. Through a primal decomposition, a two-level optimization is relaxed with Lagrangian dualization and solved by the gradient algorithm. The low-level optimization problem is further decomposed into a secondary master dual problem with four cross-layer subproblems: a rate control problem, a channel contention problem, a distortion control problem, and an energy conservation problem. The implementation of the distributed algorithm is discussed with regard to the communication overhead and dynamic network change. Simulation results validate the convergence and performance of the proposed algorithm. [ABSTRACT FROM AUTHOR]

An optimal joint source-channel coding (OJSCC) scheme is developed for encoding memoryless generalized Gaussian distribution (GGD) sources and transmission over noisy channels, including memoryless channels and bursty channels. In particular, we are able to incorporate both transition probability and bit error rate of a bursty channel model into an end-to-end rate-distortion (R-D) function. This R-D function results in an optimum tradeoff between source coding accuracy and channel error protection under a fixed transmission rate. Based on the results of OJSCC, we further proposed a robust joint source-channel coding (RJSCC) scheme based on a combination of OJSCC with allpass filtering source reshaping. Experimental results based on a broad class of GGD sources suggest that the RJSCC scheme consistently outperforms the OJSCC scheme when the shape factor, a parameter of GGD, is less than 2.0. The comparison of RJSCC with the state-of-the-art robust quantization (RQ) [IEEE Trans. Image Process. 7 (1998) 496-505] shows that RJSCC is comparable with RQ at low bit rate and significantly better than RQ at high bit rate. In addition, an image transmission system using RJSCC demonstrates that the RJSCC system can be efficiently employed in practical applications. [ABSTRACT FROM AUTHOR]