Search

We propose a novel two-stage training strategy with ambiguity boosting for the self-supervised learning of single view depths from stereo images. Our proposed two-stage learning strategy first aims to obtain a coarse depth prior by training an auto-encoder network for a stereoscopic view synthesis task. This prior knowledge is then boosted and used to self-supervise the model in the second stage of training in our novel ambiguity boosting loss. Our ambiguity boosting loss is a confidence-guided type of data augmentation loss that improves the accuracy and consistency of generated depth maps under several transformations of the single-image input. To show the benefits of the proposed two-stage training strategy with boosting, our two previous depth estimation (DE) networks, one with t-shaped adaptive kernels and the other with exponential disparity volumes, are extended with our new learning strategy, referred to as DBoosterNet-t and DBoosterNet-e, respectively. Our self-supervised DBoosterNets are competitive, and in some cases even better, compared to the most recent supervised SOTA methods, and are remarkably superior to the previous self-supervised methods for monocular DE on the challenging KITTI dataset. We present intensive experimental results, showing the efficacy of our method for the self-supervised monocular DE task. [ABSTRACT FROM AUTHOR]

Although high dynamic range (HDR) display has become popular recently, the legacy content such as standard dynamic range (SDR) video is still in service and needs to be properly converted on HDR display devices. Therefore, it is desirable for HDR TV sets to have the capability of automatically converting input SDR video into HDR video, which is called reverse tone mapping (RTM). In this paper, we propose a novel learning-based low-complexity RTM scheme that not only expands the suppressed dynamic ranges (DR) of the SDR videos (or images), but also effectively restores lost detail in the SDR videos. Most existing conventional RTM schemes have focused on how to expand the DR of global contrast, resulting in limitations in recovering lost detail of SDR videos. On the other hand, the recent convolutional neural network-based approaches show promising results, but they are too complex to be applied on the users’ devices in practice. In this paper, our learning-based RTM scheme is computationally simple but effective in recovering lost detail. To learn the SDR-to-HDR relation, training “SDR-HDR” images are first separated into their base layer components and detail layer components by applying a guided filter. The detail layer components of the “SDR-HDR” pairs are used to train the SDR-to-HDR mapping. The mapping matrices are computed based on kernel ridge regression. In the meantime, the global contrast of the base layers is expanded by a nonlinear function that suppresses darker regions and amplifies brighter regions to fit the full DR of a target HDR display. To verify the effectiveness of our learning-based RTM scheme, we performed subjective quality assessment for images and videos. The experimental results show that our RTM scheme outperforms the existing RTM scheme with the successful restoration of lost detail in SDR images. [ABSTRACT FROM AUTHOR]

Asymmetric video coding based on binocular suppression provides a prospect for improving coding efficiency since one view in stereoscopic video can be encoded at lower bitrates than the other view without a loss of perceptual video quality. In this paper, we propose a joint rate control scheme for asymmetric stereoscopic video coding in a hybrid video codec system, which supports binocular suppression-based asymmetric stereoscopic video coding in a theoretical basis within an optimization framework. To obtain an optimal solution to quantization steps for joint rate control, we apply the Karush–Kuhn–Tucker (KKT) conditions in minimizing the perceptual distortion of decoded video under the conditions that: 1) the sum of bits generated from two video encoders is constrained within a given bit budget; 2) the quality of a primary view is superior to that of an auxiliary view with an allowable difference between two view qualities; and 3) the distortion of an auxiliary view is less than a given threshold value for high bitrate encoding. As a result, the proposed rate control scheme effectively enhances the perceptual quality based on an optimization framework by taking into account the binocular suppression and also simultaneously controlling the rates of both the left and right views of stereoscopic video under the constraints with a given target bit budget and an allowed quality difference. Experimental results demonstrate that, compared with the independent rate control scheme, the proposed joint rate control scheme with KKT conditions for asymmetric stereoscopic video coding not only demonstrates comparable 3-D perceived visual quality with an overall gain of 0.33 dB, but also attains 2-D visual quality gain by an average of 2.49 dB while accurately satisfying the given constraints in the proposed optimization framework. [ABSTRACT FROM AUTHOR]

In this paper, a frame-level rate control scheme is proposed based on texture and nontexture rate models for High Efficiency Video Coding (HEVC). Due to more complicated coding structures and the adoption of new coding tools, the statistical characteristics of transform residues are significantly different depending on the depth levels of coding units (CUs) from which the residues are obtained. A new texture rate model is constructed for the transform residues, which are categorized into three types of CUs: low-, medium- and high-textured CUs. One single Laplacian probability PDF model is used for each residue category to derive a rate-quantization model. Based on the Laplacian PDF, a simplified rate model for texture bits is derived using entropy. In addition, an analytic rate model for nontexture bits is proposed, which also takes into account the different characteristics of nontexture bits occurring in various depths of CUs in HEVC. The nontexture bitrates are modeled based on the linear relation between the total nontexture data and the dominant nontexture data in each CU category. Based on the proposed rate models for the texture and nontexture bits, accurate rate control can be achieved owing to more precise rate estimation. The experimental results show that the proposed rate control scheme achieves the average PSNR with 0.44 dB higher and the average PSNR standard deviation of 0.32 point lower with the buffer status levels maintained very close to target buffer levels, compared to the conventional methods. Finally, the proposed rate control scheme remarkably outperforms the conventional schemes especially for the sequences of complex texture and large motion. [ABSTRACT FROM PUBLISHER]

Peak signal-to-noise ratio (PSNR) monitoring is an important application for video quality assessment of video systems at the receiver sides where no-reference PSNR estimation is essential. Most of the PSNR estimation methods for H.264/AVC bitstreams ignore or do not consider the effect of deblocking filtering. Instead, they only focus on estimating the mean squared error (MSE) due to quantization. However, the PSNR estimation affected by deblocking filtering cannot be negligible for sequences of large picture resolutions. In this paper, we first present an MSE estimation method on H.264/AVC bitstreams by considering the deblocking filtering effect so that more accurate PSNR estimation can be made. For this, the total MSE between the original and reconstructed frames is separated into two terms for PSNR estimation: one due to quantization error and the other due to the deblocking filtering effect in H.264/AVC. In the proposed PSNR estimation, the contribution of deblocking filtering to the total MSE is quantified by a compensation factor of each encoded picture type between the original and the deblocked frames. Experimental results show that the proposed method effectively reflects the contribution of deblocking filtering to PSNR estimation, thus yielding more accurate PSNR estimates. [ABSTRACT FROM PUBLISHER]

In this study, the performance of hierarchical transform coding is analysed with design of an order‐16 integer transform kernel. The proposed hierarchical transform‐coding structure is constructed with a set of 4 × 4, 8 × 8 and 16 × 16 integer transforms of variable transform block sizes, which takes the advantages of both lower and higher transform kernels by flexibly adapting to varying image characteristics of video sequences with homogeneous and complex regions. The proposed hierarchical transform‐coding structure is implemented as an extension to H.264/advanced video coding joint model. The authors show the effectiveness of the hierarchical variable‐sized block transform scheme by analysing the quantisation effects and the correlation among neighbouring pixels in video sequences of different spatial resolutions. The experimental results show that: (i) the variable‐sized block transform scheme with the hierarchical structure is advantageous to the texture regions with strong local edges and (ii) the higher‐order‐16 integer transform kernel itself is more effective for the homogeneous texture regions, which are often encountered in higher resolution sequences. Therefore these two features can complementarily work in an rate‐distortion (RD) optimised manner for various characteristics of the input signals. [ABSTRACT FROM AUTHOR]

Highlights: [•] First addresses object detection and tracking with graph-based representation in H.264/SVC. [•] The block partitions with motion information are modeled as spatial and spatio-temporal graphs. [•] Graph-based analyses are performed to remove false objects and precisely identify real objects. [•] Interlayer graph mapping reliably detects and tracks object in the higher spatial layer. [•] It reduces the computation complexity of object detection and tracking in the higher spatial layer. [Copyright &y& Elsevier]

High Efficiency Video Coding (HEVC), a new video coding standard currently being established, adopts a quadtree-based Coding Unit (CU) block partitioning structure that is flexible in adapting various texture characteristics of images. However, this causes a dramatic increase in computational complexity compared to previous video coding standards due to the necessity of finding the best CU partitions. In this paper, a fast CU splitting and pruning method is presented for HEVC intra coding, which allows for significant reduction in computational complexity with small degradations in rate-distortion (RD) performance. The proposed fast splitting and pruning method is performed in two complementary steps: 1) early CU split decision and 2) early CU pruning decision. For CU blocks, the early CU splitting and pruning tests are performed at each CU depth level according to a Bayes decision rule method based on low-complexity RD costs and full RD costs, respectively. The statistical parameters for the early CU split and pruning tests are periodically updated on the fly for each CU depth level to cope with varying signal characteristics. Experimental results show that our proposed fast CU splitting and pruning method reduces the computational complexity of the current HM to about 50% in encoding time with only 0.6% increases in BD rate. [ABSTRACT FROM PUBLISHER]

Video quality estimation is considered a means of monitoring quality of service in broadcasting or IPTV services. In this paper, a no-reference peak signal-to-noise ratio (PSNR) estimation method is first presented for a quadtree-based motion estimation or compensation and transform coding scheme such as HEVC test model (HM), which is expected to be popularly used due to its highly enhanced coding efficiency, in 2-D and 3-D high resolution videos. The proposed no-reference PSNR estimation method is based on a Laplacian mixture distribution, which takes into account the distribution characteristics of residual transform coefficients in different quadtree depths and coding types of coding units (CUs). In order to predict the model parameters of the Laplacian mixture distribution for all zero quantized coefficients case, an exponential regression scheme is employed over quadtree depth levels of CUs. The proposed no-reference PSNR estimation method yields fairly accurate results from 0.970 to 0.983 in correlation and from 0.530 to 0.890 in RMSE between the actual and the estimated PSNR values for HM encoded bitstreams, outperforming single PDF based models. [ABSTRACT FROM PUBLISHER]

Although the temporal scalability coding supported by both H.264/AVC and H.264/SVC provides a good means of adapting bitstream scalability in temporal dimension, it entails heavy computational complexity due to bidirectional motion estimation and compensation for B-pictures in temporal scalability levels. To make it more applicable, this paper introduces a fast inter-prediction mode decision method which is efficiently performed using statistical hypothesis testing. The hypothesis testing is performed on mean and variance of pixels in 16 \times 16 and 8 \times 8 blocks to decide early termination against further processing for their sub-blocks. Fallback check is employed to compromise the tradeoff between the compression efficiency and encoding time saving. The proposed method exhibits effective performance for early termination in the inter-prediction mode decision, thus leading to a significant reduction of the total encoding time up to average 64.8% with small increments in bit amounts and small degradation in visual quality. [ABSTRACT FROM PUBLISHER]

Selective protection can exploit dependency coding properties to effectively perform partial protection on scalable video coding (SVC) bitstreams since protecting frames in lower scalability layers affects visual quality of the reconstructed frames in higher scalability layers. In this paper, we propose a selective protection scheme that maximizes protection effects by the minimum number of encoded frames to be protected in the SVC bitstream domain. We first model the SVC dependency coding structure as a directed acyclic graph which is characterized with an estimated visual quality value as the attribute at each node. In addition, a visual quality estimation model is proposed based on the proportions of intra-predicted and inter-predicted MBs, amounts of residues, and estimated visual quality of reference frames. The proposed selective protection scheme traverses the dependency graph to find optimal protection paths that can give the maximum visual quality degradation. Experimental results show that, compared to the existing protection schemes, the proposed selective protection scheme reduces computation complexity in the number of protected frames, the amount of protected data, and the protection time saving. The SVC file format specification supports the carriage of selectively protected bitstreams based on the concept of our selective protection in dependency coding structure of SVC. [ABSTRACT FROM AUTHOR]

In this paper, a fast mode decision method for the spatial higher layers (SHLs) in scalable video coding (SVC) is proposed based on coding dependency between two adjacent spatial lower and higher layers. The proposed fast mode decision method detects zero motion and zero transform coefficient blocks in the current spatial layer using the already encoded information of the corresponding blocks from the lower layer. The information for zero motion vectors and zero transform coefficients is used to induce a reduced set of candidate modes in SHLs of SVC, which reduces the computation complexity up to about 75% of the total encoding time while maintaining the coding performance with negligible amounts of degradation in peak signal-to-noise ratio values and bitrates. [ABSTRACT FROM PUBLISHER]

Although generative adversarial networks (GANs) are successfully applied to diverse fields, training GANs on synthetic aperture radar (SAR) data is a challenging task due to speckle noise. On the one hand, in a learning perspective of human perception, it is natural to learn a task by using information from multiple sources. However, in the previous GAN works on SAR image generation, information on target classes has only been used. Due to the backscattering characteristics of SAR signals, the structures of SAR images are strongly dependent on their pose angles. Nevertheless, the pose angle information has not been incorporated into GAN models for SAR images. In this paper, we propose a novel GAN-based multi-task learning (MTL) method for SAR target image generation, called PeaceGAN, that has two additional structures, a pose estimator and an auxiliary classifier, at the side of its discriminator in order to effectively combine the pose and class information via MTL. Extensive experiments showed that the proposed MTL framework can help the PeaceGAN's generator effectively learn the distributions of SAR images so that it can better generate the SAR target images more faithfully at intended pose angles for desired target classes in comparison with the recent state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Presents a video segmentation algorithm for object-oriented programming comprising the MPEG-4 standard. Method for separating moving objects from the background in movement sequences; Utilization of temporal information for localizing moving objects and spatial information.

Focuses on the use of linear and nonlinear adaptive networks in target discrimination in synthetic aperture radar (SAR) imagery. Neural networks for pattern classification; Performance of analysis using pattern recognition perspective; Detection of the Neyman-Person criterion.

Recently, due to the strength of deep convolutional neural networks (CNN), many CNN-based image quality assessment (IQA) models have been studied. However, previous CNN-based IQA models likely have yet to utilize the characteristics of the human visual system (HVS) fully for IQA problems when they simply entrust everything to the CNN, expecting it to learn from a training dataset. Therefore, the performance capabilities of such deep-learning-based methods are somewhat saturated. However, in this article, we propose a novel saliency-channel attention residual network based on the just-noticeable-difference (JND) concept for full-reference image quality assessments (FR-IQA). It is referred to as JND-SalCAR and shows significant improvements in large IQA datasets with various types of distortion. The proposed JND-SalCAR effectively learns how to incorporate human psychophysical characteristics, such as visual saliency and JND, into image quality predictions. In the proposed network, a SalCAR block is devised so that perceptually important features can be extracted with the help of saliency-based spatial attention and channel attention schemes. In addition, a saliency map serves as a guideline for predicting a patch weight map in order to afford stable training of end-to-end optimization for the JND-SalCAR. To the best of our knowledge, our work presents the first HVS-inspired trainable FR-IQA network that considers both visual saliency and the JND characteristics of the HVS. When the visual saliency map and the JND probability map are explicitly given as priors, they can be usefully combined to predict IQA scores rated by humans more precisely, eventually leading to performance improvements and faster convergence. The experimental results show that the proposed JND-SalCAR significantly outperforms all recent state-of-the-art FR-IQA methods on large IQA datasets in terms of the Spearman rank order coefficient (SRCC) and the Pearson linear correlation coefficient (PLCC). [ABSTRACT FROM AUTHOR]

In this paper, we propose a new R/D estimation method for intra-predictive coding with deep block partitioning structures. In our proposed R/D prediction, we adopt a particle filtering based prediction (PFP) to precisely predict intermediate R/D estimates for the next frame in a stochastic manner, which helps increasing the prediction accuracy of fast changing R/D values. Then, based on the intermediate R/D estimates by PFP, we infer an optimal model parameter of the TC’s probability density function (pdf) via convex optimization. We found that the proposed method brings about more stable R/D estimation performance thanks to both the improved prediction accuracy using the PFP for abrupt changes in true R/D values and the precise estimation of the optimal model parameter. Experimental results show that our method significantly reduces the normalized root mean square error from average 3.17 to 0.79 (74.90% reduction) for rate and from average 2.32 to 0.82 (64.61% reduction) for distortion, compared to the state-of-the art method. [ABSTRACT FROM AUTHOR]

Recently, deep convolutional neural networks (CNNs) have started to be applied for automatic target recognition (ATR) problems of synthetic aperture radar (SAR) images. In conventional SAR-ATR algorithms, the pose angle information of the target has been importantly used. However, recent deep learning-based SAR-ATR algorithms often only utilize the intensity information. In this paper, based on the prior works that the pose angle is an important latent variable for boosting target recognition performance, we propose a CNN-based SAR target recognition network with pose angle marginalization learning, called SPAM-Net that marginalizes the conditional probabilities of SAR targets over their pose angles to precisely estimate the true class probabilities. The proposed SPAM-Net consists of two sub-nets: (i) a sub-net for class-conditional probability estimation, called CP sub-net, and (ii) a sub-net for pose angle probability estimation, called PP sub-net. The two sub-nets are jointly learned via an end-to-end manner in a Bayesian framework so that the SPAM-Net incorporates the pose angle information into target recognition task effectively. The SPAM-Net outperforms our baseline network that does not utilize the pose angle information. In the experiments, we intensively analyze the effectiveness of pose angle information for SAR-ATR, revealing that more accurate pose angle information helps the SPAM-Net precisely estimate target classes for the misclassified target groups that are obtained by the baseline network. Furthermore, our method also outperforms the other state-of-the-art SAR-ATR algorithms, yielding the correct target recognition rate with average 99.61%. [ABSTRACT FROM AUTHOR]

Joint exploration model (JEM) reference codecs of ISO/IEC and ITU-T utilize multiple types of integer transforms based on DCT and DST of various transform sizes for intra- and inter-predictive coding, which has brought a significant improvement in coding efficiency. JEM adopts three types of integer DCTs (DCT-II, DCT-V, and DCT-VIII), and two types of integer DSTs (DST-I and DST-VII). The fast computations of Integer DCT-II and DST-I are well known, but few studies have been performed for the other types such as DCT-V, DCT-VIII, and DST-VII for all transform sizes. In this paper, we present fast computation methods of N-point DCT-V and DCT-VIII. For this, we first decompose the DCT-VIII into a preprocessing matrix, the DST-VII and a post-processing matrix to quickly compute it by using the linear relation between DCT-VIII and DST-VII. Then, we approximate integer kernels of N = 4, 8, 16, and 32 for DCT-V, DCT-VIII, and DST-VII with norm scaling and bit-shift to be compatible with quantization in each stage of multiplications between decomposed matrices for video coding. In various experiments, the proposed fast computation methods have shown to effectively reduce the total complexity of the matrix operations with little loss in BDBR performance. In particular, our methods reduce the number of addition and multiplication operations by 38% and 80.3%, respectively, in average, compared to the original JEM. [ABSTRACT FROM AUTHOR]

In this paper, we present a novel hardware-friendly super-resolution (SR) method based on a convolutional neural network (CNN) and its dedicated hardware (HW) on field programmable gate array (FPGA). Although CNN-based SR methods have shown very promising results for SR, their computational complexities are prohibitive for hardware implementation. To the best of our knowledge, we are the first to implement a real-time CNN-based SR HW that upscales 2K full high-definition video to 4K ultra high-definition (UHD) video at 60 frames per second (fps). In our dedicated CNN-based SR HW, low-resolution input frames are processed line-by-line, and the number of convolutional filter parameters is reduced significantly by incorporating depth-wise separable convolutions with a residual connection. Our CNN-based SR HW incorporates a cascade of 1D convolutions having large receptive fields along horizontal lines while keeping vertical receptive fields minimal, which allows us to save required line memory space in achieving comparable SR performance against full 2D convolution operations. For efficient HW implementation, we use a simple and effective quantization method with little peak signal-to-noise ratio (PSNR) degradation. Also, we propose a compression method to efficiently store intermediate feature map data to reduce the number of line memories used in HW. Our HW implementation on the FPGA generates 4K UHD frames of higher PSNR values at 60 fps and shows better visual quality, compared with conventional CNN-based SR methods that are trained and tested in software. [ABSTRACT FROM AUTHOR]

We present a novel and effective learning-based frame rate upconversion (FRUC) scheme, using linear mapping. The proposed learning-based FRUC scheme consists of: 1) a new hierarchical extended bilateral motion estimation (HEBME) method; 2) a light-weight motion deblur (LWMD) method; and 3) a synthesis-based motion-compensated frame interpolation (S-MCFI) method. First, the HEBME method considerably enhances the accuracy of the motion estimation (ME), which can lead to a significant improvement of the FRUC performance. The proposed HEBME method consists of two ME pyramids with a three-layered hierarchy, where the motion vectors (MVs) are searched in a coarse-to-fine manner via each pyramid. The found MVs are further refined in an enhanced resolution of four times by jointly combining the MVs from the two pyramids. The HEBME method employs a new elaborate matching criterion for precise ME which effectively combines a bilateral absolute difference, an edge variance, pixel variances, and an MV difference among two consecutive blocks and its neighboring blocks. Second, the LWMD method uses the MVs found by the HEBME method and removes the small motion blurs in original frames via transformations by linear mapping. Third, the S-MCFI method finally generates interpolated frames by applying linear mapping kernels for the deblurred original frames. In consequence, our FRUC scheme is capable of precisely generating interpolated frames based on the HEBME for accurate ME, the S-MCFI for elaborate frame interpolation, and the LWMD for contrast enhancement. The experimental results show that our FRUC significantly outperforms the state-of-the-art non-deep learning-based schemes with an average of 1.42 dB higher in the peak signal-to-noise-ratio and shows comparable performance with the state-of-the-art deep learning-based scheme. [ABSTRACT FROM AUTHOR]

Conventional predictive video coding-based approaches are reaching the limit of their potential coding efficiency improvements, because of severely increasing computation complexity. As an alternative approach, perceptual video coding (PVC) has attempted to achieve high coding efficiency by eliminating perceptual redundancy, using just-noticeable-distortion (JND) directed PVC. The previous JNDs were modeled by adding white Gaussian noise or specific signal patterns into the original images, which were not appropriate in finding JND thresholds due to distortion with energy reduction. In this paper, we present a novel discrete cosine transform-based energy-reduced JND model, called ERJND, that is more suitable for JND-based PVC schemes. Then, the proposed ERJND model is extended to two learning-based just-noticeable-quantization-distortion (JNQD) models as preprocessing that can be applied for perceptual video coding. The two JNQD models can automatically adjust JND levels based on given quantization step sizes. One of the two JNQD models, called LR-JNQD, is based on linear regression and determines the model parameter for JNQD based on extracted handcraft features. The other JNQD model is based on a convolution neural network (CNN), called CNN-JNQD. To our best knowledge, our paper is the first approach to automatically adjust JND levels according to quantization step sizes for preprocessing the input to video encoders. In experiments, both the LR-JNQD and CNN-JNQD models were applied to high efficiency video coding (HEVC) and yielded maximum (average) bitrate reductions of 38.51% (10.38%) and 67.88% (24.91%), respectively, with little subjective video quality degradation, compared with the input without preprocessing applied. [ABSTRACT FROM AUTHOR]

In this paper, a High Efficiency Video Coding (HEVC)-compliant perceptual video coding (PVC) scheme is introduced based on just-noticeable difference (JND) models in both transform and pixel domains. We adopt an existing pixel-domain JND model for the transform skip mode of HEVC and propose a transform-domain JND model for the transform nonskip modes of HEVC. The proposed transform-domain JND model is designed by considering the spatial JND characteristics such as contrast sensitivity, luminance adaptation, and contrast masking effects as well as by considering the summation effects of variable block-sized transforms in HEVC. A temporal JND model is additionally incorporated into the proposed transform-domain JND model to further reduce perceptual redundancy. To incorporate the transform- and pixel-domain JND models into the encoding process in an HEVC-compliant manner, the transform coefficients and residues are suppressed in harmonization with the transform/quantization process and the quantization-only process of HEVC, respectively. To make the JND-based suppression effective, a distortion compensation factor is also proposed to reflect the perceptual distortion in the rate-distortion optimization-based encoding process. Based on subjective quality assessments of the encoded bit streams of test sequences, the proposed HEVC-compliant PVC scheme yields remarkable bitrate reductions of a maximum 49.10% and an average 16.10% with negligible subjective quality loss, compared with an HEVC reference software HEVC test model (HM 11.0). In addition, the proposed HEVC-compliant PVC scheme increases the encoding complexity of HM 11.0 only by an average of 11.25%. [ABSTRACT FROM AUTHOR]

Recently, a new video coding standard, High Efficiency Video Coding (HEVC), has shown greatly improved coding efficiency by adopting hierarchical structures of coding unit (CU), prediction unit (PU), and transform unit (TU). To best achieve the coding efficiency, the best combinations of CU, PU, and TU must be found in the sense of the minimum rate-distortion (R-D) costs. Owing to this, a large computational complexity occurs. Among these CU, PU, and TU, the determination of CU sizes most significantly affects the R-D performance of HEVC encoders, which causes large computational costs in operation with PU and TU size determinations. In spite of recent works in the complexity reduction of HEVC encoders, most of the research has focused on the complexity reduction with fast CU split in intra slice coding and with early TU split in both intra and inter slice. In this paper, we propose a fast and an efficient CU encoding scheme based on the spatiotemporal encoding parameters of HEVC encoders, which consists of an improved early CU SKIP detection method and a fast CU split decision method. For the current CU block under encoding, the proposed scheme utilizes sample-adaptive-offset parameters as the spatial encoding parameter to estimate the texture complexity that affects the CU partition. In addition, the motion vectors, TU size, and coded block flag information are used as the temporal encoding parameters to estimate the temporal complexity that also affects the CU partition. The proposed scheme effectively utilizes the spatiotemporal encoding parameters that are the byproducts during the encoding process of HEVC without additionally required computation. The proposed novel fast CU encoding scheme significantly reduces the total encoding time with negligible RD-performance loss. The experimental results show that the proposed scheme achieves the total encoding time savings of average 49.6% and 42.7% only with average 1.4% and 1.0% bit-rate losses for various test sequences under random access and low delay B conditions, respectively. The proposed scheme has an advantage on the implementation for parallel processing in pipeline structures of HEVC encoders due to its independency with neighboring CU blocks. [ABSTRACT FROM PUBLISHER]

Due to the excessive number of TV program contents available at user's side, efficient access to the preferred TV program content becomes a critical issue for smart TV user interaction. In this paper, we propose an automatic recommendation scheme of TV program contents in sequence using sequential pattern mining (SPM). Motivation of sequential TV program recommendation is based on TV viewer's behaviors for watching multiple TV program contents in a row. A sequence of TV program contents for recommendation to a target user is constructed based on the features such as an occurrence and net occurrence of frequently watched TV program contents from the similar user group to which the target user belongs. Three types of SPM methods are presented-offline, online and hybrid SPM. To extract sequential patterns of preferably watched TV program contents, we propose a preference weighted normalized modified retrieval rank (PW-NMRR) metric for similar user clustering. In the offline SPM method, we effectively construct the sequential patterns for recommendation using a projection method, which yields good performance for relatively longer sequential patterns. The online SPM method mines sequential patterns online by effectively reflecting the recent preference characteristics of users for TV program contents, which is effective for short-sequence recommendation. The hybrid SPM method combines the offline and online SPM methods. The maximum precisions of 0.877, 0.793 and 0.619 for length-1, -2 and -3 sequence recommendations are obtained from the online, hybrid and offline SPM methods, respectively. [ABSTRACT FROM AUTHOR]

Recently, 3DTV has drawn much attention as a new broadcasting service. In spite of technical advances in 3DTV broadcasting services, there are two barriers that hinder the new services from being launched for terrestrial broadcasting: the lack of available bandwidth for transmission of additional-view video via terrestrial channel and backward compatibility with the legacy 2-D HDTV services. As an alternative, a hybrid stereoscopic video coding scheme is proposed for a stereoscopic 3DTV service, where one-view video is transmitted through the legacy broadcasting systems and the other one is delivered via broadband networks to which TV terminals are connected. In this paper, the proposed hybrid stereoscopic video coding scheme incorporates an MPEG-2 codec for backward compatibility with a legacy 2-D HDTV service and an HEVC codec with inter-view prediction coding as an extension to encode additional-view video sequences with high coding efficiency. The proposed inter-view prediction coding scheme in the extended HEVC incorporates an advanced motion and disparity vector prediction (AMDVP) method for enhanced motion- and disparity-compensated coding. The experimental results demonstrate that the proposed hybrid stereoscopic video coding scheme achieves an average coding efficiency of 38.22% (32.86%) in BD-rate or 1.394 dB (1.235 dB) in BD-PSNR for the out-band (in-band) scenario. [ABSTRACT FROM PUBLISHER]

Due to the rapid increase of contents available under the convergence of broadcasting and Internet, efficient access to personally preferred contents has become an important issue. In this paper, an automatic recommendation scheme based on collaborative filtering is presented for intelligent personalization of (IP)TV services. The proposed scheme does not require TV viewers (users) to make explicit ratings on their watched TV program contents. Instead, it implicitly infers the users' interests on the watched TV program contents. For the recommendation of user preferred TV program contents, our proposed recommendation scheme first clusters TV users into similar groups based on their preferences on the content genres from the user's watching history of TV program contents. For the personalized recommendation of TV program contents to an active user, a candidate set of preferred TV program contents is obtained via collaborative filtering for the group to which the active user belongs. The candidate TV programs for recommendation are then ranked by a proposed novel ranking model. Finally, a set of top-N ranked TV program contents is recommended to the active user. The experimental results show that the proposed TV program recommendation scheme yields about 77% of average precision accuracy and 0.135 value of ANMRR (Average Normalized Modified Retrieval Rank) with top five recommendations for 1,509 people. [ABSTRACT FROM PUBLISHER]

Abstract: Currently, digital broadcasting tends to personalize the TV watching environment by offering services that can adapt to viewer preferences. In this paper, an intelligent broadcasting system for enhanced personalized-services, based on the semantics of broadcasting contents, is proposed. To implement the system, the MPEG-7 and TV-Anytime Forum (TVAF), as well as agent technology, are employed. For content-level services, real-time content filtering, personalized video skimming, and content-based retrieval using audio characteristic are implemented. To verify the usefulness of the proposed system, we demonstrate it with a test-bed on which content-level personalized services are implemented 1 [1] This work was supported in part by the SmarTV project of the Ministry of Information and Communication, Republic of Korea. . [Copyright &y& Elsevier]

A video summary abstracts the entirety with the gist without losing the essential content of the original video and also facilitates efficient content-based access to the desired content. In this article, we propose a novel method for summarizing a news video based on multimodal analysis of the content. The proposed method exploits the closed caption (CC) data to locate semantically meaningful highlights in a news video and speech signals in an audio stream to align the CC data with the video in a time-line. Then, the extracted highlights are described in a multilevel structure using the MPEG-7 Summarization Description Scheme (DS). Specifically, we use the HierarchicalSummary DS that allows efficient accessing of the content through such functionalities as multilevel abstracts and navigation guidance in a hierarchical fashion. Intensive experiments with our prototypical systems are presented to demonstrate the validity and reliability of the proposed method in real applications. © 2004 Wiley Periodicals, Inc. Int J Imaging Syst Technol 13, 267–274, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ima.10067 [ABSTRACT FROM AUTHOR]