Search

Universal access to and provisioning of multimedia content is now a reality. It is easy to generate, distribute, share, and consume any multimedia content, anywhere, anytime, or any device. Open media standards took a crucial role toward enabling all these use cases leading to a plethora of applications and services that have now become a commodity in our daily life. Interestingly, most of these services adopt a streaming paradigm, are typically deployed over the open, unmanaged Internet, and account for most of today’s Internet traffic. Currently, the global video traffic is greater than 60% of all Internet traffic [1] , and it is expected that this share will grow to more than 80% in the near future [2] . In addition, Nielsen’s law of Internet bandwidth states that the users’ bandwidth grows by 50% per year, which roughly fits data from 1983 to 2019 [3] . Thus, the users’ bandwidth can be expected to reach approximately 1 Gb/s by 2022. At the same time, network applications will grow and utilize the bandwidth provided, just like programs and their data expand to fill the memory available in a computer system. Most of the available bandwidth today is consumed by video applications, and the amount of data is further increasing due to already established and emerging applications, e.g., ultrahigh definition, high dynamic range, or virtual, augmented, mixed realities, or immersive media applications in general.

LiDAR odometry (LO) describes the task of finding an alignment of subsequent LiDAR point clouds. This alignment can be used to estimate the motion of the platform where the LiDAR sensor is mounted on. Currently, on the well-known KITTI Vision Benchmark Suite state-of-the-art algorithms are non-learning approaches. We propose a network architecture that learns LO by directly processing 3D point clouds. It is trained on the KITTI dataset in an end-to-end manner without the necessity of pre-defining corresponding pairs of points. An evaluation on the KITTI Vision Benchmark Suite shows similar performance to a previously published work, DeepCLR [1], even though our model uses only around 3.56% of the number of network parameters thereof. Furthermore, a plane point extraction is applied which leads to a marginal performance decrease while simultaneously reducing the input size by up to 50%., 5 pages, 4 figures, accepted at IEEE ICIP 2021

Digital whole-slide images (WSI) are scanned representations of histological tissue at microscopic scale, enabling computer aided diagnosis and remote pathology applications. However, the data sizes may hinder a widespread use, as raw files can easily exceed 10-20GB. In this work, we explore the Scalable High Effciency Video Coding (SHVC) as a replacement for the JPEG standard currently found in most vendor formats. Besides a comparison of the compression rates, this work comprises a user-study to estimate SHVC quantization parameters (QP) and JPEG quality level that threshold the just-noticeable distortions (JND) for a compression below the JND.

We propose a novel method for content based image retrieval based on the features extracted from the convolutional layers of the deep neural network architecture. Some of the popular approaches form the feature vectors from the fully connected layers of the convolutional neural networks or directly concatenate the features from the convolutional layers. However, the main problem with the use of feature vectors from fully connected layers is that the spatial information about the objects are lost. This motivated us to use the features from the convolutional layer. We incorporate a pyramid pooling based approach to form more compact and location invariant feature vectors. We have measured the Mean Average Precision (MAP) on benchmark databases such as the Holidays and Oxford5K datasets using features extracted from the AlexNet model. The proposed method gives better retrieval results compared to other state-of-the-art approaches which use feature vectors from fully connected layers and convolutionallayers without spatial pooling.

In-loop filtering is an important task in video coding, as it refines both the reconstructed signal for display and the pictures used for inter-prediction. In order to remove coding artifacts, machine learning based methods are assumed to be beneficial, as they utilize some prior knowledge on the characteristics of raw images. In this contribution, a dictionary learning / sparse coding based inloop filter and a frequency adaptation model based on the lp-ballenergy in the spectral domain is proposed. Thereby the dictionary is trained on raw data and the algorithms are controlled mainly by the parameter for the sparsity. The frequency adaption model results in further improvement of the sparse coding based loop filter. Experimental results show that the proposed method results in coding gains up to l-4.6 % at peak and -1.74 % on average against HEVC in a Random Access coding configuration.

In this paper, a new approach for an improved video coding scheme is presented, which combines hybrid video coding and texture synthesis based on motion distribution statistics. Considering that the utilized texture synthesis approach provides high-quality visual results, while it is developed only for synthe- sizing the identified dynamic textures within a certain area, a new framework is presented, which allows to identify of areas for synthesis and combine conventional coding with synthesis. Also, a new representation and compression of synthesis parameters is presented, which is required due to the updated coding structure. When combining the proposed approach with conventional en- coder (HEVC reference software, HM 16.6), significantly reduced bit rates of the compressed video sequences with the texture replaced can be obtained. Moreover, because the synthesized textures have similar perceptual characteristics to those of the original textures, the video sequences with the texture replaced are also visually similar to the original sequences. Video results are provided online to allow assessing the visual quality of the tested content.

The main idea of this paper is to extract the 3D scene geometry for the observed scene and use it for synthesizing a more precise prediction using Image-Based Rendering (IBR) for motion compensation in a hybrid coding scheme. The proposed method first extracts camera parameters using Structure from Motion (SfM). Then, a Patch-based Multi-View Stereo (PMVS) technique is employed to generate the scene Point-Cloud (PC) only from already decoded key-frames. Since the PC could be really sparse in poorly reconstructed regions, a depth expansion mechanism is also used. This 3D information helps to properly warp textures from the key-frames to the target frame. This IBR-based prediction is then used as an additional reference for motion compensation. In this way, the encoder can choose between the rendered prediction and the regular reference pictures through a rate- distortion optimization. On average, the simulation results show about 2.16% bitrate reduction compared to the reference HEVC implementation, for tested dynamic and static scene video sequences.

In 360° video, a complete scene is captured, as it can be seen from a single point in any direction. Since the captured 360 images are spherical, they cannot be converted to planar images without introducing geometric distortions. The nature of these distortion depends on the used projection format.This paper introduces an approach to reduce artifacts occurring when encoding 360° video which has been projected to the faces of a cube. In order to achieve this, the operation of the deblocking filter is modified such that the correct pixels with respect to the 3D geometry are used for filtering of edges.The method is evaluated on the set of sequences defined by the Joint Call for Proposals on Video Compression with Capability beyond HEVC. While the method has almost no impact on the objective coding performance, the visual quality is still clearly enhanced. Edges of the cube, previously visible as coding artifacts, are mostly removed with the proposed method.

HEVC compensates motions by predicting a frame based on already decoded frames called reference frames. The more precise the prediction is, the fewer bits would be needed to transmit. The main idea of this paper is to employ the 3D geometry of scene to generate a more precise prediction. The proposed method first extracts point cloud and the 3D model of the scene based on a subset of 2D key-frames and then employs it to predict the remaining frames by virtually synthesizing them. Structure from Motion (SfM) and MultiView Reconstruction (MVR) have been employed for estimating camera parameters and constructing the 3D model of the scene. The virtual views are then synthesized based on the 3D model and Depth Image Based Rendering (DIBR) methods. This 3D model-based prediction is then offered to HEVC as an additional reference for motion compensation. It means HEVC could choose between our prediction and its own built-in prediction through a ratedistortion optimization. If the 3D model-based additional reference is precise enough, HEVC will prefer it and hopefully this results in less bitrate. Our simulation results show 2.5% and 8.7% bitrate reduction for dynamic and static scenes, respectively.

This paper presents improvements to a dynamic texture synthesis approach which is based on motion distribution statistics, able to produce high visual quality of synthesised dynamic textures. The aim is to recreate synthetically highly textured regions like water, leaves and smoke, instead of processing them with a conventional codec such as HEVC. The method involves two steps: analysis, where motion distribution statistics are computed, and synthesis, where the texture region is synthesized. Dense optical flow is utilized for estimating the random motion of dynamic textures. The performance of our dynamic texture analysis and synthesis approach is tested on cropped sequences, containing water, leaves and smoke. Simulation results show potential bitrate savings up to 50% on texture sequences at comparable visual quality.

Dynamic textures are characterized with irregular motions that are often challenging for motion compensation as applied in the state of the art video codecs. Due to rapid and randomly evolving nature of such a signal, it is accompanied with very high energy in the residual. As a result, B-pictures as used in HEVC layer are relative expensive to code. This leads to an overall increase in the bitrate. Further, increasing QP offset worsens the quality by forcing lower rate to these B-pictures, leading to strong blurring and blocking artefacts. In this paper, we exploit Steerable Pyramid (SP) for coding pictures with t id > 2 in a downsampled format. At the decoder side, details are synthesized for these low resolution pictures by adding back the high frequencies using motion compensation from the nearest key picture followed by an inverse SP transform. The paper synthesizes details for the dynamic textures that are expensive to code. Our investigation shows up to 31% saving in bitrate, while visual quality is kept acceptable.

3D scene reconstruction from multi-view images has many practical applications, including games, virtual/augmented reality, and digital archives of cultural heritage. In this paper, we introduce a new application in video compression. The proposed idea is to have the decoder reconstruct a 3D scene model based on a subset of decoded frames and then reproject the 3D model to 2D for prediction or reconstruction of intermediate and/or future frames; this can also include a further motion compensation step in 2D. Structure from Motion (SfM) has been employed as a tool to estimate 3D point clouds and camera parameters. This approach has been integrated to generate additional reference pictures in an HEVC codec, and was tested so far on two 4K video sequences: A computer generated sequence with moving objects and a natural but stationary scene captured from a moving camera. Initial simulation results show around 0.8% bit-rate reduction compared to HEVC Test Model (HM16.7). It is asserted that the method offers headroom for further improvements by enhancing the reconstruction algorithms.

International audience; —Dynamic texture motions like flowing water, motion of leaves etc. have a complex random character. A sequence containing such a content is challenging to encode even when using state of the art High Efficiency Video Coding (HEVC) especially, if the available bandwidth is limited. It is observed that often when predicting dynamic textures, codec switches to intra prediction. At lower rates, dynamic texture content shows visually annoying blurring and blocking artifacts. For dynamic textures, both spatial and temporal details are perceptually of less importance. This property of the Human Visual System (HVS) can be exploited when coding dynamic texture content, as suggested in this paper. At the encoder, preprocessing is done by skipping even numbered B pictures. At the decoder side, skipped pictures are synthesized using linear phase shift interpolation of the complex wavelet coefficients, from the adjacent already decoded pictures. Subjective evaluation of proposed approach is done by using a pair wise comparison test between the proposed results and the conventional HEVC decoded bitstream at similar bitrates. The evaluation results show that viewers prefer the proposed result over conventional HEVC.

A decoder-side motion vector derivation algorithm for hybrid video coding is proposed. The algorithm is based on template matching and aims to reduce motion parameter bit-rate by re-estimating the applicable motion parameters at the decoder side. An average bit-rate savings of about 6%-8% is observed compared to the reference H.264/AVC. Decoder-side motion vector derivation was included in multiple proposals for the new High Efficiency Video Coding standard. This paper details and analyzes the algorithm and discusses its relation to other coding tools.

Conventionally, complex motion in video sequences is approximated by smaller block units in order to be representable by a translational motion model. This approximation results in a fine block partitioning and a high prediction error, both at cost of more data rate than potentially necessary. A worthwhile data reduction has been shown to be achievable by adding a higher order motion model to the most recent video coding standard, High Efficiency Video Coding (HEVC). The benefit of this additional option of inter-frame prediction is due to the more accurate motion compensation as well as the usage of larger block sizes. This paper deals with more efficient encoding of higher order motion parameters in this context. The geometrically accurate prediction of higher order motion parameters from a neighbored block needs to consider the dependency of the block-to-block parameter difference based on the spatial relation between two block centers. An algorithm is introduced for correcting the translational component and reducing the difference between the actual and the predicted motion when determining higher order parameters from neighbored blocks. Additionally, a further increase of the maximum block size up to 512×512 pixels is investigated.

Informed Source Separation (ISS) is a topic unifying the research fields of both source separation and source coding. Its main objective is to recover audio objects out of a mixture with a source separation step assisted by a set of compact parameters extracted with complete knowledge of the sources. ISS can be used for applications such as active listening and remixing of music (e.g. karaoke). In this paper, we propose a new ISS method which includes a semi-blind source separation (SBSS) step in the ISS decoder to decrease the amount of parameter bit rate. SBSS is conducted by fac-torizing the mixture in time-frequency domain by nonnegative matrix factorization (NMF). The transmitted parameters consist of a compact NMF initialization as well as residuals calculated in the NMF domain. We show in simulations that using SBSS in the decoder increases the separation quality and that our scheme improves the rate-distortion performance in comparison to a state-of-the art method.

In many cases, view synthesis prediction (VSP) for 3D video coding suffers from inaccurate depth maps (DMs) caused by non-reliable point correspondences in low textured areas. In addition, occlusion handling is a major issue if prediction is performed from one reference view. In this paper, we address VSP for non-coplanar 3D video sequences modified by a linear DM enhancement model. In this model, points close to edges in the DM are treated as key points, on which depth estimation is assumed to work well. Other points are assumed to have unreliable depth information and the depth is modelled by the solution of the Laplace equation. As another preprocessing stage for DMs, the influence of guided image depthmap filtering is investigated. Moreover, we present an inpainting method for occluded areas, which is based on the solution of the Laplace equation and corresponding depth information. Combining these techniques and applying it to VSP for 3D video sequences, which are captured by an arc-arranged camera array, our approach outperforms state of the art models in coding efficiency. Experimental results show bit rate savings up to 4.3 % compared to HEVC-3D for coding two views and corresponding depth in an All Intra encoder configuration.

The current article focuses on the scalable video coding (SVC) extension in terms of technology, performance, and targeted application scenarios.

During the design and drafting phase of a video coding specification, the involved experts develop a common understanding of the technical tools and features which are adopted into the draft. For the final specification, an unambiguous representation of this common understanding is required which further must be completely independent of anything that was commonly understood in the development phase. In this chapter, scope of and requirements on specification text are presented. While there are general rules applicable to all standards and recommendations specified by the standardization bodies, some additional constraints on the video coding specification design as applied during the development of HEVC are presented as well. In the light of the aforementioned guidelines, this chapter summarizes the general concepts and principles behind the development of a video coding specification. It further includes definitions and explanations of terminology and methodology used for specification and introduces the structure of the specification and how to assess it. Furthermore, aspects of determination and selection of tools appropriate for inclusion in the specification are discussed and the methodology for the evaluation of tools is presented.

A novel method for generating a reliable initial surface mesh of the interested scene from a quasi-dense point cloud is presented in this article. Given multiple images taken from different points of view, a robust quasi-dense point cloud is acquired by accumulative triangulation of the potentially matched feature points. In our proposed method, the feature points are detected with Harris-corner and SIFT detectors. SIFT and DAISY descriptors are used to describe the neighboring environment of the detected features and provide efficient matching possibilities. The proposed method can be parallelized almost at each phase, which makes it suitable for image datasets of different sizes. The accuracy of the proposed method is evaluated quantitatively and qualitatively on different types and sizes of image datasets.

In scalable video coding, a stream contains multiple layers of different temporal resolution, spatial resolution, or quality. Each enhancement layer can utilize the information from the lower layers in order to improve the overall coding efficiency. In the draft scalable extension of HEVC the enhancement layer uses the lower layer reconstructed pixel values for prediction. While this dual-loop approach is efficient in terms of rate-distortion performance, it doubles the decoder complexity. In this paper a method is presented that allows for single-loop decoding using a new coefficient refinement scheme for the enhancement layer. The scheme conceptually allows for binary mapping between arbitrary quantizer step sizes in base and enhancement layer. Simulation results reveal a promising rate-distortion performance, especially for small quantizer deltas.

3D Video is a new technology, which requires the transmission of depth data alongside conventional 2D video. The additional depth information allows to synthesize arbitrary viewpoints at the receiver and enables adaptation of the perceived depth impression and driving of multi-view auto-stereoscopic displays. In contrast to natural video signals, depth maps are characterized by piecewise smooth regions bounded by sharp edges along depth discontinuities. Conventional video coding methods tend to introduce ringing artifacts along these depth discontinuities, which lead to visually disturbing geometric distortions in the view synthesis process. Preserving the described signal characteristics of depth maps is therefore a crucial requirement for new depth coding algorithms. In this paper a novel model-based intra-coding mode is presented, which works as an addition to conventional transform-based intra coding tools. The proposed intra coding mode yields up to 42% BD-rate savings in terms of depth rate and up to 2:5% in terms of total rate. The average bitrate savings are approximately 24% for depth rate and 1.5% for the total rate including texture and depth.

We present a framework with optimized error protection for different network abstraction layer (NAL) units of the H.264/AVC scalable video coding (SVC) streams for real-time multicast streaming on multiple channels. In order to efficiently utilize the network resource, different spatial layers are transmitted on different radio channels. Users request streams of different resolution by subscribing to the necessary radio channels. On each radio channel, we optimize the error protection for the spatial layer transmitted on that channel, considering the importance of different NAL units and channel quality of subscribers on that channel. As a further development of our previous framework for multicast streaming application of SVC stream with quality scalability on single radio channel, the framework in this paper supports spatial scalability and guarantees optimized error protection for different spatial layers. Our simulations show that graceful degradation can be achieved for videos with different resolutions. The user distribution with different channel quality is considered in the framework and a good tradeoff in terms of the video quality is achieved on each channel among different users.

Video streaming over wireless networks is subjected to instability of channel quality, which results in packet loss and reduced video quality at the user side. To mitigate the effect of packet loss, we proposed a framework using scalable video and unequal error protection for multicast and unicast streaming. A fast algorithm in the framework determines the optimal error protection for each NAL unit according to its real utility and the instantaneous channel quality. This framework adapts the error protection strength very well to the change of the channel quality and achieves the optimal system-level utility for multicast streaming. The framework is first verified with simulation tools. Then, a demo of video streaming is implemented for this framework within the 802.11n networks to test the suitability of this concept for realtime streaming.

We present an algorithm to optimize the allocation of channel bitrate to different network abstraction layer (NAL) units of the H.264/AVC scalable video bitstreams for real-time multicast streaming over heterogeneous networks. We focus on the problem of achieving a high robustness of video streaming under varying channel conditions in terms of the reconstructed video qualities at different users. As an extension of our previous work for unicast streaming, the proposed algorithm can achieve an optimized allocation of channel bitrate for multicast streaming with any user distribution. Our simulations show that a good performance on the video qualities among the multicast users can be achieved for different user distributions. A gain in terms of the overall multicast PSNR can be achieved against the protection strategies targeting at users with medium channel qualities in our experiments.

The ongoing increase of computing performance facilitates a higher algorithmical complexity in video coding systems. The decoder may be able to estimate or derive prediction parameters based on the previously decoded signal. In this paper we present an extension to H.264/AVC where the explicit coding of motion parameters is adaptively replaced by a template matching algorithm that is performed identically at the encoder and decoder. The decision between explicit coding and derivation of motion parameters is done by the rate-distortion optimised mode decision and coded into the bitstream. Compared to previous work, the provided scheme has been extended to bidirectional prediction (B pictures). Simulation results show an improved coding efficiency over a wide range of test sequences, especially for higher spatial resolutions.

In this paper, we present an algorithm using dynamic texture synthesis for closed-loop video coding. Video textures, or so-called dynamic textures are video sequences with moving texture showing some stationarity properties over time, like water surfaces, whirlwind, clouds, crowds, or even parts of head-and-shoulder scenes. By learning the temporal statistics of such content, we can in principle synthesize the corresponding areas in future frames of the video. In this paper we show that this synthesized image content can also be used for prediction in a closed-loop hybrid video coding system, where the encoder decides about usage of such synthesized content and possible transmission of a residual error signal. This is done in an adaptive and rate-distortion optimized way, such that higher compression performance can be achieved for both high and low bitrates. We show that local adaptation of the algorithm can lead to better compression performance and reduce the computation complexity considerably. If PSNR is used as the quality criterion, savings of up to 15 % in bitrate have been observed experimentally.

Decoder-side motion vector derivation (DMVD) using template matching has been shown to improve coding efficiency of H.264/AVC based video coding. Instead of explicitly coding motion vectors into the bitstream, the decoder performs motion estimation in order to derive the motion vector used for motion compensated prediction. In previous works, DMVD was performed using a full template matching search in a limited search range. In this paper, a candidate based fast search algorithm replaces the full search. While the complexity reduction especially for the decoder is quite significant, the coding efficiency remains comparable. While for the full search algorithm BD-Bitrate savings of 7.4% averaged over CIF and HD sequences according to the VCEG common conditions for IPPP high profile are observed, the proposed fast search achieves bitrate reductions of up to 7.5% on average. By further omitting sub-pel refinement, average savings observed for CIF and HD are still up to 7%.

In this paper, a multihypothesis prediction scheme for inter frame video coding is proposed. Using a template matching algorithm, motion vectors are derived at the decoder side instead of explicitly coding the motion vectors into the bitstream. Therefore, higher numbers of hypotheses can be used in the averaging process at no additional coding cost. The proposed scheme has been implemented into the H.264/AVC reference software. Simulation results show bitrate reductions compared to H.264/AVC of 7.7% on average for the tested video sequences. It is shown that part of the performance gain is due to rounding effects in H.264/AVC sub-pixel interpolation which can be exploited in the averaging calculation of the proposed multihypothesis prediction. Experiments with an improved interpolation filter for both reference scheme and the proposed scheme still yield bitrate reductions of 4.7% on average.

Personalized multimedia content which suites user preferences and the usage environment, and as a result improves the user experience, gains more importance. In this paper, we describe an architecture for personalized video adaptation and presentation for mobile applications which is guided by automatically generated annotations. By including this annotation information, more intelligent adaptation techniques can be realized which only reduce the quality of unimportant regions in case a bit rate reduction is necessary. Furthermore, presentation layers are added to enable advanced multimedia viewers to optimally present the interesting parts of a video in case the user wants to zoom in. This architecture is the result of collaborative research done in the EU FP6 IST INTERMEDIA project.

This contribution presents results of the MPEG verification test that was carried out for the new Scalable Video Coding (SVC) Amendment of H.264/AVC. The test consisted of a series of subjective comparisons of SVC and single layer H.264/AVC coding for different application scenarios including conversational applications, broadcasting over mobile channels, and HD broadcasting. The results show that a reasonable degree of spatial and quality scalability can be supported with a bit rate overhead of less than or about 10% and an indistinguishable visual quality compared to the state of the art single layer coding. This paper describes the coding conditions, the test procedure, and presents the results of the SVC verification test.

Current trends in smart homes suggest that several multimedia services will soon converge towards common standards and platforms. However this rapid evolution gives rise to several issues related to the management of a large number of multimedia streams in the home communication infrastructure. An issue of particular relevance is how a context acquisition system can be used to support the management of such a large number of streams with respect to the Quality of Service (QoS), to their adaptation to the available bandwidth or to the capacity of the involved devices, and to their migration and adaptation driven by the userspsila needs that are implicitly or explicitly notified to the system. Under this scenario this paper describes the experience of the INTERMEDIA project in the exploitation of context information to support QoS, migration, and adaptation of multimedia streams.

Dynamic textures are sequences of frames exhibiting certain stationarity properties over time; examples are sea-waves, whirlwind or moving crowds. We present an algorithm for dynamic texture extrapolation using only few training frames. A dynamic texture synthesizer using this algorithm has been integrated into a state-of-the-art H.264/AVC coding system, such that synthesized frames can be used by the encoder and decoder for inter prediction. For sequences not containing dynamic textures the same performance as with the conventional encoding system was achieved. In the case of sequences containing dynamic textures, intra coded macroblocks can be avoided by using the synthesized frame. Bitrate savings of up to 10% have been observed experimentally.

In this paper, a decoder side motion vector derivation scheme for inter frame video coding is proposed. Using a template matching algorithm, motion information is derived at the decoder instead of explicitly coding the information into the bitstream. Based on Lagrangian rate-distortion optimisation, the encoder locally signals whether motion derivation or forward motion coding is used. While our method exploits multiple reference pictures for improved prediction performance and bitrate reduction, only a small template matching search range is required. Derived motion information is reused to improve the performance of predictive motion vector coding in subsequent blocks. An efficient conditional signalling scheme for motion derivation in Skip blocks is employed. The motion vector derivation method has been implemented as an extension to H.264/AVC. Simulation results show that a bitrate reduction of up to 10.4% over H.264/AVC is achieved by the proposed scheme.

Efficient intra prediction is an important aspect of video coding with high compression efficiency. H.264/AVC applies directional prediction from neighboring pixels on an adjustable block size for local decorrelation. In this paper, we present an extended prediction scheme in the context of H.264/AVC that comprises two additional prediction methods exploiting self-similar properties of the encoded texture. A new macroblock type is implemented, allowing for flexible selection of the available prediction methods for sub-partitions of the macroblock. Depending on the content of the encoded video sequence, substantial gains in rate-distortion performance are achieved. The results may indicate directions towards an enhanced intra coding scheme with improved rate-distortion performance.

A new approach to combine R/D-optimization and lookahead coding is proposed. The dependent-R/D idea has been applied to blocks with no coded coefficients. This is an important case at low bit rates and whenever the motion model, which is used in virtually all modern video coders, fits accurately enough. The requirement of the motion model's applicability suggests the necessity to include spatial or temporal aliasing reducing filtering to support the proposed strategy.

A hierarchical spatially scalable wavelet video coder is presented. The lowpass band of the wavelet decomposition is the base layer in the scalable scheme. It is encoded using forward motion compensation. Here, warping prediction is employed. Since warping prediction produces a smooth prediction error it is well suited for wavelet decomposition. The coder employs backward motion compensation in the enhancement layers, therefore no enhancement layer motion vectors have to be transmitted. The coarser levels of the wavelet decomposition of the current frame are used for motion estimation and motion compensation of the lowpass band of the next finer level. Experimental results show that the application of warping prediction is suited to improve the coding gain of the presented scheme significantly.

In this paper a hierarchical spatial scalable wavelet video coder is presented. The scheme employs backward motion compensation, therefore no motion vectors have to be transmitted. The coarser levels of the wavelet decomposition of the current frame are used for motion estimation and motion compensation of the lowpass band of the next finer level. The lowpass band of the coarsest level has to be coded separately, e.g. using DPCM.

In this paper a new algorithm for joint detection and segmentation of human faces in color images sequence is presented. A skin probability image is generated using a model for skin color. Instead of a binary segmentation to detect skin regions, connected operators are used to analyze the skin probability image at different threshold levels. A hierarchical scheme of operators using shape and texture simplifies the skin probability image. For the remaining connected components, the likelihood of being a face is estimated using principal components analysis. To track a detected face region through the sequence, the connected component that represent the face in the previous frame is projected into the current frame. Using the projected segment as a marker, connected operators extract the actual face region from the skin probability image.

Temporal redundancies are the key to high compression ratios in video coding. In order to improve the prediction gain of motion compensation the concept of long-term memory motion-compensated prediction has been developed. More frames than the previously decoded frame can be taken into account for motion compensation. Usually the motion is estimated in the encoder, where all unencoded frames are accessible. We investigate the applicability of long-term memory motion-compensated prediction to a scalable wavelet video coding scheme using backward motion compensation, where the motion is estimated in both, the encoder and the decoder.

The hybrid coding scheme is employed in all established coding standards. A forward motion vector field is estimated and applied for motion compensation. The remaining prediction error and the motion vectors are transmitted to the decoder. The discrete cosine transform is used for transform coding of the prediction error. The extension of this coding scheme to scalability is not easily achieved and the performance of standard video coders when using scalability options can often be reduced to the performance of simulcast coding. In this paper a hierarchical spatial scalable wavelet video coder is presented. A backward motion compensation scheme is used and no motion vectors have to be transmitted. In this coding scheme the lowpass band of the decomposition is coded first without motion compensation using DPCM. The coarser levels of the signal decomposition which are known to both, the encoder and the decoder, are then employed for motion estimation of the next levels. Due to the hierarchical structure of the presented scheme the impact of quantization is strongly interdependent between the decomposition levels. In contrast to still image coding quantization affects the reconstruction quality as well as the motion estimation and motion compensation efficiency. In this paper the bit allocation for the decomposition levels of the hierarchical wavelet video coder is investigated.

In this paper a H.263 compatible region-of-interest coding system is presented. A face detection and tracking algorithm is applied to find the region-of-interest (ROI). The input image is filtered by a region-adaptive lowpass filter which blurs the image outside the facial region. This region- adaptive lowpass filtering leads to a graceful degradation of the image quality outside the ROI, whereas the ROI remains unchanged. Since no modification is necessary at the encoder, the ROI preprocessing step is compatible to any existing implementation of the H.263 encoder. For face detection, color information is integrated into a detection algorithm based on principal components analysis. Once a face is detected, tracking is based on color information. The ROI filter can be combined with any video sequence coder. Simulation results are given using the H.263 standard. The bitrate can be significantly reduced while retaining a high perceptual quality.