Search

In robot visual servoing system, effective detected the region of interest area (ROI) in target images is the first problem should be solved, but the detection is susceptible to an unstructured environment. In this paper, an instance segmentation network algorithm based on Mask Region Convolutional Neural Networks (Mask R-CNN) framework is proposed for ROI image preprocessing. As instance segmentation technology can distinguish complex environmental information, so first with this advantage filter out message such as shape or similar area to target image, then with semantic segmentation technology add special category labels to the filtered images and distinguish different individual instances in similar categories. Finally through the series steps aforementioned, robot vision system can overcome the impact of environmental factors and identify the target image. The proposed method is used to detect target image under five constraints such as occlusion and reflection, result shows the algorithm can effectively deal with challenges that brought by complex constraints, and even can predict the location data of missing information based on some image information. In addition, based on the algorithm proposed in this paper, we used one seven axis robot visual servo platform, executed visual servoing experiment under different unstructured environments, further verifies the effectiveness of our proposed method.

Scene recognition is the key to achieving accurate and fast indoor positioning. Deep network has become a research central issue lately with its outstanding performance. This paper puts forward an advanced AlexNet network model combined with Harris feature detection, which guides the image processing of the original model according to the detected Harris characteristics, it reduces the randomness error and improves the generalization ability and robustness of the model. In addition, for the campus indoor scene positioning environment, the original structure of the AlexNet network model and the data augmentation method are improved, so that the positioning model can cope with the complex and variable positioning environment, and its accuracy and speed reach a high level. The method can be combined with the existing mainstream visual indoor positioning method to enhance the accurateness and speed of positioning system.

Robot control based on visual information perception is a hot topic in the industrial robot domain and makes robots capable of doing more things in a complex environment. However, complex visual background in an industrial environment brings great difficulties in recognizing the target image, especially when a target is small or far from the sensor. Therefore, target recognition is the first problem that should be addressed in a visual servo system. This paper considers common complex constraints in industrial environments and proposes a You Only Look Once Version 2 Region of Interest (YOLO-v2-ROI) neural network image processing algorithm based on machine learning. The proposed algorithm combines the advantages of YOLO (You Only Look Once) rapid detection with effective identification of ROI (Region of Interest) pooling structure, which can quickly locate and identify different objects in different fields of view. This method can also lead the robot vision system to recognize and classify a target object automatically, improve robot vision system efficiency, avoid blind movement, and reduce the calculation load. The proposed algorithm is verified by experiments. The experimental result shows that the learning algorithm constructed in this paper has real-time image-detection speed and demonstrates strong adaptability and recognition ability when processing images with complex backgrounds, such as different backgrounds, lighting, or perspectives. In addition, this algorithm can also effectively identify and locate visual targets, which improves the environmental adaptability of a visual servo system

With the development of depth data acquisition technologies, access to high-precision depth with more than 8-b depths has become much easier and determining how to efficiently represent and compress high-precision depth is essential for practical depth storage and transmission systems. In this paper, we propose a layered high-precision depth compression framework based on an 8-b image/video encoder to achieve efficient compression with low complexity. Within this framework, considering the characteristics of the high-precision depth, a depth map is partitioned into two layers: 1) the most significant bits (MSBs) layer and 2) the least significant bits (LSBs) layer. The MSBs layer provides rough depth value distribution, while the LSBs layer records the details of the depth value variation. For the MSBs layer, an error-controllable pixel domain encoding scheme is proposed to exploit the data correlation of the general depth information with sharp edges and to guarantee the data format of LSBs layer is 8 b after taking the quantization error from MSBs layer. For the LSBs layer, standard 8-b image/video codec is leveraged to perform the compression. The experimental results demonstrate that the proposed coding scheme can achieve real-time depth compression with satisfactory reconstruction quality. Moreover, the compressed depth data generated from this scheme can achieve better performance in view synthesis and gesture recognition applications compared with the conventional coding schemes because of the error control algorithm.

In this paper, we present a gesture recognition approach to enable real-time manipulating projection content through detecting and recognizing speakers gestures from the depth maps captured by a depth sensor. To overcome the limited measurement accuracy of depth sensor, a robust background subtraction method is proposed for effective human body segmentation and a distance map is adopted to detect human hands. Potential Active Region (PAR) is utilized to ensure the generation of valid hand trajectory to avoid extra computational cost on the recognition of meaningless gestures and three different detection modes are designed for complexity reduction. The detected hand trajectory is temporally segmented into a series of movements, which are represented as Motion History Images. A set-based soft discriminative model is proposed to recognize gestures from these movements. The proposed approach is evaluated on our dataset and performs efficiently and robustly with 90% accuracy.

Mobile applications are becoming increasingly popular. More and more people are using their phones to enjoy ubiquitous location-based services (LBS). The increasing popularity of LBS creates a fundamental problem: mobile localization. Besides traditional localization methods that use GPS or wireless signals, using phone-captured images for localization has drawn significant interest from researchers. Photos contain more scene context information than the embedded sensors, leading to a more precise location description. With the goal being to accurately sense real geographic scene contexts, this article presents a novel approach to mobile visual localization according to a given image (typically associated with a rough GPS position). The proposed approach is capable of providing a complete set of more accurate parameters about the scene geo-context including the real locations of both the mobile user and perhaps more importantly the captured scene, as well as the viewing direction. To figure out how to make image localization quick and accurate, we investigate various techniques for large-scale image retrieval and 2D-to-3D matching. Specifically, we first generate scene clusters using joint geo-visual clustering, with each scene being represented by a reconstructed 3D model from a set of images. The 3D models are then indexed using a visual vocabulary tree structure. Taking geo-tags of the database image as prior knowledge, a novel location-based codebook weighting scheme proposed to embed this additional information into the codebook. The discriminative power of the codebook is enhanced, thus leading to better image retrieval performance. The query image is aligned with the models obtained from the image retrieval results, and eventually registered to a real-world map. We evaluate the effectiveness of our approach using several large-scale datasets and achieving estimation accuracy of a user's location within 13 meters, viewing direction within 12 degrees, and viewing distance within 26 meters. Of particular note is our showcase of three novel applications based on localization results: (1) an on-the-spot tour guide, (2) collaborative routing, and (3) a sight-seeing guide. The evaluations through user studies demonstrate that these applications are effective in facilitating the ideal rendezvous for mobile users.

Unlike traditional RGB video, Kinect-like depth is characterized by its large variation range and instability. As a result, traditional video compression algorithms cannot be directly applied to Kinect-like depth compression with respect to coding efficiency. In this paper, we propose a lossy Kinect-like depth compression framework based on the existing codecs, aiming to enhance the coding efficiency while preserving the depth features for further applications. In the proposed framework, the Kinect-like depth is reformed first by divisive normalized bilateral filter (DNBL) to suppress the depth noises caused by disparity normalization, and then block-level depth padding is implemented for invalid depth region compensation in collaboration with mask coding to eliminate the sharp variation caused by depth measurement failures. Before the traditional video coding, the inter-frame correlation of reformed depth is explored by proposed 2D+T prediction, in which depth volume is developed to simulate 3D volume to generate pseudo 3D prediction reference for depth uniqueness detection. The unique depth region, called active region is fed into the video encoder for traditional intra and inter prediction with residual coding, while the inactive region is skipped during depth coding. The experimental results demonstrate that our compression scheme can save 55%-85% in terms of bit cost and reduce coding complexity by 20%-65% in comparison with the traditional video compression algorithms. The visual quality of the 3D reconstruction is also improved after employing our compression scheme.

Interactive screen sharing requires extremely low latency end-to-end transmission, which in turn requires highly efficient and low-complexity screen compression. In this paper, we present a block-based low-complexity screen compression scheme, in which multiple block modes are adopted to exploit the intra- and inter-frame redundancies. In particular, we classify the intra-coded blocks to pictorial blocks and textual blocks using a proposed fast block classification algorithm, which exploits the discriminative features between the pictorial and the textual blocks. Then, we design a low-complexity, yet efficient, algorithm to compress the textual blocks. We use base colors and escape colors to represent and quantize the textual pixels, which not only achieves high compression ratios but also preserves a high quality on textual pixels. The two-dimensionally predictive index coding and hierarchical pattern coding technologies are used to exploit local spatial correlations and global pattern correlation, respectively. To further utilize the correlation between the luminance and chrominance channels, we propose a joint-channel index coding method. We compare the coding efficiency and the computational complexity of the proposed scheme against the standard image coding schemes such as JPEG, JPEG2000, and PNG, the compound image compressor HJPC, and the popular video coding standard H.264. We also compare the visual quality of the proposed scheme against H.264 intra coding, JPEG2000, and HJPC. The evaluation results show that the proposed scheme achieves superior or comparable compression efficiency with much lower complexity than other schemes in most of the cases.

This paper describes a novel multimodal interactive image search system on mobile devices. The system, the Joint search with ImaGe, Speech, And Word Plus (JIGSAW ${+}$ ), takes full advantage of the multimodal input and natural user interactions of mobile devices. It is designed for users who already have pictures in their minds but have no precise descriptions or names to address them. By describing it using speech and then refining the recognized query by interactively composing a visual query using exemplary images, the user can easily find the desired images through a few natural multimodal interactions with his/her mobile device. Compared with our previous work JIGSAW, the algorithm has been significantly improved in three aspects: 1) segmentation-based image representation is adopted to remove the artificial block partitions; 2) relative position checking replaces the fixed position penalty; and 3) inverted index is constructed instead of brute force matching. The proposed JIGSAW ${+}$ is able to achieve 5% gain in terms of search performance and is ten times faster.

Techniques describe submitting a video clip as a query by a user. A process retrieves images and information associated with the images in response to the query. The process decomposes the video clip into a sequence of frames to extract the features in a frame and to quantize the extracted features into descriptive words. The process further tracks the extracted features as points in the frame, a first set of points to correspond to a second set of points in consecutive frames to construct a sequence of points. Then the process identifies the points that satisfy criteria of being stable points and being centrally located in the frame to represent the video clip as a bag of descriptive words for searching for images and information related to the video clip.

In this paper, we propose a novel approach for high-dynamic-range (HDR) texture compression (TC) suitable for rendering systems of different capacities. Based on the previously proposed DHTC scheme, we first work out an improved joint-channel compression framework, which is robust and flexible enough to provide compressed HDR textures at different bit rates. Then, two compressed HDR texture formats based on the proposed framework are developed. The 8 bpp format is of near lossless visual quality, improving upon known state-of-the-art algorithms. And, to our knowledge, the 4 bpp format is the first workable 4 bpp solution with good quality. We also show that HDR textures in the proposed 4 bpp and 8 bpp formats can compose a layered architecture in the texture consumption pipeline, to significantly save the memory bandwidth and storage in real-time rendering. In addition, the 8 bpp format can also be used to handle traditional low dynamic range (LDR) RGBA textures. Our scheme exhibits a practical solution for compressing HDR textures at different rates and LDR textures with alpha maps.

In this paper, image compression utilizing visual redundancy is investigated. Inspired by recent advancements in image inpainting techniques, we propose an image compression framework towards visual quality rather than pixel-wise fidelity. In this framework, an original image is analyzed at the encoder side so that portions of the image are intentionally and automatically skipped. Instead, some information is extracted from these skipped regions and delivered to the decoder as assistant information in the compressed fashion. The delivered assistant information plays a key role in the proposed framework because it guides image inpainting to accurately restore these regions at the decoder side. Moreover, to fully take advantage of the assistant information, a compression-oriented edge-based inpainting algorithm is proposed for image restoration, integrating pixel-wise structure propagation and patch-wise texture synthesis. We also construct a practical system to verify the effectiveness of the compression approach in which edge map serves as assistant information and the edge extraction and region removal approaches are developed accordingly. Evaluations have been made in comparison with baseline JPEG and standard MPEG-4 AVC/H.264 intra-picture coding. Experimental results show that our system achieves up to 44% and 33% bits-savings, respectively, at similar visual quality levels. Our proposed framework is a promising exploration towards future image and video compression.

The motion compensated temporal filtering (MCTF) technique, which is extensively used in 3-D wavelet video coding schemes nowadays, leads to signal coupling among various spatial subbands because motion alignment is introduced in the temporal filtering. Using all spatial subbands as a reference enables MCTF to fully take advantage of temporal correlation across frames but inevitably brings drifting problem in supporting spatial scalability. This paper first analyzes the signal coupling phenomenon and then proposes a quantitative model to describe signal propagation across spatial subbands during the MCTF process. The signal propagation is modeled for a single MC step based on the shifting effect of wavelet synthesis filters and then it is extended to multilevel MCTF. This model is called subband coupling aware signal propagation (SCASP) model in this paper. Based on the model, we further propose a subband coupling aware rate allocation scheme as one possible solution to the above dilemma in supporting spatial scalability. To find the optimal rate allocation among all subbands for a specified reconstruction resolution, the SCASP model is used to approximate the reconstruction process and derive the synthesis gain of each subband with regard to that reconstruction. Experimental results have fully demonstrated the advantages of our proposed rate allocation scheme in improving both objective and subjective qualities of reconstructed low-resolution video, especially at middle bit rates and high bit rates.

This paper provides an overview of the Barbell lifting coding scheme that has been adopted as common software by the MPEG ad hoc group on further exploration of wavelet video coding. The core techniques used in this scheme, such as Barbell lifting, layered motion coding, 3D entropy coding and base layer embedding, are discussed. The paper also analyzes and compares the proposed scheme with the oncoming scalable video coding (SVC) standard because the hierarchical temporal prediction technique used in SVC has a close relationship with motion compensated temporal lifting (MCTF) in wavelet coding. The commonalities and differences between these two schemes are exhibited for readers to better understand modern scalable video coding technologies. Several challenges that still exist in scalable video coding, e.g., performance of spatial scalable coding and accurate MC lifting, are also discussed. Two new techniques are presented in this paper although they are not yet integrated into the common software. Finally, experimental results demonstrate the performance of the Barbell-lifting coding scheme and compare it with SVC and another well-known 3D wavelet coding scheme, MC embedded zero block coding (MC-EZBC).

Compared with the video programs taken by professionals, home videos are always with low quality content resulted from non-professional capture skills. In this paper, we present a novel spatiotemporal quality assessment scheme in terms of low-level content features for home videos. In contrast to existing frame-level-based quality assessment approaches, a type of temporal segment of video, subshot, is selected as the basic unit for quality assessment. A set of spatiotemporal visual artifacts, regarded as the key factors affecting the overall perceived quality (i.e., unstableness and jerkiness as temporal factors; infidelity, blurring, brightness, and orientation as spatial factors), are mined from each subshot based on particular characteristics of home videos. The relationship between the overall quality metric and these factors are exploited by three different methods, including user study-based, rule-based and learning-based. To validate the proposed scheme, we present a scalable quality-based home video summarization system from a novel perspective-achieving the best visual quality while simultaneously preserving the most informative content. A comparison user study between this system and the attention model-based video skimming approach demonstrated the effectiveness of the proposed quality assessment scheme

The conventional multiview video coding (MVC) schemes, utilizing both neighboring temporal frames and view frames as possible references, have only shown a slight gain over those using temporal frames alone in terms of coding efficiency. The reason for this is that the neighboring temporal frames exhibit stronger correlation with the current frame and the view frames often fail to be selected as references. This paper proposes an elegant MVC framework using high dimensional wavelet, which rightly matches the inherent high dimension property of multiview video. It also makes a better usage of both temporal and view correlations thanks to the hierarchical decomposition. Besides the proposed framework, this paper also investigates MVC coding from the following aspects. First, a disparity-compensated view filter (DCVF) with pixel alignment is proposed, which can accommodate both global and local view disparities among view frames. The proposed DCVF and the existing motion-compensated temporal filter (MCTF) unify the view and temporal decompositions as a generic lifting transform. Second, an adaptive decomposition structure based on the analysis of the temporal and view correlations is proposed. A Lagrangian cost function is derived to determine the optimum decomposition structure. Third, the major components of the proposed MVC coding are figured out, including macroblock type design, subband coefficient coding, and rate allocation. Extensive experiments are carried out on the MPEG 3DAV test sequences and the superior performance of the proposed MVC coding is demonstrated. In addition, the proposed MVC framework can easily support temporal, spatial, SNR, as well as view scalabilities

Two schemes are proposed to efficiently compress video contents into bitstreams that support drift-free switching at predictive frames. They are inspired by the original SP coding scheme presented in the early H.26L. First, we propose a Flex SP coding scheme in which the prediction signal of the SP frame is directly subtracted from the input without quantization and de-quantization. The decoded video quality of the Flex SP scheme is significantly improved when additional inverse discrete cosine transform (DCT) and post-filter are provided. Then, the Hybrid SP scheme is presented to further improve the quality of the display image, as well as the reconstructed reference, by defining two coding modes for each DCT coefficient. Moreover, a rate-distortion algorithm is proposed to determine the coding mode for each coefficient. The bitstreams generated by the two proposed schemes can be decoded successfully by a decoder that complies with MPEG-4 AVC/H.264. In addition, we also investigate how to choose the quantization parameters for switching. An empirical method is proposed to achieve a good tradeoff between high coding efficiency of SP frames and small size of switching bits.

The consideration of better motion compensation techniques for inter-frame prediction is one of the key reasons why the new H.264 (MPEG-4 AVC) video coding standard can achieve considerably better coding efficiency compared to older standards such as MPEG-2/4 and H.263. These include the use of multiple references and block sizes, a better interpolation filter for subpixel motion compensation, and more efficient exploitation of the spatio-temporal correlation between motion vectors of adjacent blocks through the consideration of SKIP and DIRECT modes. In this paper, we introduce additional methods into H.264 that further enhance motion compensation and can lead to additional improvements in coding efficiency. This is achieved by further exploiting motion vector temporal correlation through the introduction of a new DIRECT macroblock type and an enhancement to the existing Skip Macroblock type within Predictive (P) slices. These new macroblock types can lead to a considerable reduction in the bits required for encoding motion information, while retaining or even improving quality under a Rate Distortion Optimization Framework. Our simulation results suggest that the proposed improvements can lead up to 7.6% average bitrate reduction or equivalently 0.39 dB quality improvement over the current H.264 standard.

The rapid development of wireless networks and mobile devices has made mobile video communication a particularly promising service. We previously proposed an effective video form, scalable portrait video. In low-bandwidth conditions, portrait video possesses clearer shape, smoother motion, and much cheaper computational cost than discrete cosine transform (DCT)-based schemes. However, the bit rate of portrait video cannot be accurately modeled by a rate-distortion function as in DCT-based schemes. How to effectively control the bit rate is a hard challenge for portrait video. In this paper, we propose a novel model-based rate-control method. Although the coding parameters cannot be directly calculated from the target bit rate, we build a model between the bit-rate reduction and the percentage of less probable symbols (LPS) based on the principle of entropy coding, which is referred to as the LPS-rate model. We use this model to obtain the desired coding parameters. Experimental results show that the proposed method not only effectively controls the bit rate, but also significantly reduces the number of skipped frames. The principle of this method can also be applied to general bit plane coding in other image processing and video compression technologies.

This paper presents an advanced motion threading technique for improved performance in 3D wavelet coding. First, we extend an original motion threading idea of ours to a lifting- based implementation. Methods for enabling fractional-pixel alignment in motion threading and for processing many-to-one pixel mapping and non-referred pixels are proposed to reduce the wavelet boundary effects. Second, we devise an advanced motion threading technique, in which one set of motion vectors is generated for each temporal layer of wavelet coefficients for temporal scalability. In order to reduce the motion overhead information, especially at low bit rates, several correlated motion prediction modes at the macroblock level are defined to exploit the intra/inter layer correlation in motion vector coding. Finally, rate-distortion optimization is utilized in motion estimation to select the best motion prediction mode for each macroblock. With the new motion threading technique, we are able to achieve 1.5–6.0 dB gain in average PSNR in 3D wavelet coding over our previous implementation of motion threading.

A motion compensated lifting (MCLIFT) framework for the 3D wavelet video coding is proposed in this paper. By using bi-directional motion compensation in each lifting step of the temporal direction, the video frames are effectively de-correlated. With the proper entropy coding and bit-stream packaging schemes, the MCLIFT wavelet video coder is scalable at frame rate and quality level. Experimental results show that the MCLIFT video coder outperforms the 3D wavelet video coder without motion by an average of 0.9-1.3dB, and outperforms MPEG-4 coder by an average of 0.2-0.6dB.

Three-dimensional (3D) wavelet-based scalable video coding provides a viable alternative to standard MC-DCT coding. However, many current 3D wavelet coders experience severe boundary effects across group of pictures (GOP) boundaries. This paper proposes a memory-efficient transform technique via lifting that effectively computes wavelet transforms of a video sequence continuously on the fly, thus eliminating the boundary effects due to limited length of individual GOPs. Coding results show that the proposed scheme completely eliminates the boundary effects and gives superb video playback quality.

This paper proposes an efficient content adaptive screen image scaling scheme for the real-time screen applications like remote desktop and screen sharing. In the proposed screen scaling scheme, a screen content classification step is first introduced to classify the screen image into text and pictorial regions. Afterward, we propose an adaptive shift linear interpolation algorithm to predict the new pixel values with the shift offset adapted to the content type of each pixel. The shift offset for each screen content type is offline optimized by minimizing the theoretical interpolation error based on the training samples respectively. The proposed content adaptive screen image scaling scheme can achieve good visual quality and also keep the low complexity for real-time applications., ICIP 2013

The emerging cloud gaming technology has been growing fast, driving up huge mobile consumer demands. The video streaming based cloud gaming scenario renders the game scenes in the cloud servers, and streams the encoded sequences to the thin clints where the game scenes are decoded and displayed to the players. However, current existing clouding gaming services have some problems, such as the latency and bandwidth limitation. The size of the video stream is usually quite large which requires heavy transmission. Worse still, the frame data rate will burst when the game scenes contain fast translation or rotation, resulting in strong latency problem. In this paper, we propose a novel video streaming based cloud gaming algorithm which reduces the burst of the frame rate significantly. There are mainly two innovations in this paper. Firstly, based on the analysis of the motion estimation strategy in the video codec, we introduce image homography technique for better motion prediction. Meanwhile, according to the rasterization rules of the game engine, we present a special designed interpolation algorithm named Edge Preserved Interpolation (EPI), for more accurate edge interpolation and further reduce the residues in the edge regions. The proposed algorithm is implemented on the x264 platform. Experimental results show that our algorithm has 18.0% BD-rate reduction compared with x264.

In this paper, we propose a high frame rate screen video compression scheme aiming at improving the interactive user experience on screen sharing applications. The proposed screen video compression is performed as two-layer coding: a base layer coding using the conventional video codec and an enhancement layer coding using the proposed open-loop coding scheme. For efficient frame level layer selection and compression, the content update of each frame is evaluated through global motion detection. The screen frame with significant content update is fed to the conventional video encoder in base layer. In contrast, the frame with little update is compressed in enhancement layer in which the duplicate content is indicated by global motion vector and skip flag while the updated content is encoded by distinct intra modes in terms of inherent local features. The experimental results demonstrate that for the screen video containing interaction, the proposed coding scheme can achieve 3.09ms/frame encoding rate and 2.33ms/frame decoding rate with efficient rate distortion performance.

This paper presents an efficient video coding algorithm: Three-dimensional embedded subband coding with optimized truncation (3-D ESCOT), in which coefficients in different subbands are independently coded using fractional bit-plane coding and candidate truncation points are formed at the end of each fractional bit-plane. A rate-distortion optimized truncation scheme is used to multiplex all subband bitstreams together into a layered one. A novel motion threading technique is proposed to form threads along the motion trajectories in a scene. For efficient coding of motion threads, memory-constrained temporal wavelet transforms are applied along entire motion threads. Block-based motion threading is implemented in conjunction with 3-D ESCOT in a real video coder. Extension of 3-D ESCOT to object-based coding is also addressed. Experiments demonstrate that 3-D ESCOT outperforms MPEG-4 for most test sequences at the same bit rate.

Video-object coding is one of the most important functionalities proposed by MPEG-4. We propose a new wavelet method to encode the texture of an arbitrarily shaped object, both for the still and for the video-object. The method uses the shape adaptive wavelet transform (SA-DWT) in MPEG-4 still object coding, but with a computationally more efficient lifting implementation. The transformed object coefficients are then quantized and entropy encoded with a partial bit-plane embedded coder, which greatly improves the coding efficiency. We denote the coding algorithm as the video-object wavelet (VOW) coder. Experimental results show that VOW significantly outperforms MPEG-4 in still-object coding, and achieves a comparable performance in video-object coding in terms of PSNR. Moreover, the VOW decoded object looks better subjectively, with less annoying blocking artifacts than that of MPEG-4.

This paper presents a shape-adaptive wavelet coding technique for coding arbitrarily shaped still texture. This technique includes shape-adaptive discrete wavelet transforms (SA-DWTs) and extensions of zerotree entropy (ZTE) coding and embedded zerotree wavelet (EZW) coding. Shape-adaptive wavelet coding is needed for efficiently coding arbitrarily shaped visual objects, which is essential for object-oriented multimedia applications. The challenge is to achieve high coding efficiency while satisfying the functionality of representing arbitrarily shaped visual texture. One of the features of the SA-DWTs is that the number of coefficients after SA-DWTs is identical to the number of pixels in the original arbitrarily shaped visual object. Another feature of the SA-DWT is that the spatial correlation, locality properties of wavelet transforms, and self-similarity across subbands are well preserved in the SA-DWT. Also, for a rectangular region, the SA-DWT becomes identical to the conventional wavelet transforms. For the same reason, the extentions of ZTE and EZW to coding arbitrarily shaped visual objects carefully treat "don't care" nodes in the wavelet trees. Comparison of shape-adaptive wavelet coding with other coding schemes for arbitrarily shaped visual objects shows that shape-adaptive wavelet coding always achieves better coding efficiency than other schemes. One implementation of the shape-adaptive wavelet coding technique has been included in the new multimedia coding standard MPEG-4 for coding arbitrarily shaped still texture. Software implementation is also available.

This paper presents an algorithm proposal submitted to MPEG-4 for video coding. The proposed algorithm addresses the functionality of improved coding efficiency for compression. It uses vector-based techniques for coding intraframes (the first frame and subsequent refreshing key frames) and motion-compensated difference frames. It uses the same motion estimation and motion compensation techniques as H.263. A video frame (I or P frame) is first decomposed into a set of vector bands using a vector wavelet transform. This stage of vector-based signal processing makes subsequent vector quantization in the vector bands very efficient. Lattice vector quantization is then used in the vector bands. A 100% labeling efficiency is achieved for lattice vector quantization by using a set of generalized labeling algorithms for various important lattices with pyramid and sphere boundaries. Finally, entropy coding is used to code the indexes generated from lattice vector quantization. Our coding results have shown that a gain in peak signal-to-noise ratio (PSNR) up to 8 dB for intraframe coding and up to 6 dB for interframe coding can be achieved over H.263. Subjective quality improvement of the proposed algorithm over H.263 can be easily observed.

In real-time screen remoting system, frame rate is one of essential factors that affect user experience. Therefore, how to compress diversity of screen contents fast and efficiently is a key issue. Existing video codecs such as H.264 are always used in such a system for screen compression. However, arbitrary-sized regions with large motion always exist in typical screen content videos, which lead to a lower encoding speed and higher bit-rate, thereby decrease the frame rate. This paper proposes an efficient motion detection algorithm, which is fast and efficient for large motion regions. In specific, a region-based motion detection is used to find motion vectors instead of traditional block based motion estimation. The motion vectors are then utilized by H.264 encoder for normal motion compensated prediction. Experimental results show that the proposed algorithm can reduce both encoding time and bit-rate significantly.

Modern digital family technology enables people surf the Internet and watch videos via a large screen. This paper proposes an effective scheme for using hand gestures rather than the common remote controllers to browse the web pages on a large TV screen. The proposed scheme models four gesture modes: mouse mode, scroll mode, zoom mode and input mode to help the user browse Web pages naturally and comfortably. Then we combine RGB, depth, motion information and face detection to achieve accurate and real-time hand segmentation and gesture recognition for enabling the four gesture modes. The experiments show the proposed scheme works well in various illumination environments and complicated backgrounds with multiple moving humans. The recognition accuracy of hand shapes in the proposed scheme arrives at 98.50%, and the successful rate for visual digits input reaches 89.00%. Furthermore, the frame rate of the hand-gesture detection and recognition is about 18 fps. Thus the scheme is accurate, real-time and natural.

In this paper, we propose a layered screen video coding scheme based on existing video codecs to leverage hardware video codec for efficient screen video compression. In this scheme, the screen video compression is performed as two-layer coding: base layer coding and enhancement layer coding. The screen video is first analyzed in both frame and block levels for useful temporal and spatial information extraction to assist coding content selection in each layer. The non-skip screen frames are directly compressed by the conventional video codec in the base layer, while the screen contents sensitive to the video quality degradation are selected for improved coding in the enhancement layer. For contents to be enhanced, two intra coding modes are designed to improve the quality of the compressed text/graphics contents and suppress the artifacts introduced by chroma downsampling. The experimental results demonstrate that the screen video quality is improved objectively and subjectively by the proposed scheme with low cost on bitrate and computation complexity. Moreover, an average of 2.95dB coding gain is achieved in high bitrate.

Due to the divergent characteristics of image contents and text contents in screen videos, how to make the joint optimization leveraging rate-distortion (R-D) optimized block classification and bit allocation is critical to the compression performance. In this paper, a general model-based solution is proposed as an attempt to solve this problem. The contributions of this paper are twofold: First, the rate and distortion characteristics of image blocks and text blocks in block-based content-adaptive screen video encoder (BASC) are carefully studied, and the rate and distortion models are proposed. Second, with the proposed rate and distortion models, the R-D optimized block classification and bit allocation are derived using bisection searched Lagrange multiplier method. Experimental results demonstrate that the proposed R-D optimized block classification and bit allocation algorithms are able to adapt to diverse screen contents, which results in a significant gain of up to 4.5dB in PSNR.

Mobile visual search has attracted extensive attention for its huge potential for numerous applications. Research on this topic has been focused on two schemes: sending query images, and sending compact descriptors extracted on mobile phones. The first scheme requires about 30-40KB data to transmit, while the second can reduce the bit rate by 10 times. In this paper, we propose a third scheme for extremely low bit rate mobile visual search, which sends compressed visual words consisting of vocabulary tree histogram and descriptor orientations rather than descriptors. This scheme can further reduce the bit rate with few extra computational costs on the client. Specifically, we store a vocabulary tree and extract visual descriptors on the mobile client. A light-weight pre-retrieval is performed to obtain the visited leaf nodes in the vocabulary tree. The orientation of each local descriptor and the tree histogram are then encoded to be transmitted to server. Our new scheme transmits less than 1KB data, which reduces the bit rate in the second scheme by 3 times, and obtains about 30% improvement in terms of search accuracy over the traditional Bag-of-Words baseline. The time cost is only 1.5 secs on the client and 240 msecs on the server. © 2012 ACM.

This paper presents a block-based low-complexity screen compression scheme. In this scheme, the input screen is split into non-overlapping blocks which are classified as pictorial blocks and textual blocks. We design a low-complexity yet efficient algorithm to compress the textual blocks. We use base colors plus escape pixels to represent and quantize the text pixels, and such quantization mechanism not only achieves high compression efficiency but also keeps low encoding/decoding complexity. We also propose the two-direction predictive index coding and hierarchical pattern coding technologies to utilize the local spatial correlation and the global pattern correlation for text pixels. In addition, to utilize the correlation between the luminance and chrominance channels, we propose a joint-channel index coding method to further improve the compression efficiency. The compression efficiency and complexity of the proposed method is evaluated against the popular image codecs JPEG/JPEG200 and PNG, a recent published screen compression scheme HJPC as well as the popular video codec H.264.

While on the go, more and more people are using their phones to enjoy ubiquitous location-based services (LBS). One of the fundamental problems of LBS is localization. Researchers are now investigating ways to use a phone-captured image for localization as it contains more scene context information than the embedded sensors. In this paper, we present a novel approach to mobile visual localization that accurately senses geographic scene context according to the current image (typically associated with a rough GPS position). Unlike most existing visual localization methods, the proposed approach is capable of providing a complete set of more accurate parameters about the scene geo---including the actual locations of both the mobile user and perhaps more importantly the captured scene along with the viewing direction. Our approach takes advantage of advanced techniques for large-scale image retrieval and 3D model reconstruction from photos. Specifically, we first perform joint geo-visual clustering in the cloud to generate scene clusters, with each scene represented by a 3D model. The 3D scene models are then indexed using a visual vocabulary tree structure. The phone-captured image is used to retrieve the relevant scene models, then aligned with the models, and further registered to the real-world map. Our approach achieves an estimation accuracy of user location within 14 meters, viewing direction within 9 degrees, and scene location within 21 meters. Such a complete set of accurate geo-parameters can lead to various LBS applications for routing that cannot be achieved with most existing methods. In particular, we showcase three novel applications: 1) accurate self-localization, 2) collaborative localization for rendezvous routing, and 3) routing for photographing. The evaluations through user studies indicate these applications are effective for facilitating the perfect rendezvous for mobile users.

The Kinect-like depth compression becomes increasingly important due to the growing requirement on Kinect depth data transmission and storage. Considering the temporal inconsistency of Kinect depth introduced by the random depth measurement error, we propose 2D+T prediction algorithm aiming at fully exploiting the temporal depth correlation to enhance the Kinect depth compression efficiency. In our 2D+T prediction, each depth block is treated as a subsurface, and it the motion trend is detected by comparing with the reliable 3D reconstruction surface, which is integrated by accumulated depth information stored in depth volume. The comparison is implemented under the error tolerant rule, which is derived from the depth error model. The experimental results demonstrate our algorithm can remarkably reduce the bitrate cost and the compression complexity. And the visual quality of the 3D reconstruction results generated from our reconstructed depth is similar to that of traditional video compression algorithm.

In this paper, we deal with the problem of detecting the existence and the location of salient objects for thumbnail images on which most search engines usually perform visual analysis in order to handle web-scale images. Different from previous techniques, such as sliding window-based or segmentation-based schemes for detecting salient objects, we propose to use a learning approach, random forest in our solution. Our algorithm exploits global features from multiple saliency indicators to directly predict the existence and the position of the salient object. To validate our algorithm, we constructed a large image database collected from Bing image search, that contains hundreds of thousands of manually labeled web images. The experimental results using this new database and the resized MSRA database [16] demonstrate that our algorithm outperforms previous state-of-the-art methods.

Compound video compression is crucial for remote control and data assessment. In this paper, we propose a content-aware layered video coding scheme as an attempt to efficiently compress the compound video. In this scheme, the compound video is analyzed and processed progressively at three pyramid levels: block, object and layer. Firstly, the compound video is analyzed by a block type classification technique to access each block's spatial and temporal properties. Secondly, the natural video object is detected adaptively in each frame based on the block type. Finally, the compound video content is distributed into different layers and specifically designed video coding algorithms are employed to compress each layer. Experiments demonstrate that our proposed scheme can preserve the advantages of the employed compression algorithms for each layer and outperform each of them in the compound video compression.

Accuracy and stability of Kinect-like depth data is limited by its generating principle. In order to serve further applications with high quality depth, the preprocessing on depth data is essential. In this paper, we analyze the characteristics of the Kinect-like depth data by examing its generation principle and propose a spatial-temporal denoising algorithm taking into account its special properties. Both the intra-frame spatial correlation and the inter-frame temporal correlation are exploited to fill the depth hole and suppress the depth noise. Moreover, a divisive normalization approach is proposed to assist the noise filtering process. The 3D rendering results of the processed depth demonstrates that the lost depth is recovered in some hole regions and the noise is suppressed with depth features preserved.

The emergence of Kinect facilitates the possibility of depth capture in real-time and with low cost by consumers. It also provides powerful tool and inspiration for researchers to engage in new array of technology development. However, the quality of the depth map captured from Kinect is still inadequate for many applications due to holes, noises and artifacts existing within the depth information. In this paper, we present a texture assisted Kinect depth inpainting framework, aiming at obtaining improved depth information. In this framework, the relationship between texture and depth is investigated, and the characteristics of depth are also exploited. More specifically, texture edge information is extracted to assist the depth inpainting. Furthermore, filtering and diffusion are designed for hole-filling and edge alignment. Experiment results demonstrate that the Kinect depth can be appropriately repaired in both smooth and edge region. Comparing with the original depth, the inpainted depth information enhances the quality of advanced processing such as 3D reconstruction.

Likely variations in the capture conditions (e.g. light, blur, scale, occlusion) and in the viewpoint between the query image and the images in the collection are the factors due to which image retrieval based on the Query-by-Example (QBE) principle is still not reliable enough. In this paper, we propose a novel QBE-based image retrieval system where users are allowed to submit a short video clip as a query to improve the retrieval reliability. Improvement is achieved by integrating the information about different viewpoints and conditions under which object and scene appearances can be captured across different video frames. Rich information extracted from a video can be exploited to generate a more complete query representation than in the case of a single-image query and to improve the relevance of the retrieved results. Our experimental results show that video-based image retrieval (VBIR) is significantly more reliable than the retrieval using a single image as a query.

In this paper, we present a browser-friendly hybrid JPEG/PNG codec for compound images. First we employ a simple yet efficient block classification algorithm to identify the blocks to pictorial and textural ones. And then JPEG is used to encode the pictorial blocks and PNG is used for textual blocks. Our evaluation results show that our codec significantly outperforms JPEG and PNG in terms of rate-distortion performance, and also it outperforms JPEG, JPEG2000 and DjVu in terms of visual quality. Moreover, since JPEG and PNG are naturally supported by modern browsers, the coded images generated from our proposed coder can be natively supported by browsers and possible to be widely deployed in Web applications.

In this paper, we propose a simple and effective scheme for screen image scaling, targeting at real-time applications like remote desktop and screen sharing. To balance visual quality and complexity, we build our scheme upon bilinear interpolation and devise a content adaptive post-processing method. Our scheme keeps the text/graphics regions of screen images as sharp as the original while avoiding magnifying noise and artifacts in the picture regions. Further, the involved operations are quite suitable for hardware acceleration. When implemented on commodity GPUs, our scheme achieves a frame rate more than 300 fps, which is fast enough for practical use.

In this paper, we propose a novel compression scheme for medical images with extended bit depth. Our scheme efficiently utilizes the VOI (value of interest) settings of the medical images, so that only a small part of the coded bit-stream is needed at the decoder to lossless display the original image with recommended VOI parameters. Gray-Golomb coding is used to support VOI progressive transmission. In addition, the pixel domain bit-plane coding enables accurate distortion estimation at pixel level for any truncated bit-stream. Compared to JPEG2000, our scheme achieves much higher VOI performance with very slight overall bit-rate increase.

This paper proposes a novel energy minimization based approach to video scene segmentation. In video content analysis, scene is defined as a set of adjacent shots related to a particular setting or a continuous action in one place. This indicates that not only the global distribution of time and content, but also the local temporal continuity should be taken into account for scene segmentation. Motivated from this fact, we formulate the segmentation procedure as a unified energy minimization framework, in which the global and local constraint is represented by content and context energy, respectively. This energy minimization problem is optimized by two steps in an iterative fashion: first find an initial estimation of scene label for content energy by a generative model; and then iterated conditional modes (ICM) is used for context energy to find the global optimization. Furthermore, a boundary voting procedure is devised to decide the optimal scene boundaries. We apply EMS on an extensive set of home videos and feature movies, and report superior performance compared with several existing key approaches to scene segmentation.

We present a novel 2-D wavelet transform scheme of adaptive directional lifting (ADL) in image coding. Instead of alternately applying horizontal and vertical lifting, as in present practice, ADL performs lifting-based prediction in local windows in the direction of high pixel correlation. Hence, it adapts far better to the image orientation features in local windows. The ADL transform is achieved by existing 1-D wavelets and is seamlessly integrated into the global wavelet transform. The predicting and updating signals of ADL can be derived even at the fractional pixel precision level to achieve high directional resolution, while still maintaining perfect reconstruction. To enhance the ADL performance, a rate-distortion optimized directional segmentation scheme is also proposed to form and code a hierarchical image partition adapting to local features. Experimental results show that the proposed ADL-based image coding technique outperforms JPEG 2000 in both PSNR and visual quality, with the improvement up to 2.0 dB on images with rich orientation features.

In this paper, we present a new image compression scheme, which is specially designed for computer generated compound color images. First we classify the image content into two kinds: text/graphic content and picture content. Then a rate-distortion optimized color quantization algorithm is designed for text/graphic content, which introduces distortion to text content and minimizes the bit rate produced by the following lossless entropy compression algorithm. The picture content is compressed using conventional image algorithms like JPEG. The results show that the proposed scheme achieves better coding performance than other images compression algorithms such as JPEG2000 and DjVu.

In an H.264 video encoder, motion estimation (ME) is the most time-consuming component. The ME process consists of two stages, integer pixel search and fractional pixel search. Since the complexity of integer pixel search has been greatly reduced by numerous fast ME algorithms, the computation overhead required by fractional pixel ME has become relatively significant. To reduce the complexity of fractional pixel ME, we propose a prediction-based directional fractional pixel ME algorithm. We utilize more accurate motion vector predictions and directional search to achieve better computation reduction. We further propose an early termination method to decrease the amount of search. Experimental results show that, compared to the full search sub-pel ME and the fast sub-pel ME proposed in H.264, the proposed method can reduce up to 84% and 74% of fractional pixel search points respectively, with a negligible degradation in quality.

To handle the mismatch problems of spatial-domain motion aligned temporal filtering (MATF) in providing spatial scalability, this paper presents a novel in-scale motion aligned temporal filtering (ISMATF) for highly scalable video coding. For each lifting step of ISMATF, the predict or update is carried out at various resolutions in a scale-by-scale way and the operation at higher resolution is designed to embed the operations at lower resolutions to prevent redundancy in frame representation. The proposed ISMATF has several main features, which as a whole differentiates it from other schemes (e.g. spatial-domain MATF, in-band MATF, straightforward multi-resolution MATF) and makes it effective in video coding with spatial scalabilities. These features include 1) Multi-resolution lifting with perfect reconstruction capability; 2) non-redundancy in frame representation, which favors highly scalable video coding; 3) temporal filtering performed in the image-domain of corresponding resolution, which is different from in-band schemes and preserves high efficiency of motion compensation; 4) no mismatch between the encoder and decoder, no matter which resolution is decoded. The proposed ISMATF scheme solves the mismatch problems of SDMATF at low resolution while maintains a good performance at high resolution. It can outperform redundant multi-resolution MATF schemes by up to 1.0 dB at high resolution.