Search

Abstract With ongoing social progress, three-dimensional (3D) video is becoming increasingly prevalent in everyday life. As a key component of 3D video technology, depth video plays a crucial role by providing information about the distance and spatial distribution of objects within a scene. This study focuses on deep video encoding and proposes an efficient encoding method that integrates the Convolutional Neural Network (CNN) with a hyperautomation mechanism. First, an overview of the principles underlying CNNs and the concept of hyperautomation is presented, and the application of CNNs in the intra-frame prediction module of video encoding is explored. By incorporating the hyperautomation mechanism, this study emphasizes the potential of Artificial Intelligence to enhance encoding efficiency. Next, a CNN-based method for variable-resolution intra-frame prediction of depth video is proposed. This method utilizes a multi-level feature fusion network to reconstruct coding units. The effectiveness of the proposed variable-resolution coding technique is then evaluated by comparing its performance against the original method on the high-efficiency video coding (HEVC) test platform. The results demonstrate that, compared to the original test platform method (HTM-16.2), the proposed method achieves an average Bjøntegaard delta bit rate (BDBR) savings of 8.12% across all tested video sequences. This indicates a significant improvement in coding efficiency. Furthermore, the viewpoint BDBR loss of the variable-resolution coding method is only 0.15%, which falls within an acceptable margin of error. This suggests that the method is both stable and reliable in viewpoint coding, and it performs well across a broad range of quantization parameter settings. Additionally, compared to other encoding methods, the proposed approach exhibits superior peak signal-to-noise ratio, structural similarity index, and perceptual quality metrics. This study introduces a novel and efficient approach to 3D video compression, and the integration of CNNs with hyperautomation provides valuable insights for future innovations in video encoding.

Abstract Digital quantification of gait can be used to measure aging- and disease-related decline in mobility. Gait performance also predicts prognosis, disease progression, and response to therapies. Most gait analysis systems require large amounts of space, resources, and expertise to implement and are not widely accessible. Thus, there is a need for a portable system that accurately characterizes gait. Here, depth video from two portable cameras accurately reconstructed gait metrics comparable to those reported by a pressure-sensitive walkway. 392 research participants walked across a four-meter pressure-sensitive walkway while depth video was recorded. Gait speed, cadence, and step and stride durations and lengths strongly correlated (r > 0.9) between modalities, with root-mean-squared-errors (RMSE) of 0.04 m/s, 2.3 steps/min, 0.03 s, and 0.05–0.08 m for speed, cadence, step/stride duration, and step/stride length, respectively. Step, stance, and double support durations (gait cycle percentage) significantly correlated (r > 0.6) between modalities, with 5% RMSE for step and stance and 10% RMSE for double support. In an exploratory analysis, gait speed from both modalities significantly related to healthy, mild, moderate, or severe categorizations of Charleson Comorbidity Indices (ANOVA, Tukey’s HSD, p

Digital quantification of gait can be used to measure aging- and disease-related decline in mobility. Gait performance also predicts prognosis, disease progression, and response to therapies. Most gait analysis systems require large amounts of space, resources, and expertise to implement and are not widely accessible. Thus, there is a need for a portable system that accurately characterizes gait. Here, depth video from two portable cameras accurately reconstructed gait metrics comparable to those reported by a pressure-sensitive walkway. 392 research participants walked across a four-meter pressure-sensitive walkway while depth video was recorded. Gait speed, cadence, and step and stride durations and lengths strongly correlated (r > 0.9) between modalities, with root-mean-squared-errors (RMSE) of 0.04 m/s, 2.3 steps/min, 0.03 s, and 0.05–0.08 m for speed, cadence, step/stride duration, and step/stride length, respectively. Step, stance, and double support durations (gait cycle percentage) significantly correlated (r > 0.6) between modalities, with 5% RMSE for step and stance and 10% RMSE for double support. In an exploratory analysis, gait speed from both modalities significantly related to healthy, mild, moderate, or severe categorizations of Charleson Comorbidity Indices (ANOVA, Tukey's HSD, p < 0.0125). These findings demonstrate the viability of using depth video to expand access to quantitative gait assessments. [ABSTRACT FROM AUTHOR]

The depth map compression of 3D High-Efficiency Video Coding (3D-HEVC) inherits prediction structure adopted by HEVC and develops supplementary coding modes to better express depth image. These new coding modes combine with existing technology implement high coding efficiency, but lead to an extremely huge increase in coding time. This article proposes a rapid coding method for depth map in 3D-HEVC. The proposed scheme utilizes the Bayesian decision rule and the correlations between corresponding texture video and spatially adjacent treeblocks to analyze the treeblock features of depth map. According to the analysis, we propose two approaches, including early SKIP/Merge mode selection, and adaptive CU pruning termination. Simulation consequences illustrate that this paper can save 51.2% complexity while maintaining the rate distortion (RD) performance.

Abstract In this paper, two methods of zoom motion estimation for color and depth videos by using depth information are proposed. Color and depth videos are independently estimated for zoom motion. Zoom for color video is scaled by spatial domain, and depth video is scaled by both spatial and depth domains. For color video, instead of existing methods of zoom motion estimation that apply all of possible zoom ratios for a current block, the zoom ratio of the proposed method is determined as the ratio of the average depth values of the current and reference blocks. Then, the reference block is resized by multiplying the zoom ratio and the reference block is mapped to the current block. For depth video, the reference block is first scaled in the spatial direction by the same methodology used for color video and then scaled by a distance ratio from a camera to the objects. Compared to the conventional motion estimation method, the proposed method reduces MSE by up to about 30% for the color video and up to about 85% for the depth video.

Most existing deep-learning-based fall detection methods use either 2D neural network without considering movement representation sequences, or whole sequences instead of only those in the fall period. These characteristics result in inaccurate extraction of human action features and failure to detect falls due to background interferences or activity representation beyond the fall period. To alleviate these problems, a skeleton-based 3D consecutive-low-pooling neural network (S3D-CNN) for fall detection is proposed in this paper. In the S3D-CNN, an activity feature clustering selector is designed to extract the skeleton representation in depth videos using pose estimation algorithm and form optimized skeleton sequence of fall period. A 3D consecutive-low-pooling (3D-CLP) neural network is proposed to process these representation sequences by improving network in terms of layer number, pooling kernel size, and single input frame number. The proposed method is evaluated on public and self-collected datasets respectively, outperforming the existing methods. [ABSTRACT FROM AUTHOR]

In this study, an estimation method for human height is proposed using color and depth information. Color images are used for deep learning by mask R-CNN to detect a human body and a human head separately. If color images are not available for extracting the human body region due to low light environment, then the human body region is extracted by comparing between current frame in depth video and a pre-stored background depth image. The topmost point of the human head region is extracted as the top of the head and the bottommost point of the human body region as the bottom of the foot. The depth value of the head top-point is corrected to a pixel value that has high similarity to a neighboring pixel. The position of the body bottom-point is corrected by calculating a depth gradient between vertically adjacent pixels. Two head-top and foot-bottom points are converted into 3D real-world coordinates using depth information. Two real-world coordinates estimate human height by measuring a Euclidean distance. Estimation errors for human height are corrected as the average of accumulated heights. In experiment results, we achieve that the estimated errors of human height with a standing state are 0.7% and 2.2% when the human body region is extracted by mask R-CNN and the background depth image, respectively. [ABSTRACT FROM AUTHOR]

In the stereoscopic or multiview display, the depth video illustrates visual distances between objects and camera. To promote the computational efficiency of depth video encoder, we exploit the intra prediction of depth videos under Versatile Video Coding (VVC) and observe a diverse distribution of intra prediction modes with different coding unit sizes. We propose a hybrid scheme to further boost fast depth video coding. In the first stage, we adaptively predict the HADamard (HAD) costs of intra prediction modes and initialize a candidate list according to the HAD costs. Then, the candidate list is further improved by considering the probability distribution of candidate modes with different CU sizes. Finally, early termination of CU splitting is performed at each CU depth level based on the Bayesian theorem. Our proposed method is incorporated into VVC intra prediction for fast coding of depth videos. Experiments with 7 standard sequences and 4 Quantization parameters (Qps) validate the efficiency of our method. [ABSTRACT FROM AUTHOR]

In this paper, two methods of zoom motion estimation for color and depth videos by using depth information are proposed. Color and depth videos are independently estimated for zoom motion. Zoom for color video is scaled by spatial domain, and depth video is scaled by both spatial and depth domains. For color video, instead of existing methods of zoom motion estimation that apply all of possible zoom ratios for a current block, the zoom ratio of the proposed method is determined as the ratio of the average depth values of the current and reference blocks. Then, the reference block is resized by multiplying the zoom ratio and the reference block is mapped to the current block. For depth video, the reference block is first scaled in the spatial direction by the same methodology used for color video and then scaled by a distance ratio from a camera to the objects. Compared to the conventional motion estimation method, the proposed method reduces MSE by up to about 30% for the color video and up to about 85% for the depth video. [ABSTRACT FROM AUTHOR]

In 3-D applications, a high-quality depth video codec with low complexity is in great demand for consumer devices, which generally have limited computational resources and energy. Based on the compressed sensing theory, we propose a low complexity depth video codec to compress depth videos effectively. The proposed codec decomposes depth blocks via an adaptive wavelet decomposition algorithm by following the principle of local entropy minimization. The decomposition can reduce the amount of local data, and separate depth blocks by frequency precisely. In order to reduce temporal redundancy, a block average value-based fast motion estimation scheme is also designed. Further, a joint optimization method is proposed to select the best combination of quantization parameter and measurement rate for each block. The experimental results demonstrate that the proposed codec, compared with H.265 and H.264, achieves BD-PSNR improvement up to 1.34 dB and 4.28 dB at most respectively in “AllIntra” mode. In “IPPP” mode, the proposed codec also achieves BD-PSNR improvement up to 0.62 dB and 1.79 dB on average respectively. Moreover, compared with other methods, the complexity and energy consumption of the proposed codec is much lower, which is adapted to consumer devices well. [ABSTRACT FROM AUTHOR]

In this study, an estimation method for human height is proposed using color and depth information. Color images are used for deep learning by mask R-CNN to detect a human body and a human head separately. If color images are not available for extracting the human body region due to low light environment, then the human body region is extracted by comparing between current frame in depth video and a pre-stored background depth image. The topmost point of the human head region is extracted as the top of the head and the bottommost point of the human body region as the bottom of the foot. The depth value of the head top-point is corrected to a pixel value that has high similarity to a neighboring pixel. The position of the body bottom-point is corrected by calculating a depth gradient between vertically adjacent pixels. Two head-top and foot-bottom points are converted into 3D real-world coordinates using depth information. Two real-world coordinates estimate human height by measuring a Euclidean distance. Estimation errors for human height are corrected as the average of accumulated heights. In experiment results, we achieve that the estimated errors of human height with a standing state are 0.7% and 2.2% when the human body region is extracted by mask R-CNN and the background depth image, respectively.

Two action recognition approaches that utilize depth videos and skeletal information are proposed in this paper. Dense trajectories are used to represent the depth video data. Skeletal data are represented by vectors of skeleton joints positions and their forward differences in various temporal scales. The extracted features are encoded using either Bag of Words (BoW) or Vector of Locally Aggregated Descriptors (VLAD) approaches. Finally, a Support Vector Machine (SVM) is used for classification. Experiments were performed on three datasets, namely MSR Action3D, MSR Action Pairs and Florence3D in order to measure the performance of the methods. The proposed approaches outperform all state of the art action recognition methods that operate on depth video/skeletal data in the most challenging and fair experimental setup of the MSR Action3D dataset. Moreover, they achieve 100% correct recognition in the MSR Action Pairs dataset and the highest classification rate among all compared methods on the Florence3D dataset. [ABSTRACT FROM AUTHOR]

Multi-view depth is crucial for describing positioning information in 3D space for virtual reality, free viewpoint video, and other interaction- and remote-oriented applications. However, in cases of lossy compression for bandwidth limited remote applications, the quality of multi-view depth video suffers from quantization errors, leading to the generation of obvious artifacts in consequent virtual view rendering during interactions. Considerable efforts must be made to properly address these artifacts. In this paper, we propose a cross-view multi-lateral filtering scheme to improve the quality of compressed depth maps/videos within the framework of asymmetric multi-view video with depth compression. Through this scheme, a distorted depth map is enhanced via non-local candidates selected from current and neighboring viewpoints of different time-slots. Specifically, these candidates are clustered into a macro super pixel denoting the physical and semantic cross-relationships of the cross-view, spatial and temporal priors. The experimental results show that gains from static depth maps and dynamic depth videos can be obtained from PSNR and SSIM metrics, respectively. In subjective evaluations, even object contours are recovered from a compressed depth video. We also verify our method via several practical applications. For these verifications, artifacts on object contours are properly managed for the development of interactive video and discontinuous object surfaces are restored for 3D modeling. Our results suggest that the proposed filter outperforms state-of-the-art filters and is suitable for use in multi-view color plus depth-based interaction- and remote-oriented applications. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Data Acquisition & Processing / Shu Ju Cai Ji Yu Chu Li is the property of Editorial Department of Journal of Nanjing University of Aeronautics & Astronautics and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

We propose an intelligent system to detect human fall events using a physics-based myoskeletal simulation, detecting falls by comparing the simulation with a fall velocity profile using the Hausdorff distance. Previous methods of fall detection are trained using recordings of acted falls which are limited in number, body variability and type of fall and can be unrepresentative of real falls. The paper demonstrates that the use of fall recordings are unnecessary for modelling the fall as the simulation engine can produce a variety of fall events customised to an individual’s physical characteristics using myoskeletal models of different morphology, without pre-knowledge of the falling behaviour. To validate this methodological approach, the simulation is customised by the person’s height, modelling a rigid fall type. This approach allows the detection to be tailored to cohorts in the population (such as the elderly or the infirm) that are not represented in existing fall datasets. The method has been evaluated on several publicly available datasets which show that our method outperforms the results of previously reported research in fall detection. Finally, our approach is demonstrated to be robust to occlusions that hide up to 50% of a fall, which increases the applicability of automatic fall detection in a real-world environment such as the home. [ABSTRACT FROM AUTHOR]

Due to the limits of frame rate sampling like time-of-flight and transmission bandwidth constraints of depth video, depth frame rate up-conversion is imperative. Most researches interpolate the depth frames using the motion information of the texture images, which require well-rectified images. They usually find dense correspondences to synthesize an in-between frame. Estimating the correspondences is sensitive to occlusion, disocclusion, and color or luminance changes. This paper proposes a phase-based method to interpolate depth frames without explicitly finding correspondences. Combining top-down phase difference preservation and bottom-up phase difference correction, varying amounts of motions are well modeled in a complex-steerable pyramid decomposition. The pixel amplitude is also shifted and interpolated based on the phase information. Finally, a depth value correction is employed to correct the inaccurate depth value. Experimental results show that the depth video interpolated by our proposed approach provides better subjective and objective qualities. ©2018 SPIE and IS&T [DOI: 10.1117/1.JEI.27.4.043036] [ABSTRACT FROM AUTHOR]

In this paper, a novel rate-control algorithm based on the region adaptive R- $\lambda $ model is proposed for depth maps coding. First, in order to obtain an accurate rate control for depth maps coding, a modified frame level bit allocation method based on coding bits statistical distribution of depth maps is proposed. Second, considering that different areas in a depth map have an imparity effect on virtual view rendering, the blocks of the depth map are divided into two types, namely, interested blocks for virtual view rending (IBV) and noninterested blocks for virtual view rending (NIBV). Then, two different R- $\lambda $ models are derived for IBV and NIBV, respectively. The optimal bitrates for IBV and NIBV are determined by solving an optimization problem. After that, based on the regional R- $\lambda $ models, the optimal Lagrange multipliers are calculated for both IBV and NIBV. Finally, the largest coding unit (LCU) level rate control is performed by adaptively adjusting the Lagrange multiplier to avoid blocking artifacts and smooth the quality of coding. Experimental results demonstrate that the proposed method can achieve considerable BD-PSNR gains compared with the unified rate-quantization model and conventional R- $\lambda $ model-based algorithms in terms of rendered virtual views quality. [ABSTRACT FROM AUTHOR]

This survey outlines the use of computer vision in Image and video processing in multidisciplinary applications; either in academia or industry, which are active in this field. The scope of this paper covers the theoretical and practical aspects in image and video processing in addition of computer vision, from essential research to evolution of application. In this work a various subjects of image processing and computer vision will be demonstrated, these subjects are spanned from the evolution of mobile augmented reality (MAR) applications, to augmented reality under 3D modeling and real time depth imaging, video processing algorithms will be discussed to get higher depth video compression, beside that in the field of mobile platform an automatic computer vision system for citrus fruit has been implemented, where the Bayesian classification with Boundary Growing to detect the text in the video scene. Also the paper illustrates the usability of the handed interactive method to the portable projector based on augmented reality. [ABSTRACT FROM AUTHOR]

Computer vision-based human activity recognition (HAR) has become very famous these days due to its applications in various fields such as smart home healthcare for elderly people. A video-based activity recognition system basically has many goals such as to react based on people's behavior that allows the systems to proactively assist them with their tasks. A novel approach is proposed in this work for depth video based human activity recognition using joint-based motion features of depth body shapes and Deep Belief Network (DBN). From depth video, different body parts of human activities are segmented first by means of a trained random forest. The motion features representing the magnitude and direction of each joint in next frame are extracted. Finally, the features are applied for training a DBN to be used for recognition later. The proposed HAR approach showed superior performance over conventional approaches on private and public datasets, indicating a prominent approach for practical applications in smartly controlled environments. [ABSTRACT FROM AUTHOR]

In this paper, we present a novel rate control method with optimized region-based bit allocation for depth video coding. First, a synthetic view distortion oriented segmentation method is proposed to divide depth video into different regions, including texture areas and smooth areas. Then, the expanded exponential distortion-rate (D-R) models and power quantization parameter-rate (QP-R) models are investigated to simulate rate-distortion (R-D) characteristics of different regions. Finally, an optimal bit allocation scheme is developed to adaptively allocate target bit with the division. Experimental results on various video sequences demonstrate that the proposed algorithm achieves excellent R-D efficiency and bit rate accuracy compared with benchmark algorithms. [ABSTRACT FROM AUTHOR]

In free viewpoint video system, color video and the associated depth video are utilized to synthesize arbitrary virtual viewpoint. Hence, high quality of depth video is a necessity for virtual view rendering. However, depth video estimated by depth estimate software is inconsistent and inaccurate which decreases the quality of virtual views. To solve the problem, a depth video processing algorithm is proposed in this paper. Firstly, depth video is divided into five clusters adaptively by Fuzzy C-means clustering method. Meanwhile, the edges of depth video are detected and expanded into 8×8 block-wise. Secondly, for pixels in non-edge regions of depth video, a cluster dependent filtering method is adopted according to the feature of each cluster. Finally, corners in the corresponding texture video are detected. For pixels in edge regions of depth video, a corner-aware filtering method is used. Experimental results show that the proposed algorithm enhances the depth video and significantly improves the quality of the virtual views. The peak signal to noise ratio of virtual views rendered by using the proposed algorithm is 0.43 dB higher than that of the virtual views rendered by using the original depth video. The proposed algorithm also outperforms Martin's algorithm in terms of virtual view rendering performance. [ABSTRACT FROM AUTHOR]

Multi-frame super-resolution is the process of recovering a high resolution image or video from a set of captured low resolution images. Super-resolution approaches have been largely explored in 2-D imaging. However, their extension to depth videos is not straightforward due to the textureless nature of depth data, and to their high frequency contents coupled with fast motion artifacts. Recently, few attempts have been introduced where only the super-resolution of static depth scenes has been addressed. In this work, we propose to enhance the resolution of dynamic depth videos with non-rigidly moving objects. The proposed approach is based on a new data model that uses densely upsampled, and cumulatively registered versions of the observed low resolution depth frames. We show the impact of upsampling in increasing the sub-pixel accuracy and reducing the rounding error of the motion vectors. Furthermore, with the proposed cumulative motion estimation, a high registration accuracy is achieved between non-successive upsampled frames with relative large motions. A statistical performance analysis is derived in terms of mean square error explaining the effect of the number of observed frames and the effect of the super-resolution factor at a given noise level. We evaluate the accuracy of the proposed algorithm theoretically and experimentally as function of the SR factor, and the level of noise contamination. Experimental results on both real and synthetic data show the effectiveness of the proposed algorithm on dynamic depth videos as compared to state-of-art methods. [ABSTRACT FROM AUTHOR]

Facial expression recognition (FER) plays a very significant role in computer vision, pattern recognition, and image processing applications such as human computer interaction as it provides sufficient information about emotions of people. For video-based facial expression recognition, depth cameras can be better candidates over RGB cameras as a person's face cannot be easily recognized from distance-based depth videos hence depth cameras also resolve some privacy issues that can arise using RGB faces. A good FER system is very much reliant on the extraction of robust features as well as recognition engine. In this work, an efficient novel approach is proposed to recognize some facial expressions from time-sequential depth videos. First of all, efficient Local Binary Pattern (LBP) features are obtained from the time-sequential depth faces that are further classified by Generalized Discriminant Analysis (GDA) to make the features more robust and finally, the LBP-GDA features are fed into Hidden Markov Models (HMMs) to train and recognize different facial expressions successfully. The depth information-based proposed facial expression recognition approach is compared to the conventional approaches such as Principal Component Analysis (PCA), Independent Component Analysis (ICA), and Linear Discriminant Analysis (LDA) where the proposed one outperforms others by obtaining better recognition rates. [ABSTRACT FROM AUTHOR]

Copyright of China Sciencepaper is the property of China Sciencepaper and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Spatial and temporal inconsistency of depth video deteriorates encoding efficiency in three dimensional video systems. A depth video processing algorithm based on human perception is presented. Firstly, a just noticeable rendering distortion (JNRD) model is formulated by combining the analyses of the influence of depth distortion on virtual view rendering with human visual perception characteristics. Then, depth video is processed based on the JNRD model from two aspects, spatial and temporal correlation enhancement. During the process of spatial correlation enhancement, depth video is segmented into edge, foreground, and background regions, and smoothened by Gaussian and mean filters. The operations of the temporal correlation enhancement include temporal–spatial transpose (TST), temporal smoothing filter and inverse TST. Finally, encoding and virtual view rendering experiments are conducted to evaluate the proposed algorithm. Experimental results show that the proposed algorithm can greatly reduce the bit rate while it maintains the quality of virtual view. [ABSTRACT FROM AUTHOR]

The rapid spreading of RGB-D cameras has led to wide applications of 3D videos in both academia and industry, such as 3D entertainment and 3D visual understanding. Under these circumstances, extensive research efforts have been dedicated to RGB-D camera -- oriented topics. In these topics, quality promotion of depth videos with the temporal characteristic is emerging and important. Due to the limited exposure time of RGB-D cameras, object movement can easily lead to motion blurs in intensive images, which can further result in obvious artifacts (holes or fake boundaries) in the corresponding depth frames. With regard to this problem, we propose a depth error elimination method based on time series analysis to remove the artifacts in depth images. In this method, we first locate the regions with erroneous depths in intensive images by using motion blur detection based on a time series analysis model. This is based on the fact that the depth image is calculated by intensive color images that are captured synchronously by RGB-D cameras. Then, the artifacts, such as holes or fake boundaries, are fixed by a depth error elimination method. To evaluate the performance of the proposed method, we conducted experiments on 250 images. Experimental results demonstrate that the proposed method can locate the error regions correctly and eliminate these artifacts effectively. The quality of depth video can be improved significantly by using the proposed method. [ABSTRACT FROM AUTHOR]

In this paper, a depth camera-based novel method is proposed to recognize several facial expressions from depth video. At first, Radon Transformation (RT) is done to extract features from the time-sequential depth faces that are further improved by Generalized Discriminant Analysis (GDA) to generate more robust features and then, Hidden Markov Models (HMMs) are applied to train and recognize different facial expressions successfully. Performance of the proposed facial expression recognition shows the superiority over conventional RGB camera-based approaches. [ABSTRACT FROM AUTHOR]

Human action recognition in videos is useful for many applications. However, there still exist huge challenges in real applications due to the variations in the appearance, lighting condition and viewing angle, of the subjects. In this consideration, depth data have advantages over red, green, blue (RGB) data because of their spatial information about the distance between object and viewpoint. Unlike existing works, we utilize the 3-D point cloud, which contains points in the 3-D real-world coordinate system to represent the external surface of human body. Specifically, we propose a new robust feature, the body surface context (BSC), by describing the distribution of relative locations of the neighbors for a reference point in the point cloud in a compact and descriptive way. The BSC encodes the cylindrical angular of the difference vector based on the characteristics of human body, which increases the descriptiveness and discriminability of the feature. As the BSC is an approximate object-centered feature, it is robust to transformations including translations and rotations, which are very common in real applications. Furthermore, we propose three schemes to represent human actions based on the new feature, including the skeleton-based scheme, the random-reference-point scheme, and the spatial-temporal scheme. In addition, to evaluate the proposed feature, we construct a human action dataset by a depth camera. Experiments on three datasets demonstrate that the proposed feature outperforms RGB-based features and other existing depth-based features, which validates that the BSC feature is promising in the field of human action recognition. [ABSTRACT FROM PUBLISHER]

Recently, depth data is widely used in computer vision applications such as detection and tracking, which shows great promises in complicated environments due to its complementary natures to RGB data. However, previous works mostly use depth as an auxiliary cue of RGB data and overlook its inherent advantage on motion detection. Intrinsically different from RGB data, points in depth map essentially represents 3-D positions in the world, so depth video represents the variation of these “positions,” which is motion. Motivated by this, we proposed a novel motion detection scheme based on RS-Trigger temporal logic which best fits nature of depth data on motion detection. The proposed algorithm can fast detect motion regions in the scene without statistics of background and prior knowledge of objects to detect. In following refinement modules, a depth-invariant density-constant projection is proposed which contributes to a fast spatial clustering and accurate segmentation, for it transforms dense 3-D points cloud to depth-invariant 2-D map with density-constance, not only it overcomes depth-dependent sampling of depth sensor, but also overcomes the common `scale problem' in 2-D image analysis, which makes it easy to set system parameters to de-noise and pop-out motion regions. Experimental results validate its effectiveness and efficiency. [ABSTRACT FROM AUTHOR]

Abstract: Depth video is vital to the representation of the dynamic 3-D video, which is the fundamental for the rapidly growing 3-D video applications. The issues of accuracy and temporal consistency are the main concerns of the researches on depth videos. In previous works, the stereo matching methods with global optimization can generate accurate and dense depth videos. However, the global optimization is computationally intensive, and the temporal consistency is obtained with difficulty in the optimization procedure. In this paper, a Bayesian framework is proposed to generate accurate and temporal consistent dense depth videos in an efficient way. Firstly, the spatial and temporal correlations in 3-D videos from different viewpoints are used to generate the candidates for depth, and these correlations are further measured by extracted features. These features are adopted to estimate the risk of initial depth in our Bayesian framework. Then, a two-stage decision method is designed to select candidates of depth value in the initial depth map with the minimum risk probability. Finally, depth videos are refined by improved graph cuts algorithm with global optimization. An improved graph construction method is designed and applied on graph cuts algorithm to reduce the number of nodes in graph and thus the complexity of global optimization. The experimental results demonstrate that the proposed algorithm can achieve accurate depth videos with higher efficiency up to 68.14% than traditional methods. [Copyright &y& Elsevier]

We propose a novel framework for upconversion of depth video resolution in both spatial and time domains considering spatial and temporal coherences. Although the Time-of-Flight (TOF) sensor which is widely used in computer vision fields provides depth video in realtime, it also provides a low resolution and a low frame-rate depth video. We propose a cheaper solution that enhances depth video obtained from a TOF sensor by combining it with a Charge-coupled Device (CCD) camera in 3D contents which consist of 2D-plus-depth. Temporal fluctuation problems are also considered for temporally consistent framerate up-conversion. It is important to maintain temporal coherence in depth video, because temporal fluctuation problems may cause eye fatigue and increase bit rates on video coding. We propose a Motion Compensated Frame Interpolation (MCFI) using reliable and rich motion information from a CCD camera and 3-dimensional Joint Bilateral Up-sampling (3D JBU) extended into the temporal domain of depth video. Experimental results show that depth video obtained by the proposed method provides satisfactory quality1. [ABSTRACT FROM AUTHOR]