Search

Stereo confidence estimation aims to estimate the reliability of the estimated disparity by stereo matching. Different from the previous methods that exploit the limited input modality, we present a novel method that estimates confidence map of an initial disparity by making full use of tri-modal input, including matching cost, disparity, and color image through deep networks. The proposed network, termed as Locally Adaptive Fusion Networks (LAF-Net), learns locally-varying attention and scale maps to fuse the tri-modal confidence features. Moreover, we propose a knowledge distillation framework to learn more compact confidence estimation networks as student networks. By transferring the knowledge from LAF-Net as teacher networks, the student networks that solely take as input a disparity can achieve comparable performance. To transfer more informative knowledge, we also propose a module to learn the locally-varying temperature in a softmax function. We further extend this framework to a multiview scenario. Experimental results show that LAF-Net and its variations outperform the state-of-the-art stereo confidence methods on various benchmarks.

We describe a face recognition system based on two different three-dimensional (3D) sensors. We use 3D sensors to overcome the pose-variation problems that cannot be effectively solved in two-dimensional images. We acquire input data based on a structured-light system and compare it with 3D faces that are obtained from a 3D laser scanner. Owing to differences in structure between the input data and the 3D faces, we can generate the range images of the probe and stored images. For estimating the head pose of input data, we propose a novel error-compensated singular-value decomposition that geometrically estimates the rotation angle. Face recognition rates obtained with principal component analysis on various range images of 35 people in different poses show promising results.

A two-stage algorithm is proposed for locating smooth and detailed disparity vector fields in a stereo image pair. The algorithm consists of hierarchical disparity estimation using a region-dividing technique and edge-preserving regularization. The hierarchical region-dividing disparity estimation increases the efficiency and reliability of the estimation process. At the second stage, the vector fields are regularized with an energy model that produces smooth fields while preserving discontinuities resulting from object boundaries. The minimization problem is addressed by solving a corresponding partial differential equation using a finite-difference method. Experiments show that the proposed algorithm provides accurate and spatially correlated disparity vector fields in various types of stereo images, even in the case of images with large displacements. © 2004 Society of Photo-Optical Instrumentation Engineers.

We propose rapid hybrid interpolation methods that employ more than one interpolation algorithm, and choose the most appropriate interpolation algorithm that provides high-quality images with a minimum number of operations. Although a complex interpolation algorithm generally outperforms a simple interpolation algorithm, the differences are negligible for most pixels, with major differences occurring around edges. Thus, in the proposed algorithm, we first apply a test to predict which interpolation is most appropriate for a given pixel in terms of complexity and performance. Then, a simple interpolation algorithm is used for pixels for which the simple interpolation algorithm provides acceptable performances, and a complex interpolation algorithm is used for pixels for which the complex interpolation algorithm significantly outperforms the simple interpolation algorithm. Consequently, it is possible to obtain high-quality images without significantly increasing the number of operations. © 2004 Society of Photo-Optical Instrumentation Engineers.

A multiview sequence coding technique is proposed in this paper. A multiview sequence encoder must be able to reduce redundancies in both the time and view domains because the amount of data increases as the number of views (or cameras) increases. We define a new coding structure, a group of a group of pictures (GGOP), which is compatible with MPEG-2 and is flexible with a baseline distance. The GGOP could have several possible types, e.g., one-I type, two-I type, etc., according to the number of reference frame sequences. This permits one of these types to be selected according to the baseline distance among the cameras. In addition, the GGOP is designed for MPEG-2 compatibility. The proposed multiview sequence encoder consists of preprocessing stages, disparity estimation/compensation, motion estimation/compensation, residual coding, rate control, and entropy coding. It generates two types of bit streams, the main bit stream and the auxiliary bit stream. The main bit stream contains information concerning reference sequences including I-pictures, in order to maintain MPEG-2 compatibility. The auxiliary bit stream contains information concerning the remaining multiview sequences except for the sequences that include I-pictures. The proposed encoder shows improvements in compression ratio as well as in peak signal-to-noise ratio (PSNR) compared with conventional methods. The dependency of the GGOP on the baseline distance between cameras is also confirmed.

The goal of this paper is to determine dense disparity and motion fields jointly and efficiently, given a sequence of stereoscopic images. At each time instant, the motion field of left sequences is estimated by using the current disparity field, the next disparity field, and the motion field of right sequences. The improved concepts for the proposed joint estimation are based on an edge-preserving regularization, an ambiguous vector region detection, and a median filtering for block-based vector generation. The reported approach is verified by processing a set of stereo sequences. Results are given with real stereoscopic data.

We extend our previous paper, where the focus was on error-resilient zerotree wavelet image coding on noisy channels. This concept is extended to video coding using a 3-D wavelet transform. The proposed video coding system consists primarily of a 3-D wavelet transform, coding via the 3-D set-partitioning in hierarchical trees (SPIHT) algorithm, and bit partitioning followed by bit reorganization procedure. Simulation results show that the proposed coders outperform the existing error-resilient coders from the standpoint of both noise-free channels and noisy channels. In addition, we confirm that the proposed algorithm is more error-resilient than the conventional 3-D zerotree wavelet video coding techniques for various channel-error conditions as evidenced by both objective and subjective evaluations. © 2001 Society of Photo-Optical Instrumentation Engineers.

A zero-tree wavelet image coding technique with resiliency to transmission errors that typically occur on noisy channels is proposed. Key tools employed consist of the bit partitioning algorithm, and the bit reorganization algorithm, which is called the error resilient entropy code (EREC). To take full advantage of the bit reorganization algorithm, the bit partitioning algorithm divides the data into separate code blocks, although the zero-tree wavelet coding algorithm is not a block-based compression method. The bit reorganization algorithm requires a very low redundancy for the sequential transmission of variable length blocks, offering a virtually guaranteed code and block synchronization. The wavelet transform and the wavelet packet transform are used. Simulation results verify the error resiliency of the proposed algorithm for image coding. The experimental data show that the proposed coders outperform the existing error resilient coders over both noise-free channels and noisy channels. In addition, it is confirmed that the proposed algorithm is more error resilient than previously reported error-resilient coders for various channel error conditions. © 2001 Society of Photo-Optical Instrumentation Engineers.

One of the important issues for a next generation broadcasting system is how to compress a massive amount of threedimensional (3D) video efficiently. In this paper, we propose a geometry compensation method for 3D video coding exploiting color videos, depth videos and camera parameters. In the proposed method, we first generate a compensated view, which is located at the geometrically same position with the current view, using depth and camera parameters of neighboring views. Then, the compensated view is used as a reference picture to reduce the inter-view redundancies such as disparity and motion vectors. Furthermore, considering the direction of hole-regions, we propose a hole-filling method for picture of P-view to fill up the holes based on the neighboring background pixels. The experimental results show that the proposed algorithm increases BD-PSNRs up to 0.22dB and 0.63dB for P- and B-views, respectively. Meanwhile, we achieved up to 6.28% and 18.32% BD bit-rates gain for P- and B- views, respectively.

The inter-view prediction is used as well as the temporal prediction in order to exploit both the temporal and inter-view redundancies in multiview video coding. Accordingly, the multiview video coding has two types of motion vectors that are the temporal motion vector and the disparity vector, respectively. The disparity vector is generally uncorrelated with the temporal motion vector. However, they are used together to predict the motion vector regardless of their types, therefore an efficiency of the conventional predictive coding of multiview video coding is decreased. In order to increase the accuracy of the predicted motion vector, a new motion vector prediction method including virtual temporal motion vector and virtual disparity vector is proposed for both the multiview video and multiview video plus depth formats. The experimental results show that the proposed method can reduce the coding bitrates by 6.5% in average and 14.6% at maximum in terms of Bjontegaard metric compared to the conventional method.

Multi-view video coding (MVC) is a video coding standard developed by MPEG and VCEG for multi-view video. It showed average PSNR gain of 1.5dB compared with view-independent coding by H.264/AVC. However, because resolutions of multi-view video are getting higher for more realistic 3D effect, high performance video codec is needed. MVC adopted hierarchical B-picture structure and inter-view prediction as core techniques. The hierarchical B-picture structure removes the temporal redundancy, and the inter-view prediction reduces the inter-view redundancy by compensated prediction from the reconstructed neighboring views. Nevertheless, MVC has inherent limitation in coding efficiency, because it is based on H.264/AVC. To overcome the limit, an enhanced video codec for multi-view video based on Key Technology Area (KTA) is proposed. KTA is a high efficiency video codec by Video Coding Expert Group (VCEG), and it was carried out for coding efficiency beyond H.264/AVC. The KTA software showed better coding gain than H.264/AVC by using additional coding techniques. The techniques and the inter-view prediction are implemented into the proposed codec, which showed high coding gain compared with the view-independent coding result by KTA. The results presents that the inter-view prediction can achieve higher efficiency in a multi-view video codec based on a high performance video codec such as HEVC.

In this paper, we proposed a visual fatigue monitoring system based on eye-movement and eye-blink detection. It analyzes the eye-movement and number of blinks based on the assumption that saccade movement of the eye decreases and the number of eye blink increases when visual fatigue of viewer is accumulated. The proposed system has an infrared single camera and an infrared light source. Then, the pupil of the eye can be detected by applying binary threshold to Purkinje image. The threshold is automatically selected by two constraints which are the eccentricity of ellipse fitting and the size of the pupil. Finally, total amount of eye movement and the number of eye blinks are measured by using the positions of the pupil. The results were obtained while watching stereoscopic videos after personal calibration procedure. The results show that saccade movement of the eye decreases as the visual fatigue of the viewer is accumulated. However, the number of eye blinks shows large variance along the time axis which implies it is not proper for visual fatigue monitoring system.

We propose a new deinterlacing algorithm with selective motion compensation. It has been reported that deinterlacing methods using motion compensation produce significantly improved results, although they tend to yield undesired results in fast moving areas. This is due to weak correlations between the previous and current frames. The proposed algorithm solves this problem by selectively applying motion-compensated deinterlacing. We first apply intrafield interpolation in the spatial domain, and then selectively apply motion compensations according to the type of motion vectors. Experimental results show that the proposed method produces noticeably improved performance compared to existing motion-compensated deinterlacing methods.

We propose a fast disparity estimation algorithm using background registration and object segmentation for stereo sequences from fixed cameras. Dense background disparity information is calculated in an initialization step, so that only disparities of moving object regions are updated in the main process. We propose a real-time segmentation technique using background subtraction and interframe differences, and a hierarchical disparity estimation using a region-dividing technique and shape-adaptive matching windows. Experimental results show that the proposed algorithm provides accurate disparity vector fields with an average processing speed of 15 frames/s for 320×240 stereo sequences on an ordinary PC.