Search

Stereo confidence estimation aims to estimate the reliability of the estimated disparity by stereo matching. Different from the previous methods that exploit the limited input modality, we present a novel method that estimates confidence map of an initial disparity by making full use of tri-modal input, including matching cost, disparity, and color image through deep networks. The proposed network, termed as Locally Adaptive Fusion Networks (LAF-Net), learns locally-varying attention and scale maps to fuse the tri-modal confidence features. Moreover, we propose a knowledge distillation framework to learn more compact confidence estimation networks as student networks. By transferring the knowledge from LAF-Net as teacher networks, the student networks that solely take as input a disparity can achieve comparable performance. To transfer more informative knowledge, we also propose a module to learn the locally-varying temperature in a softmax function. We further extend this framework to a multiview scenario. Experimental results show that LAF-Net and its variations outperform the state-of-the-art stereo confidence methods on various benchmarks.

In this paper, we propose a fast disparity and motion estimation for multi-view video coding (MVC). When implementing MVC, one of the most critical problems is heavy computational complexity caused by the large amount of information in multi-view sequences. Hence, a fast algorithm is essential. To reduce this computational complexity, we adaptively controlled a search range considering the reliability of each macroblock. In order to estimate this reliability, we calculated the difference between the predicted vectors that were obtained from different methods. When working with conventional encoders, vectors can be predicted using median filtering from causal blocks. Moreover, we calculated another predicted vector using multi-view camera geometry or the relationship between the disparity and motion vectors. We assumed that this difference indicated the reliability of the current macroblock. By using these properties, we were able to determine new search range and reduce the number of searching points within the limited window. The proposed MVC system was tested with several multiview sequences to evaluate performance. Experimental results showed that the proposed algorithm was able to reduce processing time by maximumly 70-80% in estimation process. Index Terms--Joint disparity/motion estimation, fast motion estimation, Multi-view Video Coding

This paper presents an efficient joint disparity-motion estimation algorithm and a fast motion estimation algorithm in an MPEG-4 compatible stereoscopic sequence codec for 3D broadcasting. In stereoscopic sequence coding, the critical problems to be solved are double bandwidth and computational complexity. However, the previous trials to reduce coding bits have increased computational complexity. In this paper, we try to implement fast algorithms for stereoscopic sequences while maintaining high coding efficiency. Joint disparity-motion estimation can increase coding efficiency and reduce the computational complexity of the stereoscopic codec by using the relationship between the disparity and motion fields. The disparity vectors are estimated by using the left and right motion vectors and the previous disparity vectors in each time frame. In order to obtain more accurate disparity vectors, we employ the spatial prediction process, after joint estimation. From joint estimation and spatial prediction, we can obtain the accurately predicted disparity vectors and then increase the coding efficiency from those predicted vectors. Moreover, we propose a fast motion estimation algorithm which utilizes correlation between the motion vectors of the neighboring blocks. We confirmed that the PSNR of the proposed method is 1.0-2.0dB higher than ones of conventional methods. In addition, the processing time is reduced by almost 1/10. Index Terms--Joint disparity-motion estimation, fast motion estimation, stereo sequences

This paper presents advanced rate control technologies for 3D-HDTV. We remodel the quadratic rate-distortion model for 3D multiview sequences based on their picture types. The proposed method consists of two levels for more accurate bits rate control. In a frame level, it sets quantization parameters based on HVS (Human Visual System) and reduces error bits by remodeling rate-quantization. In a MB level, rate control is activated to provide stricter buffer regulations and higher bit rate encoding based on the target bits and the quantization parameters calculated at the frame level. The proposed algorithm shows improvements in rate control as well as in PSNR compared to the conventional methods. It also provides more accurate rate control and higher image quality than TM5 and TMN8 at various bits rates. Index Terms--Rate control, Quality control, Multiview sequence.

In this paper, we propose a new deinterlacing algorithm, which uses directional interpolation and motion compensation. In the proposed method, intrafield interpolation is first performed in the direction that shows the highest correlation. Second, motion estimation is performed between two fields of the same parity. The motion vector is further refined in half-pixel accuracy. In the conventional motion compensated methods, a prefilter such as line averaging, is applied to interpolate missing lines prior to motion estimation between opposite parity fields. The proposed method does not require this prefilter since block matching is performed between the same parity fields. Finally, we apply a test and use either directional interpolation or motion compensated interpolation depending on the test result. Experimental results demonstrate that the proposed method provides better performances than conventional deinterlacing algorithms. Index Terms--deinterlacing, directional interpolation, interlaced scan, motion compensation.

In this paper, we present a color interpolation technique based on artificial neural networks for a single-chip CCD (charge-coupled device) camera with a Bayer color filter array (CFA). Single-chip digital cameras use a color filter array and an interpolation method in order to produce high quality color images from sparsely sampled images. We have applied 3-layer feedforward neural networks in order to interpolate a missing pixel from surrounding pixels. And we compare the proposed method with conventional interpolation methods such as the bilinear interpolation method and cubic spline interpolation method. Experiments show that the proposed interpolation algorithm based on neural networks provides a better performance than the conventional interpolation algorithms.

We describe a face recognition system based on two different three-dimensional (3D) sensors. We use 3D sensors to overcome the pose-variation problems that cannot be effectively solved in two-dimensional images. We acquire input data based on a structured-light system and compare it with 3D faces that are obtained from a 3D laser scanner. Owing to differences in structure between the input data and the 3D faces, we can generate the range images of the probe and stored images. For estimating the head pose of input data, we propose a novel error-compensated singular-value decomposition that geometrically estimates the rotation angle. Face recognition rates obtained with principal component analysis on various range images of 35 people in different poses show promising results.

A two-stage algorithm is proposed for locating smooth and detailed disparity vector fields in a stereo image pair. The algorithm consists of hierarchical disparity estimation using a region-dividing technique and edge-preserving regularization. The hierarchical region-dividing disparity estimation increases the efficiency and reliability of the estimation process. At the second stage, the vector fields are regularized with an energy model that produces smooth fields while preserving discontinuities resulting from object boundaries. The minimization problem is addressed by solving a corresponding partial differential equation using a finite-difference method. Experiments show that the proposed algorithm provides accurate and spatially correlated disparity vector fields in various types of stereo images, even in the case of images with large displacements. © 2004 Society of Photo-Optical Instrumentation Engineers.

We propose rapid hybrid interpolation methods that employ more than one interpolation algorithm, and choose the most appropriate interpolation algorithm that provides high-quality images with a minimum number of operations. Although a complex interpolation algorithm generally outperforms a simple interpolation algorithm, the differences are negligible for most pixels, with major differences occurring around edges. Thus, in the proposed algorithm, we first apply a test to predict which interpolation is most appropriate for a given pixel in terms of complexity and performance. Then, a simple interpolation algorithm is used for pixels for which the simple interpolation algorithm provides acceptable performances, and a complex interpolation algorithm is used for pixels for which the complex interpolation algorithm significantly outperforms the simple interpolation algorithm. Consequently, it is possible to obtain high-quality images without significantly increasing the number of operations. © 2004 Society of Photo-Optical Instrumentation Engineers.

A multiview sequence coding technique is proposed in this paper. A multiview sequence encoder must be able to reduce redundancies in both the time and view domains because the amount of data increases as the number of views (or cameras) increases. We define a new coding structure, a group of a group of pictures (GGOP), which is compatible with MPEG-2 and is flexible with a baseline distance. The GGOP could have several possible types, e.g., one-I type, two-I type, etc., according to the number of reference frame sequences. This permits one of these types to be selected according to the baseline distance among the cameras. In addition, the GGOP is designed for MPEG-2 compatibility. The proposed multiview sequence encoder consists of preprocessing stages, disparity estimation/compensation, motion estimation/compensation, residual coding, rate control, and entropy coding. It generates two types of bit streams, the main bit stream and the auxiliary bit stream. The main bit stream contains information concerning reference sequences including I-pictures, in order to maintain MPEG-2 compatibility. The auxiliary bit stream contains information concerning the remaining multiview sequences except for the sequences that include I-pictures. The proposed encoder shows improvements in compression ratio as well as in peak signal-to-noise ratio (PSNR) compared with conventional methods. The dependency of the GGOP on the baseline distance between cameras is also confirmed.

The goal of this paper is to determine dense disparity and motion fields jointly and efficiently, given a sequence of stereoscopic images. At each time instant, the motion field of left sequences is estimated by using the current disparity field, the next disparity field, and the motion field of right sequences. The improved concepts for the proposed joint estimation are based on an edge-preserving regularization, an ambiguous vector region detection, and a median filtering for block-based vector generation. The reported approach is verified by processing a set of stereo sequences. Results are given with real stereoscopic data.

We extend our previous paper, where the focus was on error-resilient zerotree wavelet image coding on noisy channels. This concept is extended to video coding using a 3-D wavelet transform. The proposed video coding system consists primarily of a 3-D wavelet transform, coding via the 3-D set-partitioning in hierarchical trees (SPIHT) algorithm, and bit partitioning followed by bit reorganization procedure. Simulation results show that the proposed coders outperform the existing error-resilient coders from the standpoint of both noise-free channels and noisy channels. In addition, we confirm that the proposed algorithm is more error-resilient than the conventional 3-D zerotree wavelet video coding techniques for various channel-error conditions as evidenced by both objective and subjective evaluations. © 2001 Society of Photo-Optical Instrumentation Engineers.

A zero-tree wavelet image coding technique with resiliency to transmission errors that typically occur on noisy channels is proposed. Key tools employed consist of the bit partitioning algorithm, and the bit reorganization algorithm, which is called the error resilient entropy code (EREC). To take full advantage of the bit reorganization algorithm, the bit partitioning algorithm divides the data into separate code blocks, although the zero-tree wavelet coding algorithm is not a block-based compression method. The bit reorganization algorithm requires a very low redundancy for the sequential transmission of variable length blocks, offering a virtually guaranteed code and block synchronization. The wavelet transform and the wavelet packet transform are used. Simulation results verify the error resiliency of the proposed algorithm for image coding. The experimental data show that the proposed coders outperform the existing error resilient coders over both noise-free channels and noisy channels. In addition, it is confirmed that the proposed algorithm is more error resilient than previously reported error-resilient coders for various channel error conditions. © 2001 Society of Photo-Optical Instrumentation Engineers.

One of the important issues for a next generation broadcasting system is how to compress a massive amount of threedimensional (3D) video efficiently. In this paper, we propose a geometry compensation method for 3D video coding exploiting color videos, depth videos and camera parameters. In the proposed method, we first generate a compensated view, which is located at the geometrically same position with the current view, using depth and camera parameters of neighboring views. Then, the compensated view is used as a reference picture to reduce the inter-view redundancies such as disparity and motion vectors. Furthermore, considering the direction of hole-regions, we propose a hole-filling method for picture of P-view to fill up the holes based on the neighboring background pixels. The experimental results show that the proposed algorithm increases BD-PSNRs up to 0.22dB and 0.63dB for P- and B-views, respectively. Meanwhile, we achieved up to 6.28% and 18.32% BD bit-rates gain for P- and B- views, respectively.

The inter-view prediction is used as well as the temporal prediction in order to exploit both the temporal and inter-view redundancies in multiview video coding. Accordingly, the multiview video coding has two types of motion vectors that are the temporal motion vector and the disparity vector, respectively. The disparity vector is generally uncorrelated with the temporal motion vector. However, they are used together to predict the motion vector regardless of their types, therefore an efficiency of the conventional predictive coding of multiview video coding is decreased. In order to increase the accuracy of the predicted motion vector, a new motion vector prediction method including virtual temporal motion vector and virtual disparity vector is proposed for both the multiview video and multiview video plus depth formats. The experimental results show that the proposed method can reduce the coding bitrates by 6.5% in average and 14.6% at maximum in terms of Bjontegaard metric compared to the conventional method.

Multi-view video coding (MVC) is a video coding standard developed by MPEG and VCEG for multi-view video. It showed average PSNR gain of 1.5dB compared with view-independent coding by H.264/AVC. However, because resolutions of multi-view video are getting higher for more realistic 3D effect, high performance video codec is needed. MVC adopted hierarchical B-picture structure and inter-view prediction as core techniques. The hierarchical B-picture structure removes the temporal redundancy, and the inter-view prediction reduces the inter-view redundancy by compensated prediction from the reconstructed neighboring views. Nevertheless, MVC has inherent limitation in coding efficiency, because it is based on H.264/AVC. To overcome the limit, an enhanced video codec for multi-view video based on Key Technology Area (KTA) is proposed. KTA is a high efficiency video codec by Video Coding Expert Group (VCEG), and it was carried out for coding efficiency beyond H.264/AVC. The KTA software showed better coding gain than H.264/AVC by using additional coding techniques. The techniques and the inter-view prediction are implemented into the proposed codec, which showed high coding gain compared with the view-independent coding result by KTA. The results presents that the inter-view prediction can achieve higher efficiency in a multi-view video codec based on a high performance video codec such as HEVC.

In this paper, we proposed a visual fatigue monitoring system based on eye-movement and eye-blink detection. It analyzes the eye-movement and number of blinks based on the assumption that saccade movement of the eye decreases and the number of eye blink increases when visual fatigue of viewer is accumulated. The proposed system has an infrared single camera and an infrared light source. Then, the pupil of the eye can be detected by applying binary threshold to Purkinje image. The threshold is automatically selected by two constraints which are the eccentricity of ellipse fitting and the size of the pupil. Finally, total amount of eye movement and the number of eye blinks are measured by using the positions of the pupil. The results were obtained while watching stereoscopic videos after personal calibration procedure. The results show that saccade movement of the eye decreases as the visual fatigue of the viewer is accumulated. However, the number of eye blinks shows large variance along the time axis which implies it is not proper for visual fatigue monitoring system.

We propose a new deinterlacing algorithm with selective motion compensation. It has been reported that deinterlacing methods using motion compensation produce significantly improved results, although they tend to yield undesired results in fast moving areas. This is due to weak correlations between the previous and current frames. The proposed algorithm solves this problem by selectively applying motion-compensated deinterlacing. We first apply intrafield interpolation in the spatial domain, and then selectively apply motion compensations according to the type of motion vectors. Experimental results show that the proposed method produces noticeably improved performance compared to existing motion-compensated deinterlacing methods.

We propose a fast disparity estimation algorithm using background registration and object segmentation for stereo sequences from fixed cameras. Dense background disparity information is calculated in an initialization step, so that only disparities of moving object regions are updated in the main process. We propose a real-time segmentation technique using background subtraction and interframe differences, and a hierarchical disparity estimation using a region-dividing technique and shape-adaptive matching windows. Experimental results show that the proposed algorithm provides accurate disparity vector fields with an average processing speed of 15 frames/s for 320×240 stereo sequences on an ordinary PC.