Search

Display technologies have evolved over the years. It is critical to develop practical HDR capturing, processing, and display solutions to bring 3D technologies to the next level. Depth estimation of multi-exposure stereo image sequences is an essential task in the development of cost-effective 3D HDR video content. In this paper, we develop a deep architecture for multi-exposure stereo depth estimation. The proposed architecture has two novel components. First, the stereo matching technique used in traditional stereo depth estimation is revamped. For the stereo depth estimation component of our architecture, a mono-to-stereo transfer learning approach is deployed. The proposed formulation circumvents the cost volume construction requirement, which is replaced by a dual-encoder single-decoder CNN with different weights for feature fusion. EfficientNet-based blocks are used to learn the disparity. Secondly, we combine disparity maps obtained from the stereo images at different exposure levels using a robust disparity feature fusion approach. The disparity maps obtained at different exposures are merged using weight maps calculated for different quality measures. The final predicted disparity map obtained is more robust and retains best features that preserve the depth discontinuities. The proposed CNN offers flexibility to train using standard dynamic range stereo data or with multi-exposure low dynamic range stereo sequences. In terms of performance, the proposed model surpasses state-of-the-art monocular and stereo depth estimation methods, both quantitatively and qualitatively, on challenging Scene flow and differently exposed Middlebury stereo datasets. The architecture performs exceedingly well on complex natural scenes, demonstrating its usefulness for diverse 3D HDR applications. [ABSTRACT FROM AUTHOR]

We present a marker-less AR/DR system that can replace the arm of the user with a virtual bionic prosthesis in real time including finger tracking. For this, we use a mixed reality HMD that provides the user with a stereo image based on video-see-through (VST). We apply chroma-keying to remove the user's arm from each captured image and input reconstructed background information into the removed pixels. Before rendering the prosthesis model into the image, we re-target motion capture data of the user's hand to the kinematic skeleton of the prosthesis to match the current hand pose. This system opens new research possibilities on self- and other-perception of bionic bodies. In a first evaluation study of the system, we propose that users perceive the virtual prosthesis model as a part of their body (i.e., that they experience a sense of ownership). We tested this assumption in a laboratory study with 27 individuals who used the system to perform a series of simple tasks in AR with their prosthesis. We measured body ownership and other measures with self-reports. In support of the hypothesis, users experienced a sense of body ownership. Also, a feeling of self-presence is induced during the task, and participants rated the overall experience as positive. [ABSTRACT FROM AUTHOR]

We present a novel and high-performance framework for 3D object detection using stereo vision. This framework incorporates direct instance depth estimation efficiently, improving the accuracy of the final 3D object detection. Instead of detecting objects separately in the left and right images of a stereo display, we exploit a modified 2D object detector that takes only the left image as input to generate union 2D bounding boxes for both images, and to predict the depth of the 3D box center for each object. Using the union 2D boxes, we propose a direct instance-level depth estimation network, taking the estimated depth as guidance, to predict the desired depths for pixels belonging to an object from a small search range. This approach greatly improves the efficiency and accuracy of 3D detection. Moreover, we design an adaptive spatial feature aggregation module that can weaken the effect of background points and automatically integrate important instance features to achieve accurate 3D object localization. Our method outperforms current state-of-the-art stereo-based 3D detection methods on the KITTI benchmark dataset, and it can efficiently employ a shared model for 3D multi-class detection. Code will be available at https://github.com/xjtuwh/iDepNet/tree/master. [ABSTRACT FROM AUTHOR]

Various approaches using neural networks have been proposed to address multi-view stereopsis, but most of them lack capabilities to handle large textureless regions. Hence, a compelling matching network learning comprehensive information from stereo images is constructed to enforce smoothness constraints globally. Trained over binocular stereo datasets only, we show that the network can directly handle the DTU multi-view stereo dataset. When merging together multiple depth maps obtained using either stereo matching, an additional point consolidation procedure is often needed for removing outliers and better aligning individual patches. A second network that consolidates 3D point clouds through directly projecting individual 3D points based on point distributions in their neighborhoods is proposed. Unlike the matching network, this network is trained on local information and is scalable for handling point clouds of any sizes and is capable of processing selected areas of interest as well. Quantitative evaluation on the DTU dataset demonstrates our two networks together can generate point clouds comparable to existing state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Three-dimensional (3D) reconstruction of objects and scenes from camera images is of great interests due to its wide applications. Reconstruction based on feature point correspondence is an established approach. Existing research on curve-based reconstruction is limited to certain type of curves and constrained by case-dependent reconstruction accuracy. In view of that, this paper developed a new method to reconstruct general 3D curves from stereo images. Under proposal, a B-spline curve fitting is applied to sets of 2D edge points extracted from acquired stereo images. Derived approximating parametric curves are then used to construct conic surfaces. Further, robust iterative algorithms are developed to get intersection of corresponding conic surfaces to recover 3D curve. Due to the method design, proposed approach can reconstruct general 3D curves including both open and closed curves. The curve fitting technique and developed robust algorithms can meet accuracy requirement of many real applications. Validity of the proposed method is verified through experiments on a cylinder and teacup in laboratory and a real forging within a workshop. [ABSTRACT FROM AUTHOR]

We propose an encoder–decoder CNN framework to predict depth from one single image in a self-supervised manner. To this aim, we design three kinds of encoder based on the recent advanced deep neural network and one kind of decoder which can generate multiscale predictions. Eight loss functions are designed based on the proposed encoder–decoder CNN framework to validate the performance. For training, we take rectified stereo image pairs as input of the CNN, which is trained by reconstructing image via learning multiscale disparity maps. For testing, the CNN can estimate the accurate depth information by inputting only one single image. We validate our framework on two public datasets in contrast to the state-of-the-art methods and our designed different variants, and the performance of different encoder–decoder architectures and loss functions is evaluated to obtain the best combination, which proves that our proposed method performs very well for single-image depth estimation without the supervision of ground truth. [ABSTRACT FROM AUTHOR]

Stereo image segmentation is the key technology in stereo image editing with the population of stereoscopic 3D media. Most previous methods perform stereo image segmentation on both views relying primarily on per-pixel disparities, which results in the segmentation quality closely connected to the accuracy of the disparities. Therefore, a mechanism to remove the errors of the disparities are highly demanded. To date, there's no such a method yet that can produce accurate disparity maps. In this paper, we propose a novel convolutional neural network (CNN)-based framework, which will automatically propagate the segmentation result from one view to the other. The key problem of accurate stereo image segmentation is the missing of occluded regions. To solve this problem, the CNN architecture is proposed to improve the stereo segmentation performance. In order to address the inevitable inaccuracies problem of the disparities computed from a stereo pair of images, we utilize the coherent disparity propagation that propagates segment result via those pixels with coherent disparities. The pixels by coherent disparity propagation and the high confidence pixels of the object probability map produced by the CNN architecture are then used to generate the initial reliable pixels to perform an energy minimization framework-based segmentation. A comprehensive evaluations and comparisons on Middlebury and Adobe benchmark datasets show the effectiveness of our proposed method in terms of high-quality results, and the robustness against various types of inputs. [ABSTRACT FROM AUTHOR]

Numerous stereo matching algorithms have been proposed to obtain disparity estimation for a single pair of stereo images. However, simply even applying the best of them to temporal frames independently, i.e., without considering the temporal consistency between consecutive frames, may suffer from the undesirable artifacts. Here, we proposed an adaptive, spatiotemporally consistent, constraints-based systematic method that generates spatiotemporally consistent disparity maps for stereo video image sequences. Firstly, a reliable temporal neighborhood is used to enforce the "self-similarity" assumption and prevent errors caused by false optical flow matching from propagating between consecutive frames. Furthermore, we formulate the adaptive temporal predicted disparity map as prior knowledge of the current frame. It is used as a soft constraint to enhance the temporal consistency of disparities, increase the robustness to luminance variance, and restrict the range of the potential disparities for each pixel. Additionally, to further strengthen smooth variation of disparities, the adaptive temporal segment confidence is incorporated as a soft constraint to reduce ambiguities caused by under- and over-segmentation, and retain the disparity discontinuities that align with 3D object boundaries from geometrically smooth, but strong color gradient regions. Experimental evaluations demonstrate that our method significantly improves the spatiotemporal consistency both quantitatively and qualitatively compared with other state-of-the-art methods on the synthetic DCB and realistic KITTI datasets. [ABSTRACT FROM AUTHOR]

To make a matching algorithm to satisfy the requirements of high precision and anti-interference, a novel stereo-matching algorithm with efficient matching cost and adaptive guided image filter is proposed. Firstly, we adopt a modified Census transform with a local texture metric to compute the initial cost. It can make full use of the cross-correlation information between pixels. Meanwhile, we incorporate the Census, color and gradient costs as a mixed matching cost algorithm. Then, we aggregate the costs with guided image filter based on adaptive rectangular support window instead of the traditional fixed support window. The variable kernel window is constructed by the local color similarity and spatial distance. In this way, less occluded points will be included in the support region. On this basis, we adopt integral image to further speed up the computation of this step. Finally, the initial disparity of each pixel is selected using winner takes all optimization and the final disparity maps are gained after post-processing. The experimental results demonstrate that the proposed algorithm not only achieves an average error rate of 5.22 % on the Middlebury stereo benchmark data set, but can also overcome the influence of illumination distortion in the matching effectively. [ABSTRACT FROM AUTHOR]