Search

Estimating depth from RGB images is a long-standing ill-posed problem, which has been explored for decades by the computer vision, graphics, and machine learning communities. Among the existing techniques, stereo matching remains one of the most widely used in the literature due to its strong connection to the human binocular system. Traditionally, stereo-based depth estimation has been addressed through matching hand-crafted features across multiple images. Despite the extensive amount of research, these traditional techniques still suffer in the presence of highly textured areas, large uniform regions, and occlusions. Motivated by their growing success in solving various 2D and 3D vision problems, deep learning for stereo-based depth estimation has attracted a growing interest from the community, with more than 150 papers published in this area between 2014 and 2019. This new generation of methods has demonstrated a significant leap in performance, enabling applications such as autonomous driving and augmented reality. In this paper, we provide a comprehensive survey of this new and continuously growing field of research, summarize the most commonly used pipelines, and discuss their benefits and limitations. In retrospect of what has been achieved so far, we also conjecture what the future may hold for deep learning-based stereo for depth estimation research. [ABSTRACT FROM AUTHOR]

This paper presents a novel framework to recover detailed avatar from a single image. It is a challenging task due to factors such as variations in human shapes, body poses, texture, and viewpoints. Prior methods typically attempt to recover the human body shape using a parametric-based template that lacks the surface details. As such resulting body shape appears to be without clothing. In this paper, we propose a novel learning-based framework that combines the robustness of the parametric model with the flexibility of free-form 3D deformation. We use the deep neural networks to refine the 3D shape in a Hierarchical Mesh Deformation (HMD) framework, utilizing the constraints from body joints, silhouettes, and per-pixel shading information. Our method can restore detailed human body shapes with complete textures beyond skinned models. Experiments demonstrate that our method has outperformed previous state-of-the-art approaches, achieving better accuracy in terms of both 2D IoU number and 3D metric distance. [ABSTRACT FROM AUTHOR]

This work concerns the assessment of mold parts with multi-view scanning point cloud data. With the soaring demand for complex geometry parts, multi-axis electrical discharge machining (EDM) manufacturing also faces the challenge of high-cost and the requirement of high-accuracy. As a part of EDM, well-judged life-time and wear condition of mold parts are critical to ensuring product quality. In this work, we present a novel non-contact surface inspection solution for mold part 3D reconstruction and inspection based on light-section scanning. After calibration and registration, multi-view stereo (MVS) point cloud data for the parts can be obtained, which can then be used to generate 2D section profiles. Comparison method is designed to examine if the design tolerance is met by comparing the section profiles against those based on the CAD model of the parts. Our method is verified by assessing worn electrode parts, which are commonly used in machining centers. The method proposed in this paper ensures the detection accuracy and improves the detection speed. It overcomes the problem of slow detection speed of mold part wear using 3D point cloud. [ABSTRACT FROM AUTHOR]

In this paper, the problem of human action recognition using 3D reconstruction data is deeply investigated. 3D reconstruction techniques are employed for addressing two of the most challenging issues related to human action recognition in the general case, namely view variance (i.e., when the same action is observed from different viewpoints) and the presence of (self-) occlusions (i.e., when for a given point of view a body part of an individual conceals another body part of the same or another subject). The main contributions of this paper are summarized as follows. The first one is a detailed examination of the use of 3D reconstruction data for performing human action recognition. The latter includes the introduction of appropriate local/global flow/shape descriptors, extensive experiments in challenging publicly available datasets and exhaustive comparisons with state-of-art approaches. The second one is a new local-level 3D flow descriptor, which incorporates spatial and surface information in the flow representation and efficiently handles the problem of defining 3D orientation at every local neighborhood. The third one is a new global-level 3D flow descriptor that efficiently encodes the global motion characteristics in a compact way. The fourth one is a novel global temporal-shape descriptor that extends the notion of 3D shape descriptions for action recognition, by incorporating the temporal dimension. The proposed descriptor efficiently addresses the inherent problems of temporal alignment and compact representation, while also being robust in the presence of noise (compared with similar tracking-based methods of the literature). Overall, this paper significantly improves the state-of-art performance and introduces new research directions in the field of 3D action recognition, following the recent development and wide-spread use of portable, affordable, high-quality and accurate motion capturing devices (e.g., Microsoft Kinect). [ABSTRACT FROM AUTHOR]

Over the last years, with the advent of Generative Adversarial Networks (GANs), many face analysis tasks have accomplished astounding performance, with applications including, but not limited to, face generation and 3D face reconstruction from a single “in-the-wild” image. Nevertheless, to the best of our knowledge, there is no method which can produce render-ready high-resolution 3D faces from “in-the-wild” images and this can be attributed to the: (a) scarcity of available data for training, and (b) lack of robust methodologies that can successfully be applied on very high-resolution data. In this paper, we introduce the first method that is able to reconstruct photorealistic render-ready 3D facial geometry and BRDF from a single “in-the-wild” image. To achieve this, we capture a large dataset of facial shape and reflectance, which we have made public. Moreover, we define a fast and photorealistic differentiable rendering methodology with accurate facial skin diffuse and specular reflection, self-occlusion and subsurface scattering approximation. With this, we train a network that disentangles the facial diffuse and specular reflectance components from a mesh and texture with baked illumination, scanned or reconstructed with a 3DMM fitting method. As we demonstrate in a series of qualitative and quantitative experiments, our method outperforms the existing arts by a significant margin and reconstructs authentic, 4K by 6K-resolution 3D faces from a single low-resolution image, that are ready to be rendered in various applications and bridge the uncanny valley. [ABSTRACT FROM AUTHOR]

Forward modeling of wave scattering and radar imaging mechanisms is the key to information extraction from synthetic aperture radar (SAR) images. Like inverse graphics in the optical domain, an inherently-integrated forward-inverse approach would be promising for SAR advanced information retrieval and target reconstruction. This paper presents such an attempt at inverse graphics for SAR imagery. A differentiable SAR renderer (DSR) is developed, which reformulates the mapping and projection algorithm of the SAR imaging mechanism in the differentiable form of probability maps. First-order gradients of the proposed DSR are then analytically derived, which can be back-propagated from rendered image/silhouette to the target geometry and scattering attributes. A 3D inverse target reconstruction algorithm from SAR images is devised. Several simulation and reconstruction experiments are conducted, including targets with and without background, using synthesized data or real measured inverse SAR (ISAR) data by ground radar. Results demonstrate the efficacy of the proposed DSR and its inverse approach. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel method for automatic depth estimation from color images using parameter transfer. By modeling the correlation between color images and their depth maps with a set of parameters, we get a database of parameter sets. Given an input image, we extract the high-level features to find the best matched image sets from the database. Then the set of parameters corresponding to the best match are used to estimate the depth of the input image. Compared with the past learning-based methods, our trained model consists only of trained features and parameter sets, which occupy little space. We evaluate our depth estimation method on several benchmark RGB-D (RGB + depth) data sets. The experimental results are comparable to the state-of-the-art results, while the model size is very small and very suitable for mobile devices, demonstrating the promising performance of our proposed method. [ABSTRACT FROM PUBLISHER]

The latest developments in 3D capturing, processing, and rendering provide means to unlock novel 3D application pathways. The main elements of an integrated platform, which target tele-immersion and future 3D applications, are described in this paper, addressing the tasks of real-time capturing, robust 3D human shape/appearance reconstruction, and skeleton-based motion tracking. More specifically, initially, the details of a multiple RGB-depth (RGB-D) capturing system are given, along with a novel sensors’ calibration method. A robust, fast reconstruction method from multiple RGB-D streams is then proposed, based on an enhanced variation of the volumetric Fourier transform-based method, parallelized on the Graphics Processing Unit, and accompanied with an appropriate texture-mapping algorithm. On top of that, given the lack of relevant objective evaluation methods, a novel framework is proposed for the quantitative evaluation of real-time 3D reconstruction systems. Finally, a generic, multiple depth stream-based method for accurate real-time human skeleton tracking is proposed. Detailed experimental results with multi-Kinect2 data sets verify the validity of our arguments and the effectiveness of the proposed system and methodologies. [ABSTRACT FROM PUBLISHER]

This paper proposes a virtual 3D eyeglasses try-on system driven by a 2D Internet image of a human face wearing with a pair of eyeglasses. The main technical challenge of this system is the automatic 3D eyeglasses model reconstruction from the 2D glasses on a frontal human face. Against this challenge, this paper first proposes an eyeglasses segmentation method using a convolutional neural network-based parsing algorithm to label the glasses pixels, followed with a proposed symmetry-based level-set optimization algorithm to refine the contour of the eyeglasses. With the precisely extracted silhouette image, we take advantages of the smoothness and the symmetry priors of the eyeglasses, and propose a silhouette-based 3D deformation method to deform a 3D eyeglasses model selected from a predefined 3D model database. The obtained model is plausibly approximated to the input 2D eyeglasses after a texture mapping step. Finally, we develop a virtual try-on system to interactively synthesize the reconstructed eyeglasses on a moving target face in real time. The experimental results demonstrate the efficient and convincing virtual try-on performance of our approach and the commercial potential of our proposed system. [ABSTRACT FROM PUBLISHER]

In this paper, we propose a geometric neural network with edge-aware refinement (GeoNet++) to jointly predict both depth and surface normal maps from a single image. Building on top of two-stream CNNs, GeoNet++ captures the geometric relationships between depth and surface normals with the proposed depth-to-normal and normal-to-depth modules. In particular, the “depth-to-normal” module exploits the least square solution of estimating surface normals from depth to improve their quality, while the “normal-to-depth” module refines the depth map based on the constraints on surface normals through kernel regression. Boundary information is exploited via an edge-aware refinement module. GeoNet++ effectively predicts depth and surface normals with high 3D consistency and sharp boundaries resulting in better reconstructed 3D scenes. Note that GeoNet++ is generic and can be used in other depth/normal prediction frameworks to improve 3D reconstruction quality and pixel-wise accuracy of depth and surface normals. Furthermore, we propose a new 3D geometric metric (3DGM) for evaluating depth prediction in 3D. In contrast to current metrics that focus on evaluating pixel-wise error/accuracy, 3DGM measures whether the predicted depth can reconstruct high quality 3D surface normals. This is a more natural metric for many 3D application domains. Our experiments on NYUD-V2 and KITTI datasets verify that GeoNet++ produces fine boundary details and the predicted depth can be used to reconstruct high quality 3D surfaces. [ABSTRACT FROM AUTHOR]

The teaching programming mode and offline programming mode of welding robot are essential parts in today’s manufacturing industry. However, these modes can’t meet the automation requirements and adaptive ability of welding robot. To achieve 3D path acquisition of weld seam for robot autonomous programming, a fast and accurate offline 3D seam extraction method is proposed based on monocular structured light sensor. In this method, gray code and phase-shift code raster images are projected to the workpiece, and the 3D coordinates of the weld are calculated by collecting the workpiece images with raster patterns. The path fitting of robot is completed by polynomial fitting method. The novel monocular structured light sensor used by this paper has compact structure and small volume, and can adapt to a variety of welding working scenes. The combination of gray code and phase shift code makes the measurement accurate and fast, and achieves the effect of full resolution decoding. The experimental results show that the proposed method can complete the task of 3D weld extraction and path fitting, which has the characteristics of high efficiency and strong robustness. [ABSTRACT FROM AUTHOR]

I comment on a paper describing a method for deducing the hidden topology of an object portrayed in a 2D natural line drawing. The principal problem with this paper is that it cannot be considered an advance on (or even an equal of) the state of the art as the approach it describes makes the same limiting assumptions as approaches proposed 10 years ago. There are also important omissions in the review of related work. [ABSTRACT FROM AUTHOR]

In order to improve the reliability of electronic transactions, it is necessary to study more stringent and effective methods of identification and authentication. Face recognition technology has been applied in electronic trading, but it has not been widely promoted on a global scale. In view of this, based on the 3D modeling technology, this study combines the actual situation to carry out the demand analysis. Moreover, on this basis, the face feature recognition process is set. In addition, based on the needs of network finance for identity recognition and authentication and the characteristics and deficiencies of face image identification and authentication methods, this paper proposes an image-based identity recognition and authentication method for systems to be more strictly authenticated. Finally, this paper sets the comparative experiment to compare the performance of the algorithm. The results show that the proposed algorithm has certain practical effects and can provide theoretical reference for related research. [ABSTRACT FROM AUTHOR]

During the conversion process of the short-circuit arc to the submerged power supply arc, because the short-circuit arc diameter is much larger than the submerged power supply arc, the specific position of the submerged power supply arc cannot be determined, and the initial position of the submerged power supply arc has a certain randomness. The purpose of this paper is to try to use the binocular stereo vision method to reconstruct the submerged arc image in three dimensions, obtain the moving image of the arc, obtain more comprehensive information than the two-dimensional image, and multi-field coupling dynamics of the power supply arc. In this paper, the multi-field coupling effects of electromagnetic force, thermal buoyancy, wind load and air resistance of the submerged arc of the half-wavelength transmission line are considered comprehensively, and the chain arc model is incorporated into the force equation and motion model of the submerged arc. By deeply analyzing the formation and motion mechanism of the arc root, the arc root model of the submerged power arc is established, and the optimal selection method of the current element length is proposed. The physical simulation experiment of the half-wavelength submerged arc is carried out, and the simulation results are compared with the experimental results to verify the effectiveness of the simulation model. By comparing the trajectory and self-extinguishing time of different submerged currents, wind speeds and winds to the submerged power supply arc, the effects of wind load, thermal buoyancy and electromagnetic force on the arc are analyzed, which indicates that the built arc model can effectively account for the wind load. The influence of submarine power arc motion and self-extinguishing characteristics. The results show that the submarine power arc image is acquired by constructing a binocular stereo vision system, and the obtained submersible arc image is reconstructed three-dimensionally. The three-dimensional image of the arc is obtained and compared with the actual image. The result shows that the three-dimensional reconstruction The result is effective, and the exploration study provides a new method for further obtaining the multiple variations of the true arc length of the arc with the arcing time. [ABSTRACT FROM AUTHOR]

Accurate navigation of flexible medical instruments like catheters require the knowledge of its pose, that is its position and orientation. In this paper multi-core fibers inscribed with fiber Bragg gratings (FBG) are utilized as sensors to measure the pose of a multi-segment catheter. A reconstruction technique that provides the pose of such a fiber is presented. First, the measurement from the Bragg gratings are converted to strain then the curvature is deduced based on those strain calculations. Next, the curvature and the Bishop frame equations are used to reconstruct the fiber. This technique is validated through experiments where the mean error in position and orientation is observed to be less than 4.69 mm and 6.48 degrees, respectively. The main contributions of the paper are the use of Bishop frames in the reconstruction and the experimental validation of the acquired pose. [ABSTRACT FROM AUTHOR]

Recovering the three-dimensional structure information from a monocular camera is significant for automated driving, robot navigation, and traffic safety assessment. Recent work has solved various tight issues on self-supervised monocular depth estimation of leveraging on-board videos, such as occlusion/disocclusion, dynamic objects, and scale inconsistent. Nevertheless, rare work focuses on the model’s prediction confidence and underlying relations between depths, while they are essential for a decision-making system and performance improvement, respectively. This paper proposes a novel scheme, that of correlation-aware structure, to dig into the relations between depths, converting the independent depths into a graph-like connected depth map. Subsequently, a Gaussian estimator is devised to predict the depth map and uncertainty map concurrently. The uncertainty map can show us problematic regions where it is difficult to predict, from which we further develop uncertainty-based strategies to improve the performance. Specifically, we propose a simple image preprocessing method to overcome the gradient locality issue caused by low-texture, especially in smooth roads and shadows. Also, to avoid the influence of high-uncertainty regions, we propose a solidity-aware mask to recognize the reliable pixels for training in the image. The experiments on the KITTI dataset show that our method results in a competitive performance in both depth and ego-motion estimation tasks compared with the state-of-the-art methods. Besides, additional experiments on the Make3D and Cityscapes datasets demonstrate our method’s strong generalization capability and practicality. [ABSTRACT FROM AUTHOR]

Deploying 3D single-photon Lidar imaging in real world applications presents multiple challenges including imaging in high noise environments. Several algorithms have been proposed to address these issues based on statistical or learning-based frameworks. Statistical methods provide rich information about the inferred parameters but are limited by the assumed model correlation structures, while deep learning methods show state-of-the-art performance but limited inference guarantees, preventing their extended use in critical applications. This paper unrolls a statistical Bayesian algorithm into a new deep learning architecture for robust image reconstruction from single-photon Lidar data, i.e. the algorithm’s iterative steps are converted into neural network layers. The resulting algorithm benefits from the advantages of both statistical and learning based frameworks, providing best estimates with improved network interpretability. Compared to existing learning-based solutions, the proposed architecture requires a reduced number of trainable parameters, is more robust to noise and mismodelling of the system impulse response function, and provides richer information about the estimates including uncertainty measures. Results on synthetic and real data show competitive results regarding the quality of the inference and computational complexity when compared to state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Image Super-Resolution (SR) provides a promising technique to enhance the image quality of low-resolution sensors for a wide range of optical applications. It is noted that the costs for capturing high-resolution images in various spectral ranges are significantly different, thus it is reasonable to utilize low-cost channel images (e.g., visible/depth images) as guidance to boost the accuracy of SR results of the expensive channel (e.g., thermal images) significantly. In this paper, we attempt to leverage complementary information from low-cost channels (visible/depth) to boost image quality of an expensive channel (thermal) using fewer parameters. To this end, we first build a multi-modal imaging system and present an effective method to generate pixel-wise aligned visible and thermal images via virtual 3D viewpoint rendering. Then, we design a feature-level multispectral fusion residual network model to perform high-accuracy SR of thermal images by adaptively integrating co-occurrence features presented in multispectral images. Experimental results demonstrate that this novel approach can effectively alleviate the ill-posed inverse problem of image SR by taking into account complementary information from an additional low-cost channel, significantly outperforming state-of-the-art SR approaches in terms of both accuracy and efficiency. [ABSTRACT FROM AUTHOR]

Emerging automotive applications such as in-vehicle Augmented Reality (AR) and fully automated parking require a comprehensive understanding of the vehicle’s three-dimensional surrounding represented as an object-level environmental model. In this model, not only 3D poses (positions and orientations) and 3D sizes of detected objects are registered, but 3D shapes (geometries) need to be reconstructed precisely. A combination of 3D object detection and 3D surface reconstruction techniques, referred to as object-level 3D reconstruction, is fundamental to building such environmental models. However, the possibilities to incorporate object-level 3D reconstruction in a car have not been sufficiently explored either in academic research or in the industry. This primarily stems from the cost and resource constraints associated with the automotive domain. In this paper, we address these constraints by proposing implementations of in-vehicle object-level 3D reconstruction in two specific use cases: (i) augmented reality and (ii) automated parking. For augmented reality, we propose a cost-efficient solution called monocular 3D Shaping that requires only a single frame from a monocular camera as input. For automated parking, we propose a resource-efficient alternative that generates more precise 3D reconstruction results by taking advantage of additional 3D sensors (such as Lidars). The crux of our proposed approaches lies in the use of a Latent Shape Space, where various 3D shapes are represented using only two parameters. As a result, highly complex 3D shapes can now be transmitted using a low- to medium-bandwidth in-vehicle communication infrastructure in a cost-effective manner. [ABSTRACT FROM AUTHOR]

The Structure-from-Motion (SfM) algorithm is widely used for point cloud reconstruction. However, one drawback of conventional SfM based methods is that the obtained final point sets may contain holes and noise, which could degrade the estimation of reconstructed objects especially for smooth surfaces with few features. The other drawback is the accuracy and speed of SfM based methods depend on the uncertain number of images. To overcome these limitations, this paper proposes a novel 3D reconstruction method for unstructured objects based on the structure from motion in combination with the structured light, in which the point sets of structured light and the point sets of structure from motion can come from different target objects. Since the two point sets coming from multiple sensors do not scale well for register, making it difficult to find corresponding points, a scaled principal component analysis algorithm is proposed for the registration to overcome the impact due to large scale variance. With a large scale factor, a recalculated registration center is proposed via feature region segmentation to achieve point cloud registration again. The two point sets are matched using the proposed optimization method to complete 3D reconstruction. Surface reconstruction is performed using the Poisson algorithm to obtain a smooth surface. The proposed method is tested on some simple structured objects and real-life data of complex unstructured objects collected using range sensors. Compared with several state-of-the-art algorithms, experimental results confirm its potential for surface reconstruction from depth data calculated from the two sets. [ABSTRACT FROM AUTHOR]

This paper presents a technique to reconstruct the shape of a flexible instrument in three-dimensional Euclidean space based on data from fiber Bragg gratings (FBGs) that are inscribed in multi-core fibers. Its main contributions are the application of several multi-core fibers with FBGs as shape sensor for medical instruments and a thorough presentation of the reconstruction technique. The data from the FBG sensors are first converted to strain measurements, which is then used to calculate the curvature and torsion of the fibers. The shape of the instrument is reconstructed using Frenet–Serret equations in conjunction with the calculated curvature and torsion of the instrument. The reconstruction technique is validated with a catheter sensorized with four multi-core fibers that have FBG sensors. The catheter is placed in eight different configurations and the reconstruction is compared to the ground truth. The maximum reconstruction error among all the configurations is found to be 1.05 mm. The results show that shape sensing for flexible medical instruments is feasible with FBG sensors in multi-core fibers. [ABSTRACT FROM AUTHOR]

With the rapid development of computer vision and industrial technology, the requirements of the intelligent welding robots are increasing in the real industrial production. The traditional teaching-playback mode and the off-line programming mode cannot meet the automation demand and self-adaptive ability of welding robots. In order to improve the efficiency of welding robots, this paper proposes a novel 3-D path extraction method of weld seams based on a stereo-structured light sensor. Faced with the low efficiency of the line-structured light and the poor robustness of passive vision, the seam extraction based on a point cloud processing algorithm is proposed which could well adapt to the weld seams with different types and different groove sizes. Meanwhile, the position information and pose information of weld seam are established to serve 3-D path teaching of a welding robot. The experimental results show that the maximum path extraction error of V-type butt joint is less than 0.7 mm. The proposed scheme could well serve for the 3-D path teaching task before welding. [ABSTRACT FROM AUTHOR]

Light-field cameras have now become available in both consumer and industrial applications, and recent papers have demonstrated practical algorithms for depth recovery from a passive single-shot capture. However, current light-field depth estimation methods are designed for Lambertian objects and fail or degrade for glossy or specular surfaces. The standard Lambertian photoconsistency measure considers the variance of different views, effectively enforcing point-consistency, i.e., that all views map to the same point in RGB space. This variance or point-consistency condition is a poor metric for glossy surfaces. In this paper, we present a novel theory of the relationship between light-field data and reflectance from the dichromatic model. We present a physically-based and practical method to estimate the light source color and separate specularity. We present a new photo consistency metric, line-consistency, which represents how viewpoint changes affect specular points. We then show how the new metric can be used in combination with the standard Lambertian variance or point-consistency measure to give us results that are robust against scenes with glossy surfaces. With our analysis, we can also robustly estimate multiple light source colors and remove the specular component from glossy objects. We show that our method outperforms current state-of-the-art specular removal and depth estimation algorithms in multiple real world scenarios using the consumer Lytro and Lytro Illum light field cameras. [ABSTRACT FROM PUBLISHER]

We disagree with the comments made by Varley [1] on our previous paper [2]. In this paper, we respond to his comments and show that they are not correct. [ABSTRACT FROM AUTHOR]

Photometric stereo recovers three-dimensional (3D) object surface normal from multiple images under different illumination directions. Traditional photometric stereo methods suffer from the problem of non-Lambertian surfaces with general reflectance. By leveraging deep neural networks, learning-based methods are capable of improving the surface normal estimation under general non-Lambertian surfaces. These state-of-the-art learning-based methods however do not associate surface normal with reconstructed images and, therefore, they cannot explore the beneficial effect of such association on the estimation of the surface normal. In this paper, we specifically exploit the positive impact of this association and propose a novel dual regression network for both fine surface normals and arbitrary reconstructed images in calibrated photometric stereo. Our work unifies the 3D reconstruction and rendering tasks in a deep learning framework, with the explorations including: 1. generating specified reconstructed images under arbitrary illumination directions, which provides more intuitive perception of the reflectance and is extremely useful for visual applications, such as virtual reality, and 2. our dual regression scheme introduces an additional constraint on observed images and reconstructed images, which forms a closed-loop to provide additional supervision. Experiments show that our proposed method achieves accurate reconstructed images under arbitrarily specified illumination directions and it significantly outperforms the state-of-the-art learning-based single regression methods in calibrated photometric stereo. [ABSTRACT FROM AUTHOR]

Benefits from artificial intelligence and the Internet of Vehicles (IoV), Management of modern transportation have great progress, especially in urban areas. However, traditional traffic video analysis and visualization are usually conducted in offsite and textural environments, i.e., text and number, which do not promote user’s sensorial perception and interaction. Thus, the problem that how to use modern novel techniques to analyze traffic video for improving intelligent transportation is so emergency. In this paper, we introduce a joint 3D reconstruction and object tracking approach to traffic video analysis under the IoV environment, which is an integrative framework and consists of 3D reconstruction, object detection, and visual tracking. The 3D reconstruction system is connected to the Internet of Vehicles and integrated into the system to retrieve image data for recovering the 3D model of vehicles, and then, visualizing vehicle trajectories in real-time by augmented reality. And the system can also locate the vehicle’s position in real-time. The experiments in both laboratory and practice show great feedback, which will effectively contribute to intelligent transportation. [ABSTRACT FROM AUTHOR]

Extracting spatio-temporally consistent segments from a video sequence is a challenging problem due to the complexity of color, motion and occlusions. Most existing spatio-temporal segmentation approaches have inherent difficulties in handling large displacement with significant occlusions . This paper presents a novel framework for spatio-temporal segmentation. With the estimated depth data beforehand by a multi-view stereo technique, we project the pixels to other frames for collecting the boundary and segmentation statistics in a video, and incorporate them into the segmentation energy for spatio-temporal optimization. In order to effectively solve this problem, we introduce an iterative optimization scheme by first initializing segmentation maps for each frame independently, and then link the correspondences among different frames and iteratively refine them with the collected statistics, so that a set of spatio-temporally consistent volume segments are finally achieved. The effectiveness and usefulness of our automatic framework are demonstrated via its applications for 3D reconstruction, video editing and semantic segmentation on a variety of challenging video examples. [ABSTRACT FROM PUBLISHER]

Place-of-Interest (POI) summarization by aesthetics evaluation can recommend a set of POI images to the user and it is significant in image retrieval. In this paper, we propose a system that summarizes a collection of POI images regarding both aesthetics and diversity of the distribution of cameras. First, we generate visual albums by a coarse-to-fine POI clustering approach and then generate 3D models for each album by the collected images from social media. Second, based on the 3D to 2D projection relationship, we select candidate photos in terms of the proposed crowd source saliency model. Third, in order to improve the performance of aesthetic measurement model, we propose a crowd-sourced saliency detection approach by exploring the distribution of salient regions in the 3D model. Then, we measure the composition aesthetics of each image and we explore crowd source salient feature to yield saliency map, based on which, we propose an adaptive image adoption approach. Finally, we combine the diversity and the aesthetics to recommend aesthetic pictures. Experimental results show that the proposed POI summarization approach can return images with diverse camera distributions and aesthetics. [ABSTRACT FROM PUBLISHER]

Reconstructing the shape of a 3D object from multi-view images under unknown, general illumination is a fundamental problem in computer vision. High quality reconstruction is usually challenging especially when fine detail is needed and the albedo of the object is non-uniform. This paper introduces vertex overall illumination vectors to model the illumination effect and presents a total variation (TV) based approach for recovering surface details using shading and multi-view stereo (MVS). Behind the approach are the two important observations: (1) the illumination over the surface of an object often appears to be piecewise smooth and (2) the recovery of surface orientation is not sufficient for reconstructing the surface, which was often overlooked previously. Thus we propose to use TV to regularize the overall illumination vectors and use visual hull to constrain partial vertices. The reconstruction is formulated as a constrained TV-minimization problem that simultaneously treats the shape and illumination vectors as unknowns. An augmented Lagrangian method is proposed to quickly solve the TV-minimization problem. As a result, our approach is robust, stable and is able to efficiently recover high-quality surface details even when starting with a coarse model obtained using MVS. These advantages are demonstrated by extensive experiments on the state-of-the-art MVS database, which includes challenging objects with varying albedo. [ABSTRACT FROM PUBLISHER]

The temperature measurement has great significance for the combustion diagnostics and the study of flame characteristics. In this paper, an optical sectioning tomographic (OST) technique for 3-D flame temperature measurement combined with light field imaging is proposed. First, the fundamental principle of light field OST is introduced in detail. Then, to determine the refocused depth, the point spread functions of the light field camera, and the relationship between the radiation intensity of flame and image gray value, the calibration works are carried out. Furthermore, the images of different sections of flame are acquired by the digital refocusing technology, and then the high-precision and real-time radiation intensity distribution of each section can be reconstructed using a tomographic reconstruction algorithm. Finally, in combination with the relationship between the image gray value and radiation intensity of flame, the 3-D flame temperature distribution can be calculated. The flame experiments of a candle and a single biomass particle in a fluidized bed under oxy-combustion conditions are performed to evaluate the light field OST system. Results show that the reconstructed results are in good agreement with the temperature distribution of the combustion flame. Compared with the traditional optical sectioning tomographic techniques, the light field OST has the advantages of simplicity of measurement system and fast response, and being suitable for 3-D temperature measurement of turbulent flame. [ABSTRACT FROM PUBLISHER]

We propose a novel approach to infer a high-quality depth map from a set of images with small viewpoint variations. In general, techniques for depth estimation from small motion consist of camera pose estimation and dense reconstruction. In contrast to prior approaches that recover scene geometry and camera motions using pre-calibrated cameras, we introduce in this paper a self-calibrating bundle adjustment method tailored for small motion which enables computation of camera poses without the need for camera calibration. For dense depth reconstruction, we present a convolutional neural network called DPSNet (Deep Plane Sweep Network) whose design is inspired by best practices of traditional geometry-based approaches. Rather than directly estimating depth or optical flow correspondence from image pairs as done in many previous deep learning methods, DPSNet takes a plane sweep approach that involves building a cost volume from deep features using the plane sweep algorithm, regularizing the cost volume, and regressing the depth map from the cost volume. The cost volume is constructed using a differentiable warping process that allows for end-to-end training of the network. Through the effective incorporation of conventional multiview stereo concepts within a deep learning framework, the proposed method achieves state-of-the-art results on a variety of challenging datasets. [ABSTRACT FROM AUTHOR]

Varley [1] made comments on our paper in [2] section by section. We answer them in this response paper. [ABSTRACT FROM AUTHOR]

In this paper, we present a quantitative analysis of the error in the reconstruction of a 3D scene which has been captured with Synthetic Aperture Integral Imaging system. The 3D information is obtained from 2D images for which the camera parameters are unknown. The model used for calibrating the Integral Imaging camera setup is based on fuzzy systems. These systems provide the opportunity for modeling of conditions which are inherently imprecisely defined. We demonstrate that the error in the 3D reconstruction not only depends on the number of cameras, but also to their relative positions. Our model is applied to a set of images captured experimentally from a real object. A true-color real scale 3D reconstruction is successfully achieved. [ABSTRACT FROM PUBLISHER]

Image-based 3D reconstruction is a fundamental task in computer vision and computer graphics. Although many existing approaches have obtained excellent results, they all dependent on multi-view images, then these methods are very complex. To reduce the complexity of the existing approaches, in this paper we fist propose a novel 3D reconstruction method based on single RGB image and the corresponding depth image, and design a unified framework called Single3D. Second, we trained a neural network model to reduce noises for improving the quality of the input RGB image, then leading a desirable RGB image. Third, we combine the deep guided filter and fast guided filter to fill holes on the depth image for improving the quality of 3D model. Fourth, we recover high-quality 3D model from the RGB-D image based on the proposed Single3D framework. Finally, the comprehensive experiments were conducted on the most challenging benchmarking dataset with variant light, occlusion, and clustering. Experimental results show that our method has desirable accuracy and efficiency. [ABSTRACT FROM AUTHOR]

Light-field cameras have become widely available in both consumer and industrial applications. However, most previous approaches do not model occlusions explicitly, and therefore fail to capture sharp object boundaries. A common assumption is that for a Lambertian scene, a pixel will exhibit photo-consistency, which means all viewpoints converge to a single point when focused to its depth. However, in the presence of occlusions this assumption fails to hold, making most current approaches unreliable precisely where accurate depth information is most important – at depth discontinuities. In this paper, an occlusion-aware depth estimation algorithm is developed; the method also enables identification of occlusion edges, which may be useful in other applications. It can be shown that although photo-consistency is not preserved for pixels at occlusions, it still holds in approximately half the viewpoints. Moreover, the line separating the two view regions (occluded object versus occluder) has the same orientation as that of the occlusion edge in the spatial domain. By ensuring photo-consistency in only the occluded view region, depth estimation can be improved. Occlusion predictions can also be computed and used for regularization. Experimental results show that our method outperforms current state-of-the-art light-field depth estimation algorithms, especially near occlusion boundaries. [ABSTRACT FROM PUBLISHER]

This paper proposes a new method for simultaneous 3D reconstruction and semantic segmentation for indoor scenes. Unlike existing methods that require recording a video using a color camera and/or a depth camera, our method only needs a small number of (e.g., 3~5) color images from uncalibrated sparse views, which significantly simplifies data acquisition and broadens applicable scenarios. To achieve promising 3D reconstruction from sparse views with limited overlap, our method first recovers the depth map and semantic information for each view, and then fuses the depth maps into a 3D scene. To this end, we design an iterative deep architecture, named IterNet, to estimate the depth map and semantic segmentation alternately. To obtain accurate alignment between views with limited overlap, we further propose a joint global and local registration method to reconstruct a 3D scene with semantic information. We also make available a new indoor synthetic dataset, containing photorealistic high-resolution RGB images, accurate depth maps and pixel-level semantic labels for thousands of complex layouts. Experimental results on public datasets and our dataset demonstrate that our method achieves more accurate depth estimation, smaller semantic segmentation errors, and better 3D reconstruction results over state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In this paper, we present an algorithm for online 3D reconstruction of dynamic scenes using individual times of arrival (ToA) of photons recorded by single-photon detector arrays. One of the main challenges in 3D imaging using single-photon Lidar is the integration time required to build ToA histograms and reconstruct reliably 3D profiles in the presence of non-negligible ambient illumination. This long integration time also prevents the analysis of rapid dynamic scenes using existing techniques. We propose a new method which does not rely on the construction of ToA histograms but allows, for the first time, individual detection events to be processed online, in a parallel manner in different pixels, while accounting for the intrinsic spatiotemporal structure of dynamic scenes. Adopting a Bayesian approach, a Bayesian model is constructed to capture the dynamics of the 3D profile and an approximate inference scheme based on assumed density filtering is proposed, yielding a fast and robust reconstruction algorithm able to process efficiently thousands to millions of frames, as usually recorded using single-photon detectors. The performance of the proposed method, able to process hundreds of frames per second, is assessed using a series of experiments conducted with static and dynamic 3D scenes and the results obtained pave the way to a new family of real-time 3D reconstruction solutions. [ABSTRACT FROM AUTHOR]

The increasing scale of Structure-from-Motion is fundamentally limited by the conventional optimization framework for the all-in-one global bundle adjustment. In this paper, we propose a distributed approach to coping with this global bundle adjustment for very large scale Structure-from-Motion computation. First, we derive the distributed formulation from the classical optimization algorithm ADMM, Alternating Direction Method of Multipliers, based on the global camera consensus. Then, we analyze the conditions under which the convergence of this distributed optimization would be guaranteed. In particular, we adopt over-relaxation and self-adaption schemes to improve the convergence rate. After that, we propose to split the large scale camera-point visibility graph in order to reduce the communication overheads of the distributed computing. The experiments on both public large scale SfM data-sets and our very large scale aerial photo sets demonstrate that the proposed distributed method clearly outperforms the state-of-the-art method in efficiency and accuracy. [ABSTRACT FROM AUTHOR]

To date, 3D spine reconstruction from biplanar radiographs involves intensive user supervision and semi-automated methods that are time-consuming and not effective in clinical routine. This paper proposes a new, fast, and automated 3D spine reconstruction method through which a realistic statistical shape model of the spine is fitted to images using convolutional neural networks (CNN). The CNNs automatically detect the anatomical landmarks controlling the spine model deformation through a hierarchical and gradual iterative process. The performance assessment used a set of 68 biplanar radiographs, composed of both asymptomatic subjects and adolescent idiopathic scoliosis patients, in order to compare automated reconstructions with ground truths build using multiple experts-supervised reconstructions. The mean (SD) errors of landmark locations (3D Euclidean distances) were 1.6 (1.3) mm, 1.8 (1.3) mm, and 2.3 (1.4) mm for the vertebral body center, endplate centers, and pedicle centers, respectively. The clinical parameters extracted from the automated 3D reconstruction (reconstruction time is less than one minute) presented an absolute mean error between 2.8° and 4.7° for the main spinal parameters and between 1° and 2.1° for pelvic parameters. Automated and expert’s agreement analysis reported that, on average, 89% of automated measurements were inside the expert’s confidence intervals. The proposed automated 3D spine reconstruction method provides an important step that should help the dissemination and adoption of 3D measurements in clinical routine. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel approach, 3D-RecGAN++, which reconstructs the complete 3D structure of a given object from a single arbitrary depth view using generative adversarial networks. Unlike existing work which typically requires multiple views of the same object or class labels to recover the full 3D geometry, the proposed 3D-RecGAN++ only takes the voxel grid representation of a depth view of the object as input, and is able to generate the complete 3D occupancy grid with a high resolution of $\boldsymbol{256^3}$ 256 3 by recovering the occluded/missing regions. The key idea is to combine the generative capabilities of 3D encoder-decoder and the conditional adversarial networks framework, to infer accurate and fine-grained 3D structures of objects in high-dimensional voxel space. Extensive experiments on large synthetic datasets and real-world Kinect datasets show that the proposed 3D-RecGAN++ significantly outperforms the state of the art in single view 3D object reconstruction, and is able to reconstruct unseen types of objects. [ABSTRACT FROM AUTHOR]

To realize a robust robotic grasping system for unknown objects in an unstructured environment, large amounts of grasp data and 3D model data for the object are required; the sizes of these data directly affect the rate of successful grasps. To reduce the time cost of data acquisition and labeling and increase the rate of successful grasps, we developed a self-supervised learning mechanism to control grasp tasks performed by manipulators. First, a manipulator automatically collects the point cloud for the objects from multiple perspectives to increase the efficiency of data acquisition. The complete point cloud for the objects is obtained using the hand-eye vision of the manipulator and the truncated signed distance function algorithm. Then, the point cloud data for the objects are used to generate a series of six-degrees-of-freedom grasp poses, and the force-closure decision algorithm is used to add the grasp quality label to each grasp pose to realize the automatic labeling of grasp data. Finally, the point cloud in the gripper closing area corresponding to each grasp pose is obtained and used to train the grasp-quality classification model for the manipulator. The results of performing actual grasping experiments demonstrate that the proposed self-supervised learning method can increase the rate of successful grasps for the manipulator. Note to Practitioners—Most of the existing grasp planning methods of the manipulator are based on public datasets or simulation data to train model algorithms. Owing to the limited types of objects, the limited amount of data in the public datasets, and the lack of real sensor noise in the simulation data, the robustness of the trained algorithm model is insufficient, and it is difficult to apply to unstructured production environments. To solve the above problems, we propose a 6-DOF capture planning method based on self-supervised learning and introduce a self-supervised learning mechanism to solve the problem of grasp data acquisition in real scenes. The manipulator automatically collects object data from multiple perspectives, performs desktop-level 3D reconstruction, and finally uses the force-closure decision algorithm to automatically label the data in order to realize automatic acquisition and labeling of the grasp data in a real scenario. Preliminary experiments show that this method can obtain high-quality grasp data and can be applied to grasp operations in real multi-target and cluttered environments. However, it has not been tested in actual production environments. This paper focuses on the data acquisition module in the 6-DOF grasp planning framework. In future research, we will design a more efficient grasp planning module to improve the grasp efficiency of the manipulator. [ABSTRACT FROM AUTHOR]

Translucency introduces great challenges to 3-D acquisition because of complicated light behaviors such as refraction and transmittance. In this paper, we describe the development of a unified 3-D data acquisition framework that reconstructs translucent objects using a single commercial time-of-flight (ToF) camera. In our capture scenario, we record a depth map and intensity image of the scene twice using a static ToF camera; first, we capture the depth map and intensity image of an arbitrary background, and then we position the translucent foreground object and record a second depth map and intensity image with both the foreground and the background. As a result of material characteristics, the translucent object yields systematic distortions in the depth map. We developed a new distance representation that interprets the depth distortion induced as a result of translucency. By analyzing ToF depth sensing principles, we constructed a distance model governed by the level of translucency, foreground depth, and background depth. Using an analysis-by-synthesis approach, we can recover the 3-D geometry of a translucent object from a pair of depth maps and their intensity images. Extensive evaluation and case studies demonstrate that our method is effective for modeling the nonlinear depth distortion due to translucency and for reconstruction of a 3-D translucent object. [ABSTRACT FROM PUBLISHER]

Vehicle lane-level localization is a fundamental technology in autonomous driving. To achieve accurate and consistent performance, a common approach is to use the LIDAR technology. However, it is expensive and computational demanding, and thus not a practical solution in many situations. This paper proposes a stereovision system, which is of low cost, yet also able to achieve high accuracy and consistency. It integrates a new lane line detection algorithm with other lane marking detectors to effectively identify the correct lane line markings. It also fits multiple road models to improve accuracy. An effective stereo 3D reconstruction method is proposed to estimate vehicle localization. The estimation consistency is further guaranteed by a new particle filter framework, which takes vehicle dynamics into account. Experiment results based on image sequences taken under different visual conditions showed that the proposed system can identify the lane line markings with 98.6% accuracy. The maximum estimation error of the vehicle distance to lane lines is 16 cm in daytime and 26 cm at night, and the maximum estimation error of its moving direction with respect to the road tangent is 0.06 rad in daytime and 0.12 rad at night. Due to its high accuracy and consistency, the proposed system can be implemented in autonomous driving vehicles as a practical solution to vehicle lane-level localization. [ABSTRACT FROM AUTHOR]