Search

Estimating depth from RGB images is a long-standing ill-posed problem, which has been explored for decades by the computer vision, graphics, and machine learning communities. Among the existing techniques, stereo matching remains one of the most widely used in the literature due to its strong connection to the human binocular system. Traditionally, stereo-based depth estimation has been addressed through matching hand-crafted features across multiple images. Despite the extensive amount of research, these traditional techniques still suffer in the presence of highly textured areas, large uniform regions, and occlusions. Motivated by their growing success in solving various 2D and 3D vision problems, deep learning for stereo-based depth estimation has attracted a growing interest from the community, with more than 150 papers published in this area between 2014 and 2019. This new generation of methods has demonstrated a significant leap in performance, enabling applications such as autonomous driving and augmented reality. In this paper, we provide a comprehensive survey of this new and continuously growing field of research, summarize the most commonly used pipelines, and discuss their benefits and limitations. In retrospect of what has been achieved so far, we also conjecture what the future may hold for deep learning-based stereo for depth estimation research. [ABSTRACT FROM AUTHOR]

This paper presents a novel framework to recover detailed avatar from a single image. It is a challenging task due to factors such as variations in human shapes, body poses, texture, and viewpoints. Prior methods typically attempt to recover the human body shape using a parametric-based template that lacks the surface details. As such resulting body shape appears to be without clothing. In this paper, we propose a novel learning-based framework that combines the robustness of the parametric model with the flexibility of free-form 3D deformation. We use the deep neural networks to refine the 3D shape in a Hierarchical Mesh Deformation (HMD) framework, utilizing the constraints from body joints, silhouettes, and per-pixel shading information. Our method can restore detailed human body shapes with complete textures beyond skinned models. Experiments demonstrate that our method has outperformed previous state-of-the-art approaches, achieving better accuracy in terms of both 2D IoU number and 3D metric distance. [ABSTRACT FROM AUTHOR]

This work concerns the assessment of mold parts with multi-view scanning point cloud data. With the soaring demand for complex geometry parts, multi-axis electrical discharge machining (EDM) manufacturing also faces the challenge of high-cost and the requirement of high-accuracy. As a part of EDM, well-judged life-time and wear condition of mold parts are critical to ensuring product quality. In this work, we present a novel non-contact surface inspection solution for mold part 3D reconstruction and inspection based on light-section scanning. After calibration and registration, multi-view stereo (MVS) point cloud data for the parts can be obtained, which can then be used to generate 2D section profiles. Comparison method is designed to examine if the design tolerance is met by comparing the section profiles against those based on the CAD model of the parts. Our method is verified by assessing worn electrode parts, which are commonly used in machining centers. The method proposed in this paper ensures the detection accuracy and improves the detection speed. It overcomes the problem of slow detection speed of mold part wear using 3D point cloud. [ABSTRACT FROM AUTHOR]

Over the last years, with the advent of Generative Adversarial Networks (GANs), many face analysis tasks have accomplished astounding performance, with applications including, but not limited to, face generation and 3D face reconstruction from a single “in-the-wild” image. Nevertheless, to the best of our knowledge, there is no method which can produce render-ready high-resolution 3D faces from “in-the-wild” images and this can be attributed to the: (a) scarcity of available data for training, and (b) lack of robust methodologies that can successfully be applied on very high-resolution data. In this paper, we introduce the first method that is able to reconstruct photorealistic render-ready 3D facial geometry and BRDF from a single “in-the-wild” image. To achieve this, we capture a large dataset of facial shape and reflectance, which we have made public. Moreover, we define a fast and photorealistic differentiable rendering methodology with accurate facial skin diffuse and specular reflection, self-occlusion and subsurface scattering approximation. With this, we train a network that disentangles the facial diffuse and specular reflectance components from a mesh and texture with baked illumination, scanned or reconstructed with a 3DMM fitting method. As we demonstrate in a series of qualitative and quantitative experiments, our method outperforms the existing arts by a significant margin and reconstructs authentic, 4K by 6K-resolution 3D faces from a single low-resolution image, that are ready to be rendered in various applications and bridge the uncanny valley. [ABSTRACT FROM AUTHOR]

Forward modeling of wave scattering and radar imaging mechanisms is the key to information extraction from synthetic aperture radar (SAR) images. Like inverse graphics in the optical domain, an inherently-integrated forward-inverse approach would be promising for SAR advanced information retrieval and target reconstruction. This paper presents such an attempt at inverse graphics for SAR imagery. A differentiable SAR renderer (DSR) is developed, which reformulates the mapping and projection algorithm of the SAR imaging mechanism in the differentiable form of probability maps. First-order gradients of the proposed DSR are then analytically derived, which can be back-propagated from rendered image/silhouette to the target geometry and scattering attributes. A 3D inverse target reconstruction algorithm from SAR images is devised. Several simulation and reconstruction experiments are conducted, including targets with and without background, using synthesized data or real measured inverse SAR (ISAR) data by ground radar. Results demonstrate the efficacy of the proposed DSR and its inverse approach. [ABSTRACT FROM AUTHOR]

In this paper, we propose a geometric neural network with edge-aware refinement (GeoNet++) to jointly predict both depth and surface normal maps from a single image. Building on top of two-stream CNNs, GeoNet++ captures the geometric relationships between depth and surface normals with the proposed depth-to-normal and normal-to-depth modules. In particular, the “depth-to-normal” module exploits the least square solution of estimating surface normals from depth to improve their quality, while the “normal-to-depth” module refines the depth map based on the constraints on surface normals through kernel regression. Boundary information is exploited via an edge-aware refinement module. GeoNet++ effectively predicts depth and surface normals with high 3D consistency and sharp boundaries resulting in better reconstructed 3D scenes. Note that GeoNet++ is generic and can be used in other depth/normal prediction frameworks to improve 3D reconstruction quality and pixel-wise accuracy of depth and surface normals. Furthermore, we propose a new 3D geometric metric (3DGM) for evaluating depth prediction in 3D. In contrast to current metrics that focus on evaluating pixel-wise error/accuracy, 3DGM measures whether the predicted depth can reconstruct high quality 3D surface normals. This is a more natural metric for many 3D application domains. Our experiments on NYUD-V2 and KITTI datasets verify that GeoNet++ produces fine boundary details and the predicted depth can be used to reconstruct high quality 3D surfaces. [ABSTRACT FROM AUTHOR]

Recovering the three-dimensional structure information from a monocular camera is significant for automated driving, robot navigation, and traffic safety assessment. Recent work has solved various tight issues on self-supervised monocular depth estimation of leveraging on-board videos, such as occlusion/disocclusion, dynamic objects, and scale inconsistent. Nevertheless, rare work focuses on the model’s prediction confidence and underlying relations between depths, while they are essential for a decision-making system and performance improvement, respectively. This paper proposes a novel scheme, that of correlation-aware structure, to dig into the relations between depths, converting the independent depths into a graph-like connected depth map. Subsequently, a Gaussian estimator is devised to predict the depth map and uncertainty map concurrently. The uncertainty map can show us problematic regions where it is difficult to predict, from which we further develop uncertainty-based strategies to improve the performance. Specifically, we propose a simple image preprocessing method to overcome the gradient locality issue caused by low-texture, especially in smooth roads and shadows. Also, to avoid the influence of high-uncertainty regions, we propose a solidity-aware mask to recognize the reliable pixels for training in the image. The experiments on the KITTI dataset show that our method results in a competitive performance in both depth and ego-motion estimation tasks compared with the state-of-the-art methods. Besides, additional experiments on the Make3D and Cityscapes datasets demonstrate our method’s strong generalization capability and practicality. [ABSTRACT FROM AUTHOR]

Deploying 3D single-photon Lidar imaging in real world applications presents multiple challenges including imaging in high noise environments. Several algorithms have been proposed to address these issues based on statistical or learning-based frameworks. Statistical methods provide rich information about the inferred parameters but are limited by the assumed model correlation structures, while deep learning methods show state-of-the-art performance but limited inference guarantees, preventing their extended use in critical applications. This paper unrolls a statistical Bayesian algorithm into a new deep learning architecture for robust image reconstruction from single-photon Lidar data, i.e. the algorithm’s iterative steps are converted into neural network layers. The resulting algorithm benefits from the advantages of both statistical and learning based frameworks, providing best estimates with improved network interpretability. Compared to existing learning-based solutions, the proposed architecture requires a reduced number of trainable parameters, is more robust to noise and mismodelling of the system impulse response function, and provides richer information about the estimates including uncertainty measures. Results on synthetic and real data show competitive results regarding the quality of the inference and computational complexity when compared to state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Image Super-Resolution (SR) provides a promising technique to enhance the image quality of low-resolution sensors for a wide range of optical applications. It is noted that the costs for capturing high-resolution images in various spectral ranges are significantly different, thus it is reasonable to utilize low-cost channel images (e.g., visible/depth images) as guidance to boost the accuracy of SR results of the expensive channel (e.g., thermal images) significantly. In this paper, we attempt to leverage complementary information from low-cost channels (visible/depth) to boost image quality of an expensive channel (thermal) using fewer parameters. To this end, we first build a multi-modal imaging system and present an effective method to generate pixel-wise aligned visible and thermal images via virtual 3D viewpoint rendering. Then, we design a feature-level multispectral fusion residual network model to perform high-accuracy SR of thermal images by adaptively integrating co-occurrence features presented in multispectral images. Experimental results demonstrate that this novel approach can effectively alleviate the ill-posed inverse problem of image SR by taking into account complementary information from an additional low-cost channel, significantly outperforming state-of-the-art SR approaches in terms of both accuracy and efficiency. [ABSTRACT FROM AUTHOR]

Emerging automotive applications such as in-vehicle Augmented Reality (AR) and fully automated parking require a comprehensive understanding of the vehicle’s three-dimensional surrounding represented as an object-level environmental model. In this model, not only 3D poses (positions and orientations) and 3D sizes of detected objects are registered, but 3D shapes (geometries) need to be reconstructed precisely. A combination of 3D object detection and 3D surface reconstruction techniques, referred to as object-level 3D reconstruction, is fundamental to building such environmental models. However, the possibilities to incorporate object-level 3D reconstruction in a car have not been sufficiently explored either in academic research or in the industry. This primarily stems from the cost and resource constraints associated with the automotive domain. In this paper, we address these constraints by proposing implementations of in-vehicle object-level 3D reconstruction in two specific use cases: (i) augmented reality and (ii) automated parking. For augmented reality, we propose a cost-efficient solution called monocular 3D Shaping that requires only a single frame from a monocular camera as input. For automated parking, we propose a resource-efficient alternative that generates more precise 3D reconstruction results by taking advantage of additional 3D sensors (such as Lidars). The crux of our proposed approaches lies in the use of a Latent Shape Space, where various 3D shapes are represented using only two parameters. As a result, highly complex 3D shapes can now be transmitted using a low- to medium-bandwidth in-vehicle communication infrastructure in a cost-effective manner. [ABSTRACT FROM AUTHOR]

The Structure-from-Motion (SfM) algorithm is widely used for point cloud reconstruction. However, one drawback of conventional SfM based methods is that the obtained final point sets may contain holes and noise, which could degrade the estimation of reconstructed objects especially for smooth surfaces with few features. The other drawback is the accuracy and speed of SfM based methods depend on the uncertain number of images. To overcome these limitations, this paper proposes a novel 3D reconstruction method for unstructured objects based on the structure from motion in combination with the structured light, in which the point sets of structured light and the point sets of structure from motion can come from different target objects. Since the two point sets coming from multiple sensors do not scale well for register, making it difficult to find corresponding points, a scaled principal component analysis algorithm is proposed for the registration to overcome the impact due to large scale variance. With a large scale factor, a recalculated registration center is proposed via feature region segmentation to achieve point cloud registration again. The two point sets are matched using the proposed optimization method to complete 3D reconstruction. Surface reconstruction is performed using the Poisson algorithm to obtain a smooth surface. The proposed method is tested on some simple structured objects and real-life data of complex unstructured objects collected using range sensors. Compared with several state-of-the-art algorithms, experimental results confirm its potential for surface reconstruction from depth data calculated from the two sets. [ABSTRACT FROM AUTHOR]

To realize a robust robotic grasping system for unknown objects in an unstructured environment, large amounts of grasp data and 3D model data for the object are required; the sizes of these data directly affect the rate of successful grasps. To reduce the time cost of data acquisition and labeling and increase the rate of successful grasps, we developed a self-supervised learning mechanism to control grasp tasks performed by manipulators. First, a manipulator automatically collects the point cloud for the objects from multiple perspectives to increase the efficiency of data acquisition. The complete point cloud for the objects is obtained using the hand-eye vision of the manipulator and the truncated signed distance function algorithm. Then, the point cloud data for the objects are used to generate a series of six-degrees-of-freedom grasp poses, and the force-closure decision algorithm is used to add the grasp quality label to each grasp pose to realize the automatic labeling of grasp data. Finally, the point cloud in the gripper closing area corresponding to each grasp pose is obtained and used to train the grasp-quality classification model for the manipulator. The results of performing actual grasping experiments demonstrate that the proposed self-supervised learning method can increase the rate of successful grasps for the manipulator. Note to Practitioners—Most of the existing grasp planning methods of the manipulator are based on public datasets or simulation data to train model algorithms. Owing to the limited types of objects, the limited amount of data in the public datasets, and the lack of real sensor noise in the simulation data, the robustness of the trained algorithm model is insufficient, and it is difficult to apply to unstructured production environments. To solve the above problems, we propose a 6-DOF capture planning method based on self-supervised learning and introduce a self-supervised learning mechanism to solve the problem of grasp data acquisition in real scenes. The manipulator automatically collects object data from multiple perspectives, performs desktop-level 3D reconstruction, and finally uses the force-closure decision algorithm to automatically label the data in order to realize automatic acquisition and labeling of the grasp data in a real scenario. Preliminary experiments show that this method can obtain high-quality grasp data and can be applied to grasp operations in real multi-target and cluttered environments. However, it has not been tested in actual production environments. This paper focuses on the data acquisition module in the 6-DOF grasp planning framework. In future research, we will design a more efficient grasp planning module to improve the grasp efficiency of the manipulator. [ABSTRACT FROM AUTHOR]

This paper presents a complete 3D model reconstruction of an object with edge quality enhancements using multiple inward depth sensors to create closed 3D model. In the reconstruction pipeline, a pattern of incorrect depth information was consistently observed at the edges of the mesh generated by each sensor stream, which we refer to in this paper as a “drift-effect”. In order to mitigate this, we introduced a filtering approach with a localized threshold value that is used to remove drift faces from a mesh. We also present a mesh stitching technique incorporating Laplacian mesh smoothing to generate a closed 3D model from the smoothened multi-view meshes. The primary objective of this research was to implement a system that could capture a static physical object with a minimum scan time and at a low cost while retaining accurate details in the model. For the demonstration, we used four Intel RealSense D435 depth sensors to capture a clothing article that can be imported into a virtual dressing room application. We captured the entire object within three seconds, which is quicker than traditional techniques such as table rotation and sensor rotation. The final results indicate that the system is able to provide a satisfactory reconstruction of a clothing model which can be used in a live virtual dressing room application.

Recreation of 3D crime scenes is critical for law enforcement in the investigation of serious crimes for criminal justice responses. This work presents a premier systematic literature review (SLR) that offers a structured, methodical, and rigorous approach to understanding the trend of research in 3D crime scene reconstruction as well as tools, technologies, methods, and techniques employed thereof in the last 17 years. Major credible scholarly database sources, Scopus, and Google Scholar, which index journals and conferences that are promoted by entities such as IEEE, ACM, Elsevier, and SpringerLink were explored as data sources. Of the initial 17, 912 papers that resulted from the first search string, 258 were found to be relevant to our research questions after implementing the inclusion and exclusion criteria. To summarize the existing efforts, we compared and analysed various classical 3D reconstruction approaches. This study presents the first comprehensive review of key milestones in the development of methods for 3D crime scene reconstruction, gaps for improvement and where immersive technology has been used to enhance crime scene findings. This study found that the implementation of light detection and ranging (LiDAR) scanners and immersive technologies, alongside traditional methods, has been beneficial in the recreation of crime scenes. The SLR is limited to existing applications with peer-reviewed papers published between 2005 and 2021. Results based on the analysed published data indicated that 20.2% of the articles implemented immersive technologies in crime scene reconstruction, of which Augmented Reality (AR) accounted for 15.3%, Virtual Reality (VR) accounted for 75%, Mixed reality (MR) accounted for 5.9% and VR and AR mixture accounted for 3.8%. Finally, we summarize the development trend of design and key technology prospects of crime scene recreation using immersive technology and provide insights into potential future research. To the best of the researchers’ knowledge, this is the first survey that accomplishes such goals.

With the advancement of consumer-grade RGB-D cameras, obtaining depth information for indoor 3D spaces has become increasingly accessible. This paper systematically reviews 3D reconstruction algorithms for indoor scenes using these cameras, serving as a reference for future research. We cover reconstruction processes and optimization algorithms for both static and dynamic scenes. Additionally, we discuss commonly used datasets, evaluation metrics, and the performance of various reconstruction algorithms. Findings indicate that the balance between reconstruction quality and speed in static scene reconstruction, as well as deformation, occlusion, and fast motion of objects in dynamic scenes are currently major concerns. Deep learning and Neural Radiance Fields (NeRF) are poised to provide new perspectives and methods to address these challenges.

Image-based 3D reconstruction is a challenging task that involves inferring the 3D shape of an object or scene from a set of input images. Learning-based methods have gained attention for their ability to directly estimate 3D shapes. This review paper focuses on state-of-the-art techniques for 3D reconstruction, including the generation of novel, unseen views. An overview of recent developments in the Gaussian Splatting method is provided, covering input types, model structures, output representations, and training strategies. Unresolved challenges and future directions are also discussed. Given the rapid progress in this domain and the numerous opportunities for enhancing 3D reconstruction methods, a comprehensive examination of algorithms appears essential. Consequently, this study offers a thorough overview of the latest advancements in Gaussian Splatting.

Histopathological findings depend on the distinction between normal and subtle changes in mucosal tissues. These distinctions are made based on the nature of the cell lining, nuclear hyperchromasia and cyto-architecture of glands. Diagnosis is mostly based on the experience of the pathologists, which is a subjective procedure. This research focuses on automating the process through 3D reconstruction of gastric histopathology images enhancing quantitative diagnosis. It emphasizes the advantages of 3D reconstruction from light microscopic serial section gastric images over whole slide imaging for clinical decision-making. Comparative analysis shows that processing and analyzing WSI scans are more expensive than light microscopy. While 2D imaging with light microscopy faces challenges due to limited focal depth, especially for thin sections, the objective of this research is to highlight the importance of transforming 2D gastric images into 3D by layering sequential sections. It will offer insights into glandular texture and volume, aiding in distinguishing between malignant glands and benign mimickers. The paper presents a summary and comparative examination of the registration techniques necessary for reconstructing 3D histopathology images. Additionally, it discusses future directions in the field of 3D histopathology reconstruction and acknowledges its potential benefits for research scholars. This will hold potential interest for pathologists when grading gastric carcinoma.

Despite the decreasing costs of stereo vision technologies, traditional 3D reconstruction methods still face challenges related to operational complexity and high costs, which somewhat limit their widespread adoption in various technical applications. Addressing these challenges, this paper presents an improved 3D reconstruction method combining COLMAP and OpenMVS, particularly suited for use with standard consumer-grade imaging devices such as smartphones and drones. By employing Structure from Motion (SFM) and Multi-View Stereo (MVS) techniques, this study significantly enhances image processing speed and achieves substantial improvements in model accuracy and detail reproduction. Systematic experimental validation has demonstrated that the combination of COLMAP and OpenMVS outperforms other open-source tools and combinations in terms of reconstruction speed and precision. This finding highlights the extensive potential and applicability of this combined approach in applications such as virtual reality, robotic navigation, and the digitization of cultural heritage.

Preserving and presenting historical clothing items in museum collections is challenging due to their sensitivity to handling but of great importance due to their deterioration over time. Creating digital replicas of these items can give museums unique documentation and presentation possibilities. A specific workflow is needed to create 3D digital models that can be simulated in real time to present clothing objects in motion and a realistic environment. This paper presents a detailed description of the workflow to create realistic 3D digital models of historical clothing items for motion animation. We demonstrate the use of the workflow on clothing items from the Prague City Museum’s collection.

Neural Radiation Field (NeRF) technology can learn a 3D implicit model of a scene from 2D images and synthesize realistic novel view images. This technology has received widespread attention from the industry and has good application prospects. In response to the problem that the rendering quality of NeRF images needs to be improved, many researchers have proposed various methods to improve the rendering quality in the past three years. The latest relevant papers are classified and reviewed, the technical principles behind quality improvement are analyzed, and the future evolution direction of quality improvement methods is discussed. This study can help researchers quickly understand the current state and evolutionary context of technology in this field, which is helpful in inspiring the development of more efficient algorithms and promoting the application of NeRF technology in related fields.

Reconstructing 3D garment models usually requires laborious data-fetching processes, such as expensive lidar, multiple-view images, or SMPL models of the garments. In this paper, we propose a neat framework that takes single-image inputs for generating pseudo-sparse views of 3D garments and synthesizing multi-view images into a high-quality 3D neural model. Specifically, our framework combines a pretrained pseudo sparse view generator and a volumetric signed distance function (SDF) representation-based network for 3D garment modeling, which uses neural networks to represent both the density and radiance fields. We further introduce a stride fusion strategy to minimize the pixel-level loss in key viewpoints and semantic loss in random viewpoints, which produces view-consistent geometry and sharp texture details. Finally, a multi-view rendering module utilizes the learned SDF representation to generate multi-view garment images and extract accurate mesh and texture from them. We evaluate our proposed framework on the Deep Fashion 3D dataset and achieve state-of-the-art performance in terms of both quantitative and qualitative evaluations.

The human body is a typical non-rigid object, and its 3D reconstruction is a classic problem in the field of computer vision. Due to the inherent complexity and dynamism of the human body, it is not suitable for existing non-rigid 3D motion reconstruction algorithms that assume that the number of shape bases of non-rigid bodies is known. The number of shape bases is very important for 3D reconstruction methods. If the number of shape bases is estimated incorrectly in contour deformation estimation, it can lead to unreliable or even complete failure of the reconstruction algorithm. Therefore, this paper designs a 3D non-rigid motion human object reconstruction algorithm based on reliable estimation of contour deformation degree. This algorithm leverage Scale Invariant Feature Transform (SIFT) to obtain non-rigid moving human target features. Firstly, the contour appearance model of the moving human sequence is used to extract the contour feature sequence, which is preprocessed based on the contour appearance depth feature; Furthermore, the deformation degree of the contour is reconstructed and the calculation process of the number of shape bases was optimized, which is no longer simply defined as known. This method optimizes and solves the problem of missing data, improves the reliability of estimating the degree of contour deformation, and completes target reconstruction. The experimental results show that the three-dimensional reconstruction algorithm can accurately reconstruct the changes in the movements of athletes’ shots; The accuracy of 3D reconstruction can reach 95.98%; Moreover, PSNR, SSIM, and MSE indexes performed well with smaller fluctuation range, and the distribution of three-dimensional reconstructed scattering points is very close to the three-dimensional position distribution of real scattering points, and the three-dimensional reconstruction effect is good with strong reliability.

Digitized methodologies in the recent era contribute to various fields of automation that used to hold different interests and meanings of human life. Buildings with historical significance, cultural values, and beliefs are becoming an interdisciplinary field of interest, engaging more computer scientists nowadays. Such structures need more attention towards reconstructing their values using a flavor of computerized tools instead of brickwork directly. Due to the wear of time, the tiles and engravings of most of the historical monuments are on the verge of ruin, endangering significant historical values. In this survey, we rebuild the values by delving deep into the device and methodologies by providing a comprehensive understanding of emerging fields and some experimental decisions. We discuss heritage restoration from some essential papers on 3D reconstruction, image inpainting, IoT-based methods, genetic algorithms, and image processing. The survey explains Machine Learning, Deep Learning, and Computer Vision-based methods for various restoration tasks in the related field. We divide this into certain parts contributing to different fields that restore cultural heritage. Moreover, we infer that the techniques will be faster, cheaper, and more beneficial to the context of image reconstruction in the near future.

Multi-view stereo estimates the depth maps of multiple perspective images in a scene and then fuses them to generate a 3D point cloud of the scene, which is an essential technology of 3D reconstruction. In this paper, we propose a deep learning method GDINet, applying probabilistic methods to the pyramid framework, which can significantly improve reconstruction quality. In detail, we first establish a Gaussian distribution for each image’s pixel and iterate it in the pyramid framework. The mean value is the estimated depth, and the variance represents the depth estimation error. In addition, we design a novel loss function with excellent convergence to train our network. Finally, we present an initialization module to generate the coarse Gaussian distribution, controlling the parameters in a reasonable range. Our results rank $2nd$ on both DTU and Tanks & Temples datasets, showing that our network has high accuracy, completeness, and robustness. We also make a visualization comparison on the BlendedMVS dataset (containing many aerial scene images) to demonstrate the generalization ability of our model.

This paper proposes an RGB-D sensor-aided ray-tracing simulation framework for indoor radio map construction that models indoor information, such as walls and obstacles, as a set of cuboids. Radio maps can accelerate indoor wireless systems, including resource optimization and fingerprint-based localization. We can construct an accurate radio map using a ray-tracing algorithm with precise indoor structures and obstacles information. However, it is challenging to acquire this indoor information because manual observations of these are expensive. To mitigate this critical drawback, we propose utilizing three-dimensional information obtained from RGB-D images for ray-tracing simulation. The proposed method semi-automatically models indoor information as cuboids from three-dimensional information with RGB-D images. The proposed method then feeds these cuboids into a ray-tracing simulation. Indoor experiments using wireless LANs over 5180 and 2452 MHz clarified that the proposed method accurately constructs the radio map.

Real-time dense SLAM techniques aim to reconstruct the dense three-dimensional geometry of a scene in real time with an RGB or RGB-D sensor. An indoor scene is an important type of working environment for these techniques. The planar prior can be used in this scenario to improve the reconstruction quality, especially for large low-texture regions that commonly occur in an indoor scene. This article fully explores the planar prior in a dense SLAM pipeline. First, we propose a novel plane detection and segmentation method that runs at 200 Hz on a modern graphics processing unit. Our algorithm for constructing global plane constraints is very efficient; hence, we use it in the process of each input frame for the camera pose estimation while maintaining the real-time performance. Second, we propose herein a plane-based map representation that greatly reduces the memory footprint of plane regions while keeping the geometric details on planes. The experiments reveal that our system yields superior reconstruction results with planar information running at more than 30 fps. Aside from speed and storage improvements, our technique also handles the low-texture problem in plane regions. [ABSTRACT FROM AUTHOR]

Multiple cylinders detection from large-scale and complex point clouds is a historical but challenging problem, considering the efficiency and accuracy. We propose a novel framework, named slicing-tracking-detection (STD), that detects multiple cylinders accurately and simultaneously from point clouds of large-scale and complex process plants. In this framework, the 3D cylinder detection problem is reformulated as a cylinder ingredients tracking task based on multi-object tracking (MOT). First, we generate slices from the input point cloud, and render them to slice sequence. Then, the cycle of a cylinder is modeled with a Markov Decision Process (MDP), where the ingredient is tracked with a template and the miss tracking is associated with ingredient proposals through reinforcement learning. Finally, by applying MDP for each cylinder, multiple cylinders can be detected simultaneously and accurately. Extensive experiments show that the proposed STD framework can significantly outperform the state-of-the-art approaches in efficiency, accuracy, and robustness. The source code is available at http://zhiyongsu.github.io. [ABSTRACT FROM AUTHOR]

We study the notion of consistency between a 3D shape and a 2D observation and propose a differentiable formulation which allows computing gradients of the 3D shape given an observation from an arbitrary view. We do so by reformulating view consistency using a differentiable ray consistency (DRC) term. We show that this formulation can be incorporated in a learning framework to leverage different types of multi-view observations e.g., foreground masks, depth, color images, semantics etc. as supervision for learning single-view 3D prediction. We present empirical analysis of our technique in a controlled setting. We also show that this approach allows us to improve over existing techniques for single-view reconstruction of objects from the PASCAL VOC dataset. [ABSTRACT FROM AUTHOR]

Among the most compelling applications of Augmented Reality are spatially registered tutorials. The effort of creating such instructions remains one of the obstacles precluding a wider use. We propose a system that is capable of extracting 3D instructions in a completely model-free manner from demonstrations, based on volumetric changes. The instructions are visualised later in an interactive Augmented Reality guidance application, on a mobile head-mounted display. We enable a technology that can be used by anyone in an ad-hoc tabletop setup for assemblies with rigid components. [ABSTRACT FROM AUTHOR]

In this work, we focus on real-time 3D reconstruction or localization and mapping for outdoor scene using an aerial vehicle called quadcopter. Quadcopter provides the advantages of high flexibility and wide view field in spatial movement. However, existing feature-based and direct methods (using dense or semi-dense approach) are not suitable for outdoor environment, in which multiple challenging scenarios arise such as lighting variance, jittering views, high-speed and non-smooth flight trajectory. The main reason is that the existing methods rely on the assumption of brightness constancy across multiple images and only raw pixel intensities are employed for direct image alignment. In order to tackle these scenarios, a novel method called Feature-based Direct Tracking and Mapping (FDTAM) is proposed, which i) incorporates an efficient binary feature descriptor into direct image alignment module to tackle the challenging scenarios, such as drifting issue under lighting variance problem; ii) applies semi-dense approach to obtain high reconstruction quality; iii) provides a framework with low computational complexity for real-time reconstruction. Compared to other state-of-the-art feature-based and direct methods, our proposed method is shown to tackle the challenging scenarios and improve the accuracy and robustness even in CPU (rather than GPU) platform. [ABSTRACT FROM AUTHOR]

Multi-View Stereo (MVS) methods tackle the ill-posed inverse problem of recovering an object's 3D structure from its multi-view calibrated images. High computational cost restricts most MVS methods from using global information for depth estimation. We present a depth map-based MVS method that uses global information to estimate the depths of all pixels in an image simultaneously. To this end, we transform the depth refinement problem into computing max-flow/min-cut on a 3D grid graph with offset vertices. The $s{-}t$ min-cut of this graph corresponds to the minimization of an energy functional consisting of photo-consistency and smoothness terms. Experimental results on indoor and outdoor datasets validate the efficacy of our method, especially on models with low textured regions where global information is necessary to infer the correct depth. [ABSTRACT FROM AUTHOR]

Mask-based lensless cameras offer a novel design for imaging systems by replacing the lens in a conventional camera with a layer of coded mask. Each pixel of the lensless camera encodes the information of the entire 3D scene. Existing methods for 3D reconstruction from lensless measurements suffer from poor spatial and depth resolution. This is partially due to the system ill-conditioning that arises because the point-spread functions (PSFs) from different depth planes are very similar. In this paper, we propose to capture multiple measurements of the scene under a sequence of coded illumination patterns to improve the 3D image reconstruction quality. In addition, we put the illumination source at a distance away from the camera. With such baseline distance between the lensless camera and illumination source, the camera observes a slice of the 3D volume, and the PSF of each depth plane becomes more resolvable from each other. We present simulation results along with experimental results with a camera prototype to demonstrate the effectiveness of our approach.

In previous studies, the keypoints and their feature descriptors of a hyperspectral image have not received enough attention. Compared with RGB images, the rich feature information of a hyperspectral image can not only constrain the spatial positions of the keypoints, but also provide rich spectral features for their descriptors. In this paper, we propose a deep neural network method called Hyperspectral SuperPoint Network (HSSPN) to detect the keypoints from a hyperspectral image and generate the corresponding feature descriptors. The HSSPN method contains a Shared Encoder (which is responsible for fully extracting the features of the spatial and spectral dimensions) and two decoders: the Keypoint Decoder accurately detects the keypoints according to the feature map outputted by the Shared Encoder, while the Descriptor Decoder is responsible for generating stable feature descriptors. We collected some hyperspectral images with an electronically controlled turntable, and then annotated them with a Hyperspectral Homographic Adaptation (HHA) network. The trained HSSPN gave an excellent keypoint matching performance on the test set. Hyperspectral 3D reconstruction can obtain information in a 3D space and a spectral space at the same time. Most of the previous reconstruction methods are separated for the treatments of 3D reconstruction and hyperspectral imaging, which require complicated registrations. In this study, we propose a SfM-based hyperspectral 3D reconstruction method, in which we first collect hyperspectral images from multiple viewing angles, and then apply the HSSPN method to detect the keypoints and calculate their feature descriptors. These keypoints and feature descriptors are then used to reconstruct each spectral image of the target with the SfM method. The 3D reconstruction models at these wavelengths are simply superimposed to obtain a hyperspectral 3D reconstruction model. Compared with conventional SIFT-based SfM methods, our method achieves better results in the reconstruction accuracy.

Coverage Path Planning (CPP) is an essential capability for autonomous robots operating in various critical applications such as fire fighting, and inspection. Performing autonomous coverage using a single robot system consumes time and energy. In particular, 3D large structures might contain some complex and occluded areas that shall be scanned rapidly in certain application domains. In this paper, a new Hybrid Coverage Path Planning (HCPP) approach is proposed to explore and cover unknown 3D large structures using a decentralized multi-robot system. The HCPP approach combines a guided Next Best View (NBV) approach with a developed Long Short Term Memory (LSTM) waypoint prediction approach to decrease the CPP exploration time at each iteration and simultaneously achieve high coverage. The hybrid approach is the new ML paradigm which fosters intelligence by balancing between data efficiency and generality allowing the exchange of some CPP parts with a learned model. The HCPP uses a stateful LSTM network architecture which is trained based on collected paths that cover different 3D structures to predict the next viewpoint. This architecture captures the dynamic dependencies of adjacent viewpoints in the long term sequences like the coverage paths. The HCPP switches between these methods triggered by either the number of iterations or an entropy threshold. In the decentralized multi-robot system, the proposed HCPP is embedded in each robot where each one of them shares its global 3D map ensuring robustness. The results performed in a realistic Gazebo robotic simulator confirmed the advantage of the proposed HCPP approach by achieving high coverage on different 3D unknown structures in a shorter time compared to conventional NBV.

The 3D reconstruction process is very important in a variety of computer vision applications. Bundle adjustment has a significant impact on 3D reconstruction processes, namely in Simultaneously Localization and Mapping (SLAM) and Structure from Motion (SfM). Bundle adjustment, which optimizes camera parameters and 3D points as a very important final stage, suffers from memory and efficiency requirements in very large-scale reconstruction. Multi-objective optimization (MOO) is used in solving a variety of realistic engineering problems. Multi-Objective Particle Swarm Optimization (MOPSO) is regarded as one of the state of the art for meta-heuristic MOO. MOPSO has utilized the concept of crowding distance as a measure to differentiate between solutions in the search space and provide a high level of exploration. However, this method ignores the direction of the exploration which is not sufficient to effectively explore the search space. In addition, MOPSO starts the search from a fully randomly initialized swarm without taking any prior knowledge about the initial guess into account, which is considered impractical in applications where we can estimate initial values for solutions like bundle adjustment. In this paper, we introduced a novel hybrid MOPSO-based bundle adjustment algorithm that takes advantage of initial guess, angle quantization technique, and traditional optimization algorithms like RADAM to improve the mobility of MOPSO solutions; the results showed that our algorithm can help improve the accuracy and efficiency of bundle adjustment (BA).

Due to the self-obscuration, traditional 3D reconstruction algorithms have difficulty in recovering the 3D structure of an object from a single image. With the rapid development of convolutional neural networks, 3D reconstruction based on deep learning has attracted a wide range of attention from researchers. However, it is expensive to obtain the 3D supervised data corresponding to the objects. To solve the above problems, we combine convolutional neural networks with differentiable renderer and propose the Mesh_CA in this paper, which enables reconstruction of a single image without 3D supervision data. Specifically, an ellipsoid is first initialized for each input single view, and then the features extracted by the convolutional neural network are used to guide the deformation of the ellipsoid to obtain the generated 3D object; After that, the generated object is passed into the differentiable renderer and its corresponding contour information is output; finally, calculating the error between the predicted contour and the real one, and the final 3D object is obtained after training and testing. By training and testing on five types of furniture objects on a large-scale public dataset ShapeNet, the performance of the proposed Mesh_CA surpasses current classical methods.

Scalable geometry reconstruction and understanding is an important yet unsolved task. Current methods often suffer from false loop closures when there are similar-looking rooms in the scene, and often lack online scene understanding. We propose BuildingFusion, a semantic-aware structural building-scale reconstruction system, which not only allows building-scale dense reconstruction collaboratively, but also provides semantic and structural information on-the-fly. Technically, the robustness to similar places is enabled by a novel semantic-aware room-level loop closure detection(LCD) method. The insight lies in that even though local views may look similar in different rooms, the objects inside and their locations are usually different, implying that the semantic information forms a unique and compact representation for place recognition. To achieve that, a 3D convolutional network is used to learn instance-level embeddings for similarity measurement and candidate selection, followed by a graph matching module for geometry verification. On the system side, we adopt a centralized architecture to enable collaborative scanning. Each agent reconstructs a part of the scene, and the combination is activated when the overlaps are found using room-level LCD, which is performed on the server. Extensive comparisons demonstrate the superiority of the semantic-aware room-level LCD over traditional image-based LCD. Live demo on the real-world building-scale scenes shows the feasibility of our method with robust, collaborative, and real-time performance. [ABSTRACT FROM AUTHOR]

We present a novel real-time framework for non-rigid 3D reconstruction that is robust to noise, camera poses, and large deformation from a single depth camera. KinectFusion has achieved high-quality 3D object reconstructions in real-time by implicitly representing an object’s surface with a signed distance field (SDF) representation from a single depth camera. Many studies for incremental reconstruction have been presented since then, with the surface estimation improving over time. Previous works primarily focused on improving conventional SDF matching and deformation schemes. In contrast to these works, the proposed framework tackles the problem of temporal inconsistency caused by SDF approximation and fusion to manipulate SDFs and reconstruct a target more accurately over time. In our reconstruction pipeline, we introduce a refinement evolution method, where an erroneous SDF from a depth sensor is recovered more accurately in a few iterations by propagating erroneous SDF values from the surface. Reliable gradients of refined SDFs enable more accurate non-rigid tracking of a target object. Furthermore, we propose a level-set evolution for SDF fusion, enabling SDFs to be manipulated stably in the reconstruction pipeline over time. The proposed methods are fully parallelizable and can be executed in real-time. Qualitative and quantitative evaluations show that incorporating the refinement and fusion methods into the reconstruction pipeline improves 3D reconstruction accuracy and temporal reliability by avoiding cumulative errors over time. Evaluation results show that our pipeline results in more accurate reconstruction that is robust to noise and large motions, as well as outperforms previous state-of-the-art reconstruction methods. [ABSTRACT FROM AUTHOR]

View-synthesis-based methods have shown very promising results for the task of unsupervised depth estimation in single images. Most existing approaches synthesize a new image and employ it as the supervision signal for depth and pose prediction. There are two problems in these approaches: 1) There are many combinations of pose and depth that can synthesize a certain new image; therefore, reconstructing the depth and pose based on the view-synthesis method from only two images is an inherently ill-posed problem; 2) The model is trained under the photometric consistency assumption that the brightness or gradient is constant when applied to the video sequences. However, this assumption is easily violated in realistic scenes due to light changes, reflective surfaces and occlusions. To overcome the first drawback, we exploit the point cloud consistency constraint to eliminate ambiguity. To overcome the second drawback, we use threshold masks to filter dynamic and occluded points and introduce matching point constraints that implicitly encode the geometry relationship between two matched points to improve the precision of depth prediction. In addition, we employ epipolar constraints to compensate for the instability of the photometric error in textureless regions and varying illumination conditions. The experimental results on the KITTI, Cityscapes and NYUv2 datasets show that the method can improve the accuracy of depth prediction and enhance the robustness of the model in handling textureless regions and illumination changes. The code and data are available at https://github.com/XTUPRLAB/FixUnDepth. [ABSTRACT FROM AUTHOR]

Inspired by simultaneous localization and mapping (SLAM) style workflow, this article presented an online sequential structure from motion (SfM) solution for high-frequency video and large baseline high-resolution aerial images with high efficiency and novel precision. First, as traditional SLAM systems are not good in processing low overlap images, based on our novel hierarchical feature matching paradigm with multihomography and BoW, we proposed a robust tracking method where the relative pose and its scale are estimated separately followed by a joint optimization by considering both perspective-n-point (PnP) and epipolar constraints. Second, to further optimize the camera poses for the sparse map and dense pointcloud reconstruction, we provided a graph-based optimization with reprojection and GPS constraints, which make the camera trajectory and map georeferenced. We also incrementally generated the dense point cloud in real time from keyframes after local mapping optimization. Finally, we use a publicly available aerial image dataset with sequences of different environments, to evaluate the effectiveness of the proposed method, meanwhile, the robust performance of our solution is demonstrated with applications of high-quality aerial images mosaic and digital surface model (DSM) reconstruction in real time. Compared with the state-of-the-art SLAM and traditional SfM methods, the presented system can output large-scale high-quality ortho-mosaic and DSM in real time with the low computational cost. [ABSTRACT FROM AUTHOR]

A large-scale and high-quality image collection is a fundamental demand in the 3D reconstruction scenario. Crowdsourcing can help us to collect lots of diversified images. However, it is difficult to attract people to accomplish tasks due to their self-interest. Besides, the quality of collected images is various. Low-quality images may degrade the performance of 3D reconstruction. To avoid low-quality images and motivate participants to provide high-quality images, we take image quality into account when allocating rewards. In this article, we propose a pricing mechanism, called ImgPricing, to determine the rewards of participants in 3D reconstruction. We model the process of image collection as a cooperative game, and regard image quality and the arrival sequence of images as critical factors in the reward allocation. ImgPricing differs from traditional pricing schemes, e.g., Shapley value and Banzhaf power index-based methods, in that it introduces the images’ arrival sequence to be an indispensable element. We lastly implement ImgPricing on the Android platform and extensively evaluate its performance. Our evaluation results demonstrate that ImgPricing outperforms other existing schemes in terms of computational efficiency, fairness, and robustness. In brief, our image quality-based pricing mechanism for crowdsourced 3D reconstruction is feasible and effective. [ABSTRACT FROM AUTHOR]

We present a novel online 3D scanning system for high-quality object reconstruction with a mobile device, called Mobile3DScanner. Using a mobile device equipped with an embedded RGBD camera, our system provides online 3D object reconstruction capability for users to acquire high-quality textured 3D object models. Starting with a simultaneous pose tracking and TSDF fusion module, our system allows users to scan an object with a mobile device to get a 3D model for real-time preview. After the real-time scanning process is completed, the scanned 3D model is globally optimized and mapped with multi-view textures as an efficient postprocess to get the final textured 3D model on the mobile device. Unlike most existing state-of-the-art systems which can only scan homeware objects such as toys with small dimensions due to the limited computation and memory resources of mobile platforms, our system can reconstruct objects with large dimensions such as statues. We propose a novel visual-inertial ICP approach to achieve real-time accurate 6DoF pose tracking of each incoming frame on the front end, while maintaining a keyframe pool on the back end where the keyframe poses are optimized by local BA. Simultaneously, the keyframe depth maps are fused by the optimized poses to a TSDF model in real-time. Especially, we propose a novel adaptive voxel resizing strategy to solve the out-of-memory problem of large dimension TSDF fusion on mobile platforms. In the post-process, the keyframe poses are globally optimized and the keyframe depth maps are optimized and fused to obtain a final object model with more accurate geometry. The experiments with quantitative and qualitative evaluation demonstrate the effectiveness of the proposed 3D scanning system based on a mobile device, which can successfully achieve online high-quality 3D reconstruction of natural objects with larger dimensions for efficient AR content creation. [ABSTRACT FROM AUTHOR]