Search

We introduce a comprehensive benchmark for local features and robust estimation algorithms, focusing on the downstream task -- the accuracy of the reconstructed camera pose -- as our primary metric. Our pipeline's modular structure allows easy integration, configuration, and combination of different methods and heuristics. This is demonstrated by embedding dozens of popular algorithms and evaluating them, from seminal works to the cutting edge of machine learning research. We show that with proper settings, classical solutions may still outperform the perceived state of the art. Besides establishing the actual state of the art, the conducted experiments reveal unexpected properties of Structure from Motion (SfM) pipelines that can help improve their performance, for both algorithmic and learned methods. Data and code are online https://github.com/vcg-uvic/image-matching-benchmark, providing an easy-to-use and flexible framework for the benchmarking of local features and robust estimation methods, both alongside and against top-performing methods. This work provides a basis for the Image Matching Challenge https://vision.uvic.ca/image-matching-challenge., Comment: Added: KeyNet-SOSNet, AffNet-HardNet, TFeat, MKD from kornia

Freehand sketch is a natural and intuitive communication channel for idea expression. Lines are drawn one after another to create the outline before additional lines are often drawn over the existing ones to make a sketch clearer. Typically, the sketch is transformed to be a 3D CAD model by a designer for use in subsequent operations. Sketch-based modeling has been researched to support this transformation but mainly for an online sketch on a digital device. For an offline sketch on a paper, commonly found used in practice, sketch-based modeling remains a challenge because a scanned image conceals enriched information on the sketch in a batch of data points. This paper proposes an approach for reconstructing a 3D model from a paper-based overtraced freehand sketch. Two main modules in this approach are single-line drawing identification and 3D reconstruction. The first module where image processing technique is applied is for transforming a paper-based overtraced freehand sketch to be a single-line drawing in order to generate more information about the sketch (i.e., the number of lines and their starting points and endpoints). The second module where progressive reconstruction approach with cubic corner is applied is for reconstructing a 3D model from the obtained single-line drawing. Steps to be taken in both modules have been formalized. The approach has been successfully implemented on LabVIEW, and tested with several samples.

This paper presents a novel method for automated manipulation of individual paper fiber bonds using a microrobotic platform, a computer vision algorithm and a robotic software framework. This is a challenging task due to the three-dimensional, heterogeneous and complex morphology of the fiber bonds. The goal is to automatically grasp the fiber bond, and break it by pulling apart the fibers it consists of. We present the components of the microrobotic platform, and the different rules utilized in detecting suitable grasp points from a 3D reconstruction of the bond generated from an image pair. We demonstrate the functionality of the approach with bond breaking experiments of seven fiber bonds. The time required for grasping and breaking of a bond is 10 – 15 seconds making the approach much faster than the current state-of-the-art testing, which is based on manual manipulation. The success rate of the tests is as high as 80 %. acceptedVersion

This paper presents propagation-based phase-contrast tomography in two configurations at the beamline endstation GINIX, demonstrated on the application of 1 mm human pancreatic tumor tissue biopsies., A multiscale three-dimensional (3D) virtual histology approach is presented, based on two configurations of propagation phase-contrast X-ray tomography, which have been implemented in close proximity at the GINIX endstation at the beamline P10/PETRA III (DESY, Hamburg, Germany). This enables the 3D reconstruction of characteristic morphological features of human pancreatic normal and tumor tissue, as obtained from cancer surgery, first in the form of a large-scale overview by parallel-beam illumination, followed by a zoom into a region-of-interest based on zoom tomography using a Kirkpatrick–Baez mirror with additional waveguide optics. To this end 1 mm punch biopsies of the tissue were taken. In the parallel tomography, a volumetric throughput on the order of 0.01 mm3 s−1 was achieved, while maintaining the ability to segment isolated cells. With a continuous rotation during the scan, a total acquisition time of less than 2 min was required for a full tomographic scan. Using the combination of both setups, islets of Langerhans, a three-dimensional cluster of cells in the endocrine part of the pancreas, could be located. Cells in such an islet were segmented and visualized in 3D. Further, morphological alterations of tumorous tissue of the pancreas were characterized. To this end, the anisotropy parameter Ω, based on intensity gradients, was used in order to quantify the presence of collagen fibers within the entire biopsy specimen. This proof-of-concept experiment of the multiscale approach on human pancreatic tissue paves the way for future 3D virtual pathology.

Typical skeletal muscles are composed of mixed muscle fiber types, which are classified as slow-twitch (type I) and fast-twitch (type II) fibers, whereas pectoralis major muscles (PMs) in broiler chickens are 100% composed of type IIb fast-twitch fibers. Since metabolic properties differ among muscle fiber types, the combination of muscle fiber types is involved in physiological functions and pathological conditions in skeletal muscles. In this study, using serial block-face scanning electron microscopy, we compared three-dimensional (3D) mitochondrial properties in type IIb fibers in broiler PMs and those in type I fibers of broiler gastrocnemius muscles (GMs) heterogeneously composed of slow- and fast-twitch muscle fibers. In type I fibers in the GMs, elongated mitochondria with numerous interconnections to form a substantial network among myofibrils were observed. Along with lipid droplets sandwiched by mitochondria, these features are an adaptation to effective oxidative respiration and constant oxidative damage in slow-twitch muscle fibers. In contrast, type IIb fibers in the PMs showed small and ellipsoid-shaped mitochondria with few interconnections and no lipid droplets, forming a sparse network. The mitochondrial spatial network comprises of active mitochondrial dynamics to reduce mitochondrial damage; therefore, type IIb fibers possess physiologically low capacity to maintain mitochondrial wellness due to static mitochondrial dynamics. Based on 3D mitochondrial properties, we discussed the contrasting physiological functions between type I and IIb fibers and proposed a high contractile power and low stress resistance as unique physiological properties of broiler PMs.

Significance: Dental caries is the most common oral disease, with significant effects on healthcare systems and quality of life. Developing diagnostic methods for early caries detection is key to reducing this burden and enabling non-invasive treatment as opposed to the drill-and-fill approach. Aim: The application of a thermophotonic-based 3D imaging modality [enhanced truncated-correlation photothermal coherence tomography (eTC-PCT)] to early dental caries is investigated. To this end, the detection threshold, sensitivity, and 3D lesion reconstruction capability of eTC-PCT in imaging artificially generated caries and surface erosion are evaluated. Approach: eTC-PCT employs a diode laser with pulsed excitation, a mid-IR camera, and an in-house developed image reconstruction algorithm to produce depth-resolved 2D images and 3D reconstructions. Starting with healthy teeth, dental caries and surface erosion are simulated in vitro through application of specific demineralizing/eroding acidic solutions. Results: eTC-PCT can detect artificial caries as early as 2 days after onset of artificial demineralization and after 45 s of surface erosion, with a laser power equivalent to 64% of maximum permissible exposure. In both cases, the lesion is not visible to the eye and undetected by x-rays. eTC-PCT is capable of monitoring lesion progression in 2-day increments and generating 3D tomographic reconstructions of the advancing lesion. Conclusions: eTC-PCT shows great potential for further development as a dental imaging modality combining low detection threshold, high sensitivity to lesion progression, 3D reconstruction capability, and lack of ionizing radiation. These features enable early diagnosis and frequent monitoring, making eTC-PCT a promising technology for facilitating preventive dentistry.

The structure of a common virulence factor expressed on the surface of diarrhea-causing bacteria, CFA/I pili, has been determined at 4.3 Å resolution. The role of Pro13 in stabilizing the pilus structure has been investigated using force-measuring optical tweezers on wild-type and point-mutated pili., Enterotoxigenic Escherichia coli (ETEC) are common agents of diarrhea for travelers and a major cause of mortality in children in developing countries. To attach to intestinal cells ETEC express colonization factors, among them CFA/I, which are the most prevalent factors and are the archetypical representative of class 5 pili. The helical quaternary structure of CFA/I can be unwound under tensile force and it has been shown that this mechanical property helps bacteria to withstand shear forces from fluid motion. We report in this work the CFA/I pilus structure at 4.3 Å resolution from electron cryomicroscopy (cryo-EM) data, and report details of the donor strand complementation. The CfaB pilins modeled into the cryo-EM map allow us to identify the buried surface area between subunits, and these regions are correlated to quaternary structural stability in class 5 and chaperone–usher pili. In addition, from the model built using the EM structure we also predicted that residue 13 (proline) of the N-terminal β-strand could have a major impact on the filament’s structural stability. Therefore, we used optical tweezers to measure and compare the stability of the quaternary structure of wild type CFA/I and a point-mutated CFA/I with a propensity for unwinding. We found that pili with this mutated CFA/I require a lower force to unwind, supporting our hypothesis that Pro13 is important for structural stability. The high-resolution CFA/I pilus structure presented in this work and the analysis of structural stability will be useful for the development of novel antimicrobial drugs that target adhesion pili needed for initial attachment and sustained adhesion of ETEC.

The incorporation of prior knowledge about the structures of biological macromolecules into the reconstruction process of cryo-EM structure determination is proposed. Using a novel algorithm inspired by regularization by denoising, it is shown how convolutional neural networks can be used within this framework to improve reconstructions from simulated data., Three-dimensional reconstruction of the electron-scattering potential of biological macromolecules from electron cryo-microscopy (cryo-EM) projection images is an ill-posed problem. The most popular cryo-EM software solutions to date rely on a regularization approach that is based on the prior assumption that the scattering potential varies smoothly over three-dimensional space. Although this approach has been hugely successful in recent years, the amount of prior knowledge that it exploits compares unfavorably with the knowledge about biological structures that has been accumulated over decades of research in structural biology. Here, a regularization framework for cryo-EM structure determination is presented that exploits prior knowledge about biological structures through a convolutional neural network that is trained on known macromolecular structures. This neural network is inserted into the iterative cryo-EM structure-determination process through an approach that is inspired by regularization by denoising. It is shown that the new regularization approach yields better reconstructions than the current state of the art for simulated data, and options to extend this work for application to experimental cryo-EM data are discussed.

We reconstructed serial sections of a representative adult human spleen to clarify the unknown arrangement of the splenic microvasculature, such as terminal arterioles, sheathed capillaries, the red pulp capillary network and venules. The resulting 3D model was evaluated in virtual reality (VR). Capillary sheaths often occurred after the second or third branching of a terminal arteriole and covered its capillary side or end branches. The sheaths started directly after the final smooth muscle cells of the arteriole and consisted of cuboidal CD271++ stromal sheath cells surrounded and infiltrated by B lymphocytes and macrophages. Some sheaths covered up to four sequential capillary bifurcations thus forming bizarre elongated structures. Each sheath had a unique form. Apart from symmetric dichotomous branchings inside the sheath, sheathed capillaries also gave off side branches, which crossed the sheath and freely ended at its surface. These side branches are likely to distribute materials from the incoming blood to sheath-associated B lymphocytes and macrophages and thus represent the first location for recognition of blood-borne antigens in the spleen. A few non-sheathed bypasses from terminal arterioles to the red pulp capillary network also exist. Red pulp venules are primarily supplied by sinuses, but they also exhibit a few connections to the capillary network. Thus, the human splenic red pulp harbors a primarily open microcirculation with a very minor closed part.

Purpose: Reconstruction of patient anatomy is critical to patient-specific instrument (PSI) design in total joint replacement (TJR). Conventionally, computed tomography (CT) and magnetic resonance imaging (MRI) are used to obtain the patient anatomy as they are accurate imaging modalities. However, computing anatomical landmarks from the patient anatomy for PSIs requires either high-resolution CT, increasing time of scan and radiation exposure to the patient, or longer and more expensive MRI scans. As an alternative, reconstruction from single-plane fluoroscopic x-ray provides a cost-efficient tool to obtain patient anatomical structures while allowing capture of the patient’s joint dynamics, important clinical information for TJR. Approach: We present a three-dimensional (3D) reconstruction scheme that automatically and accurately reconstructs the 3D knee anatomy from single-plane fluoroscopic x-ray based on a nonlinear statistical shape model called kernel principal component analysis. To increase robustness, we designed a hybrid energy function that integrated feature and intensity information as a similarity measure for the 3D reconstruction. Results: We evaluated the proposed method on five subjects during deep knee bending: the root-mean-square accuracy is 1.19±0.36 mm for reconstructed femur and 1.15±0.17 mm for reconstructed tibia. Conclusions: The proposed method demonstrates reliable 3D bone model reconstruction accuracy with successful elimination of prior 3D imaging and reduction of manual labor and radiation dose on patient as well as characterizing joints in motion. This method is promising for applications in medical interventions such as patient-specific arthroplasty design, surgical planning, surgical navigation, and understanding anatomical and dynamic characteristics of joints.

In the first hours of a disaster, up-to-date information about the area of interest is crucial for effective disaster management. However, due to the delay induced by collecting and analysing satellite imagery, disaster management systems like the Copernicus Emergency Management Service (EMS) are currently not able to provide information products until up to 4872 h after a disaster event has occurred. While satellite imagery is still a valuable source for disaster management, information products can be improved through complementing them with user-generated data like social media posts or crowdsourced data. The advantage of these new kinds of data is that they are continuously produced in a timely fashion because users actively participate throughout an event and share related information. The research project Evolution of Emergency Copernicus services (E2mC) aims to integrate these novel data into a new EMS service component called Witness, which is presented in this paper. Like this, the timeliness and accuracy of geospatial information products provided to civil protection authorities can be improved through leveraging user-generated data. This paper sketches the developed system architecture, describes applicable scenarios and presents several preliminary case studies, providing evidence that the scientific and operational goals have been achieved. (VLID)2350401

Laser remelting has been performed on Ni-30 wt.% Sn hypoeutectic alloy. An anomalous eutectic formed at the bottom of the molten pool when the sample was remelted thoroughly. 3D morphologies of the α-Ni and Ni3Sn phases in the anomalous eutectic region were obtained and investigated using serial sectioning reconstruction technology. It is found that the Ni3Sn phase has a continuous interconnected network structure and the α-Ni phase is distributed as separate particles in the anomalous eutectic, which is consistent with the electron backscatter diffraction pattern examinations. The α-Ni particles in the anomalous eutectic are supersaturated with Sn element as compared with the equilibrium phase diagram. Meanwhile, small wavy lamella eutectics coexist with anomalous eutectics. The Trivedi-Magnin-Kurz model was used to estimate undercooling with lamellar spacing. The results suggest that the critical undercooling found in undercooling solidification is not a sufficient condition for anomalous eutectic formation. Besides, α-Ni particles in the anomalous eutectic do not exhibit a completely random misorientation and some neighboring α-Ni particles have the same orientation. It is shown that both the coupled and decoupled growth of the eutectic two phases can generate the α-Ni + Ni3Sn anomalous eutectic structure.

In this work, data from two-dimensional (2D) images of the human retina were taken as a case study. First, the characteristic data points had been removed using the Douglas–Peucker (DP) method, and subsequently, more data points were added using random fractal interpolation approach, to reconstruct a three-dimensional (3D) model of the blood vessel. By visualizing the result, we can see that all the small blood vessels in the human retina are more visible and detailed. This algorithm of 3D reconstruction has the advantage of being fast with calculation time less than 40 s and also can reduce the 3D image storage level on a disk with a reduction ratio between 78% and 96.65%.

For 3D reconstructions of whole immune cells from zebrafish, isolated from adult animals by FAC-sorting we employed array tomography on hundreds of serial sections deposited on silicon wafers. Image stacks were either recorded manually or automatically with the newly released ZEISS Atlas 5 Array Tomography platform on a Zeiss FEGSEM. To characterize different populations of immune cells, organelle inventories were created by segmenting individual cells. In addition, arrays were used for quantification of cell populations with respect to the various cell types they contained. The detection of immunological synapses in cocultures of cell populations from thymus or WKM with cancer cells helped to identify the cytotoxic nature of these cells. Our results demonstrate the practicality and benefit of AT for high-throughput ultrastructural imaging of substantial volumes. Lay Description To look at immune cells from zebrafish we employed array tomography, a technique where arrays of serial sections deposited on solid substrates are used for imaging. Cell populations were isolated from the different organs of zebrafish involved in haematopoiesis, the production of blood cells. They were chemically fixed and centrifuged to concentrate them in a pellet that was then dehydrated and embedded in resin. Using a custom-built handling device it was possible to place hundreds of serial sections on silicon wafers as well ordered arrays. To image a whole cell at a resolution that would allow identifying all the organelles (i.e. compartments surrounded by membranes) inside the cell, stacks of usually 50–100 images were recorded in a scanning electron microscope (SEM). This recording was either done manually or automatically using the newly released Atlas Array Tomography platform on a ZEISS SEM. For the imaging of the sections a pixel size of about 5 nm was chosen, which defines membrane boundaries very well and allows segmentation of the membrane topology. After alignment of the images, cellular components were segmented to locate the individual organelles within the 3D reconstruction of the whole cell and also to create an inventory of organelles. Based on their morphologies we could identify specific cell types in the different hematopoietic organs. We could also quantify the proportion of each cell type in the whole population isolated from a given organ. Some of these specific cells from zebrafish were grown in a culture dish together with human cancer cells. By time-lapse light microscopy we observed that the fish cells attacked the cancer cells and killed them. From this we concluded that these cells must be similar to the cytotoxic cells from humans that play an important role in defence against spontaneously arising cancer cells in our bodies. They form special structures, called immunological synapses that we could also identify on our arrays and reconstruct in 3D. This is the first time the potential of zebrafish immune cells to form immunological synapses has been demonstrated. Our study is a good example for the practicality and benefit of array tomography in high-throughput ultrastructure imaging of substantial volumes, applicable to many areas of cell and developmental biology.

Introduction. For the procedure retroperitoneoscopic ureterolithotomy, the problems of access choice and thus visualization with utilizing minimally invasive surgical access (either with gasless single port method or gas insufflation) are solved. The decisions are based on the method of presurgery planning, grounded on matching the patient with a 3D model of the zone of surgical interest reconstructed according to the results of tomographic examination. Material and methods. We used a hardware–software complex (HSC) for virtual modeling of the surgery zone and choosing the optimum points for minimally invasive surgical access. The HSC was recruited to choose optimum surgical access, realize presurgery planning, and estimation of the safety of the way of access chosen. The original method of matching the system of coordinates of a virtual model with the patient was offered. Results. 12 patients with the calculus in the upper part of ureter averaging 11.5 (9–14) mm in size un derwent gasless retroperitoneoscopic ureterolithotomy with use of the HSC. Mean age of the patients was 36.4 (25–49) years old. The surgeries lasted an average of 35.5 (25–40) minutes. Blood loss was averaged at 55.0 (30–90) ml. Healing by first intention was registered with all the patients. The mean hospitalization time was 6.0 (4–7) days. There were neither any complications nor difficulties, nor conversions from incorrectly chosen surgical access. Conclusions. The choice of the optimum surgical access according to the results of a virtual 3D model of the operation zone, matching the system of coordinates of the model with patient concurrence, and presurgery planning, was effective in cases of gasless single port and with gas insufflation retroperitoneoscopic ureterolithotomy. Article history

Background This systematic review aims to ascertain how accurately 3D models can be predicted from two-dimensional (2D) imaging utilising statistical shape modelling. Methods A systematic search of published literature was conducted in September 2022. All papers which assessed the accuracy of 3D models predicted from 2D imaging utilising statistical shape models and which validated the models against the ground truth were eligible. Results 2127 papers were screened and a total of 34 studies were included for final data extraction. The best overall achievable accuracy was 0.45 mm (root mean square error) and 0.16 mm (average error). Conclusion Statistical shape modelling can predict detailed 3D anatomical models from minimal 2D imaging. Future studies should report the intended application domain of the model, the level of accuracy required, the underlying demographics of subjects, and the method in which accuracy was calculated, with root mean square error recommended if appropriate.

In recent years, with the improvement of artificial intelligence technology, it has become possible to reconstruct high-precision 3D human body models based on ordinary RGB images. The current 3D human body reconstruction technology requires complex external equipment to scan all angles of the human body, which is complicated to be implemented and cannot be popularized. In order to solve this problem, this paper applies deep learning models on reconstructing 3D human body based on monocular images. First of all, this paper uses Stacked Hourglass network to perform convolution operations on monocular images collected from different views. Then Multi-Layer Perceptrons (MLPs) are used to decode the encoded high-level images. The feature codes in the two views(main and side) are fused, and the interior and exterior points are classified by the fusion features, so as to obtain the corresponding 3D occupancy field. At last, the Marching Cube algorithm is used for 3D reconstruction with a specific threshold and then we use Laplace smoothing algorithm to remove artifacts. This paper proposes a dense sampling strategy based on the important joint points of the human body, which has a certain optimization effect on the realization of high-precision 3D reconstruction. The performance of the proposed scheme has been validated on the open source datasets, MGN dataset and the THuman dataset, provided by Tsinghua University. The proposed scheme can reconstruct features such as clothing folds, color textures, and facial details,and has great potential to be applied in different applications.

Many odontological studies held through application of traditional and modern techniques, especially when related to measurements and morphology, very much depend on methodological aspects referred to orientation of teeth. And this is particularly relevant to new imaging and 3d reconstruction implemented in dental research and practice in a wide range of disciplines from anthropology to dentistry. The current paper deals with studies of palaeoanthropological findings dating back to the Upper Palaeolithic period in Central Russia – well-known archaeological site of Sunghir. Micro-computed tomography has been used for digital reconstructions of teeth – molars and premolars representing well-preserved dental morphology of an adolescent individual. This is due to new opportunities introduced by 3d reconstruction techniques in general and high-resolution x-ray imaging in particular that this study has become relevant. Thus digital techniques do not only provide for operating convenience but, which is even more important, allow application of image processing algorithms. In the suggested methodology these are automated, based on morphological interpretations and serve for orientation of studied teeth for further measurements. At the same time micro-computed tomographic imaging allows accurate reconstruction of other morphologically important structures which are used for an alternative orientation algorithm. Comparisons of dental measurements’ results obtained through automated digital odontometry (aDo) after both orientations applied are presented in the current paper.

It is possible to construct cost-efficient three-dimensional (3D) or four-dimensional (4D) scanning systems using multiple affordable off-the-shelf RGB-D sensors to produce high-quality reconstructions of 3D objects. However, the quality of these systems’ reconstructions is sensitive to a number of factors in reconstruction pipelines, such as multi-view calibration, depth estimation, 3D reconstruction, and color mapping accuracy, because the successive pipelines to reconstruct 3D meshes from multiple active stereo sensors are strongly correlated with each other. This paper categorizes the pipelines into sub-procedures and analyze various factors that can significantly affect reconstruction quality. Thus, this paper provides analytical and practical guidelines for high-quality 3D reconstructions with off-the-shelf sensors. For each sub-procedure, this paper shows comparisons and evaluations of several methods using data captured by 18 RGB-D sensors and provide analyses and discussions towards robust 3D reconstruction. Through various experiments, it has been demonstrated that significantly more accurate 3D scans can be obtained with the considerations along the pipelines. We believe our analyses, benchmarks, and guidelines will help anyone build their own studio and their further research for 3D reconstruction.

Hypersonic vehicles have been paid more and more attention by various countries, and many developed countries have set off a competition of hypersonic technology. Under this background, this paper creatively applies the reconstruction method based on three vanishing points to the reconstruction of complex aircraft for the first time. The shape of aircraft is restored by photos from different angles, which provides a good help for preliminarily understanding and mastering the information of international mainstream advanced aircraft. In this paper, the feasibility of the whole method is verified by the classical X-37B model, and its three-dimensional structure model can be obtained relatively accurately. In terms of aerodynamic performance evaluation, the panel method is adopted to evaluate the aerodynamic performance on the basis of verifying the reliability of the engineering method. And the results show that the lift-to-drag ratio reaches the maximum of 1.22 at the angle of attack of 25°. This is quite similar to the CFD calculation of class X-37B model in other literatures. The accuracy of the model makes up for the error of the engineering estimation method to a certain extent. In view of the inaccuracy of the wing reconstruction, the genetic algorithm is used to optimize its aerodynamic performances. The lift-to-drag ratio of the wing is increased by 26.3% at 25°, which leads to an increase of 4.35% in the optimal angle of attack lift-to-drag ratio of the whole model. At last, the modularization programming approach is adopted in the optimization process, ensuring the generality of this method, and the whole method will have a broad application prospect.

To overcome the problems of long production cycle and high cost in the product manufacturing process, a P2P (platform to platform) cloud manufacturing method based on a personalized custom business model has been proposed in this paper by integrating different technologies such as deep learning and additive manufacturing (AM). This paper focuses on the manufacturing process from a photo containing an entity to the production of that entity. Essentially, this is an object-to-object fabrication. Moreover, based on the YOLOv4 algorithm and DVR technology, an object detection extractor and a 3D data generator are constructed, and a case study is carried out for a 3D printing service scenario. The case study selects online sofa photos and real car photos. The recognition rates of sofa and car were 59% and 100%, respectively. Retrograde conversion from 2D data to 3D data takes approximately 60 s. We also carry out personalized transformation design on the generated sofa digital 3D model. The results show that the proposed method has been validated, and three unindividualized models and one individualized design model have been manufactured, and the original shape is basically maintained.

After a lengthy period of scouring, the reinforced concrete surface of the dam spillway (i.e., drift spillways and flood discharge spillways) will suffer from deterioration and damage. Regular manual inspection is time-consuming and dangerous. This paper presents a robotic solution to detect automatically, count defect instance numbers, and reconstruct the surface of dam spillways by incorporating the deep learning method with a visual 3D reconstruction method. The lack of a real dam defect dataset and incomplete registration of minor defects on the 3D mesh model in fusion step are two challenges addressed in the paper. We created a multi-class semantic segmentation dataset of 1711 images (with resolutions of 848 × 480 and 1280 × 720 pixels) acquired by a wall-climbing robot, including cracks, erosion, spots, patched areas, and power safety cable. Then, the architecture of the U-net is modified with pixel-adaptive convolution (PAC) and conditional random field (CRF) to segment different scales of defects, trained, validated, and tested using this dataset. The reconstruction and recovery of minor defect instances in the flow surface and sidewall are facilitated using a keyframe back-projection method. By generating an instance adjacency matrix within the class, the intersection over union (IoU) of 3D voxels is calculated to fuse multiple instances. Our segmentation model achieves an average IoU of 60% for five defect class. For the surface model’s semantic recovery and instance statistics, our method achieves accurate statistics of patched area and erosion instances in an environment of 200 m2, and the average absolute error of the number of spots and cracks has reduced from the original 13.5 to 3.5.

Currently, the three-dimensional detection of plant root structure is one of the core issues in studies on plant root phenotype. Manual measurement methods are not only cumbersome but also have poor reliability and damage the root. Among many solutions, X-ray computed tomography (X-ray CT) can help us observe the plant root structure in a three-dimensional and non-destructive form under the condition of underground soil in situ. Therefore, this paper proposes a high-throughput method and process for plant three-dimensional root phenotype and reconstruction based on X-ray CT technology. Firstly, this paper proposes a high-throughput transmission for the root phenotyping and utilizing the imaging technique to extract the root characteristics; then, the study adopts a moving cube algorithm to reconstruct the 3D (three-dimensional) root. Finally, this research simulates the proposed algorithm, and the simulation results show that the presented method in this paper works well.

This paper addresses the challenge of enriching geometric digital twins of buildings, with a particular emphasis on capturing small but important entities from the electrical and the re-safety domain, such as signs, sockets, switches, smoke alarms, etc. Unlike most previous research that focussed on structural elements and processed laser point clouds and images separately, we propose a novel method that fuses laser scanning and photogrammetry methods to capture the relevant objects, recognise them in 2D images and then map these to a 3D space. The considered object classes include electrical elements (light switch, light, speaker, socket, elevator button), safety elements (emergency switch, smoke alarm, re extinguisher, escape sign), plumbing system elements (pipes), and other objects with useful information (door sign, board). Semantic information like class labels is extracted by applying AI-based image segmentation and then mapped to the 3D point cloud, segmenting the point cloud into point clusters. We subsequently fi t geometric primitives to the point clusters and extract text information by AI-based text detection and recognition. The final output of our proposed method is an information-rich digital twin of buildings that contains geometric information, semantic information such as object categories and useful text information which is valuable in many aspects, like condition monitoring, facility maintenance and management. In summary, the paper presents a nearly fully-automated pipeline to enrich a geometric digital twin of buildings with details and provides a comprehensive case study.

This paper proposes a fused 3D transparent visualization method with the aim of achieving see-through imaging of large-scale cultural heritage by combining photogrammetry point cloud data and 3D reconstructed models. 3D reconstructed models are efficiently reconstructed from a single monocular photo using deep learning. It is demonstrated that the proposed method can be widely applied, particularly to instances of incomplete cultural heritages. In this study, the proposed method is applied to a typical example, the Borobudur temple in Indonesia. The Borobudur temple possesses the most complete collection of Buddhist reliefs. However, some parts of the Borobudur reliefs have been hidden by stone walls and became not visible following the reinforcements during the Dutch rule. Today, only gray-scale monocular photos of those hidden parts are displayed in the Borobudur Museum. In this paper, the visible parts of the temple are first digitized into point cloud data by photogrammetry scanning. For the hidden parts, a 3D reconstruction method based on deep learning is proposed to reconstruct the invisible parts into point cloud data directly from single monocular photos from the museum. The proposed 3D reconstruction method achieves 95% accuracy of the reconstructed point cloud on average. With the point cloud data of both the visible parts and the hidden parts, the proposed transparent visualization method called the stochastic point-based rendering is applied to achieve a fused 3D transparent visualization of the valuable temple.

Three-dimensional (3D) shape estimation of the human body has a growing number of applications in medicine, anthropometry, special effects, and many other fields. Therefore, the demand for the high-quality acquisition of a complete and accurate body model is increasing. In this paper, a short survey of current state-of-the-art solutions is provided. One of the most commonly used approaches is the Shape-from-Silhouette (SfS) method. It is capable of the reconstruction of dynamic and challenging-to-capture objects. This paper proposes a novel approach that extends the conventional voxel-based SfS method with silhouette segmentation—segmented Shape from Silhouette (sSfS). It allows the 3D reconstruction of body segments separately, which provides significantly better human body shape estimation results, especially in concave areas. For validation, a dataset representing the human body in 20 complex poses was created and assessed based on the quality metrics in reference to the ground-truth photogrammetric reconstruction. It appeared that the number of invalid reconstruction voxels for the sSfS method was 1.7 times lower than for the state-of-the-art SfS approach. The root-mean-square (RMS) error of the distance to the reference surface was also 1.22 times lower.

This paper presents an autonomous grasping approach for complex-shaped objects using an anthropomorphic robotic hand. Although human-like robotic hands have a number of distinctive advantages, most of the current autonomous robotic pickup systems still use relatively simple gripper setups such as a two-finger gripper or even a suction gripper. The main difficulty of utilizing human-like robotic hands lies in the sheer complexity of the system; it is inherently tough to plan and control the motions of the high degree of freedom (DOF) system. Although data-driven approaches have been successfully used for motion planning of various robotic systems recently, it is hard to directly apply them to high-DOF systems due to the difficulty of acquiring training data. In this paper, we propose a novel approach for grasping complex-shaped objects using a high-DOF robotic manipulation system consisting of a seven-DOF manipulator and a four-fingered robotic hand with 16 DOFs. Human demonstration data are first acquired using a virtual reality controller with 6D pose tracking and individual capacitive finger sensors. Then, the 3D shape of the manipulation target object is reconstructed from multiple depth images recorded using the wrist-mounted RGBD camera. The grasping pose for the object is estimated using a residual neural network (ResNet), K-means clustering (KNN), and a point-set registration algorithm. Then, the manipulator moves to the grasping pose following the trajectory created by dynamic movement primitives (DMPs). Finally, the robot performs one of the object-specific grasping motions learned from human demonstration. The suggested system is evaluated by an official tester using five objects with promising results.

Semantic segmentation directly from the images of landfills can be utilized in the earth movers to segregate the garbage autonomously. Generally, Various segregation methods are available for garbage segregation such as IOT based waste segregation, Conveyor belt segregation in which none of them are directly from landfills. Semantic segmentation is one of the important tasks that maps the path towards the complete scene understanding. The aim of this paper is to present a smart segregation method for garbage by using semantic segmentation with DeepLab V3+ Model using the framework(Backbone model) of Xception-65 with the mean accuracy of 75.01%. This paper features the segmentation with the GarbotV1dataset which has major classifications such as Plastic, Cart-board, Wood, Metal, Sponge. The paper also contributes a method for reconstructing the segmented images to build a 3D map and this exploits the use of earth moving vehicles to navigate autonomously by localizing the segmented objects.

In this paper we introduce a new camera localization strategy designed for image sequences captured in challenging industrial situations such as industrial parts inspection. To deal with peculiar appearances that hurt standard 3D reconstruction pipeline, we exploit pre-knowledge of the scene by selecting key frames in the sequence (called as anchors) which are roughly connected to a certain location. Our method then seek the location of each frame in time-order, while recursively updating an augmented 3D model which can provide current camera location and surrounding 3D structure. In an experiment on a practical industrial situation, our method can localize over 99% frames in the input sequence, whereas standard localization methods fail to reconstruct a complete camera trajectory., Comment: This paper have been accepted and will be appeared in the proceedings of 17th International Conference on Machine Vision Applications (MVA2021)

Micro computed tomography (µCT) allows the noninvasive visualization and 3D reconstruction of internal structures of objects with high resolution. However, the current commercial µCT system relatively rotates the source-detector or objects to collect projections, referred as RCT in this paper, and has difficulties in imaging large objects with high resolutions because fabrication of large-area, inexpensive flat-panel detectors remains a challenge. In this paper, we proposed a source translation based CT (STCT) for imaging large objects with high resolution to get rid of the limitation of the detector size, where the field of view is primarily determined by the source translation distance. To compensate for the deficiency of incomplete data in STCT, we introduced multi-scanning STCT (mSTCT), from which the projections theoretically meet the conditions required for accurate reconstructions. Theoretical and numerical studies showed that mSTCT has the ability to accurately image large objects without any visible artifacts. Numerical simulations also indicated that mSTCT has a potential capability to precisely image the region of interest (ROI) inside objects, which remains a challenge in RCT due to truncated projections. In addition, an experimental platform for mSTCT has been established, from which the 2D and 3D reconstructed results demonstrated its feasibility for µCT applications. Moreover, STCT also has a great potential for security inspection and product screening by using two perpendicular STCTs, with advantages of low-cost equipment and high-speed examination.

The application of artificial intelligence and machine learning technology in the field of wireless communication has received great attention. The success of artificial intelligence in speech understanding, image recognition, natural language processing and other fields shows its great potential to solve the problem of difficult modeling. Aiming at the problem of 3D reconstruction of defocused blur spots, a method based on adaptive defocus blur radius estimation and double hidden layer BP neural network is proposed. This method first uses an adaptive segmentation algorithm to extract the approximate elliptical spot in the defocused blurred image, and then uses the gradient amplitude distribution to extract the defocused blurred area, thereby calculating the blur radius. Then, using the network to adaptively learn the geometric structure relationship between the target 3D position, the target image point position and the defocus blur radius, it is established that the center pixel coordinates of the light spot and the corresponding defocus blur radius are used as input, and the 3D coordinates of the target are used as input. Output, the two hidden layers in the middle have established a neural network for the 3D reconstruction of the defocused blur spot. The experimental results show that the double hidden layer BP network model proposed in this paper can realize the 3D reconstruction of the blurred spot after training, and the method proposed in this paper has higher 3D reconstruction accuracy than without considering the defocus blur effect.

Recent years have seen the rapid growth of new approaches to optical imaging, with an emphasis on extracting three-dimensional (3D) information from what is normally a two-dimensional (2D) image capture. Perhaps most importantly, the rise of computational imaging enables both new physical layouts of optical components and new algorithms to be implemented. This paper concerns the convergence of two advances: the development of a transparent focal stack imaging system using graphene photodetector arrays, and the rapid expansion of the capabilities of machine learning including the development of powerful neural networks. This paper demonstrates 3D tracking of point-like objects with multilayer feedforward neural networks and the extension to tracking positions of multi-point objects. Computer simulations further demonstrate how this optical system can track extended objects in 3D, highlighting the promise of combining nanophotonic devices, new optical system designs, and machine learning for new frontiers in 3D imaging., Transparent photodetectors based on graphene stacked vertically along the optical axis have shown promising potential for light field reconstruction. Here, the authors develop transparent photodetector arrays and implement a neural network for real-time 3D optical imaging and object tracking.

Depth map has been utilized for refinement of geometric information in a variety of fields such as 3D reconstruction and pose estimation in SLAM system where ill-posed problems are occurred. Currently, as learning-based approaches are successfully introduced throughout many problems of vision-based fields, several depth estimation algorithms based on CNN are suggested, which only conduct training of spatial information. Since an image sequence or video used for SLAM system tends to have temporal information, this paper proposes a recurrent CNN architecture for SLAM system to estimate depth map by exploring not only spatial but also temporal information by using convolutional GRU cell, which is constructed to remember weights of past convolutional layers. Furthermore, this paper proposes using additional layers that preserve structure of scenes by utilizing sparse depth cues obtained from SLAM system. The sparse depth cues are produced by projecting reconstructed 3D map into each camera frame, and the sparse cues help to predict accurate depth map avoiding ambiguity of depth map generation of untrained structures in latent space. Despite accuracy of depth cues according to monocular SLAM system degrades than stereo SLAM system, the proposed masking approach, which takes the confidence of depth cues with regard to a relative camera pose between current frame and previous frame, retains the performance of the proposed system with the proposed adaptive regularization in loss function. In the training phase, by preprocessing exponential quantization of ground-truth depth map to eliminate the ill-effects of the captured large distances, the depth map prediction of the proposed system improves more than other baseline methods with accomplishment of real-time system. We expect that this proposed system can be used in SLAM system to refine geometric information for more accurate 3D reconstruction and pose estimation, which are essential parts for robust navigation system of robots.

This paper proposes a method to reconstruct three-dimensional (3D) objects using real-time fusion and analysis of multiple sensor data. This paper attempts to create a realistic 3D visualization with which a remote pilot can intuitively control a remote unmanned robot by utilizing the characteristics of massive sensor data. The 3D reconstruction system proposed in this paper is comprised of 3D and two-dimensional (2D) data segmentation method, a 3D reconstruction method applied to each object, and a projective texture mapping method. Specifically, we propose applying both a 2D region extraction method and a 3D mesh modeling method to each object. The proposed schemes are implemented as a real-time application to verify real-time performance. This paper proves that 3D meshes can be modeled in real time by using the proposed method. The proposed method allows the remote control of a robot for real-time 3D rendering of remote scenes, which is essential for various tasks in areas that cannot be easily accessed by humans.

In this paper, we propose a model-free volumetric Next Best View (NBV) algorithm for accurate 3D reconstruction using a Markov Chain Monte Carlo method for high-mix-low-volume objects in manufacturing. The volumetric information gain based Next Best View algorithm can in real-time select the next optimal view that reveals the maximum uncertainty of the scanning environment with respect to a partially reconstructed 3D Occupancy map, without any priori knowledge of the target. Traditional Occupancy grid maps make two independence assumptions for computational tractability but suffer from the overconfident estimation of the occupancy probability for each voxel leading to less precise surface reconstructions. This paper proposes a special case of the Markov Chain Monte Carlo (MCMC) method, the Gibbs sampler, to accurately estimate the posterior occupancy probability of a voxel by randomly sampling from its high-dimensional full posterior occupancy probability given the entire volumetric map with respect to the forward sensor model with a Gaussian distribution. Numerical experiments validate the performance of the MCMC Gibbs sampler algorithm under the ROS-Industry framework to prove the accuracy of the reconstructed Occupancy map and the completeness of the registered point cloud. The proposed MCMC Occupancy mapping could be used to optimise the tuning parameters of the online NBV algorithms via the inverse sensor model to realise industry automation.

This paper presents an improved scheme for forward-looking sonar (FLS) visualization, including an echo acquisition method, a global threshold based on local outlier factor and a reconstruction method. Based on the model of line target in the shallow water, the paper presents an echo acquisition method with impulse response convolution and ray tracing, and considering the reverberation. Then, a transform matrix is presented for the coordinate transformation to obtain the point cloud. After that, the paper presents the global threshold based on local outlier factor (LOF) method to solving the problem of noise sensitivity in the global thresholding. Finally, the paper presents a novel reconstruction method, including the coarse step via random sample consensus (RANSC) and the refined step via Power crust. Both simulation and experimental results show that the scheme in this paper produced superior result compared with the state-of the art.

A fundamental matrix estimation based on matching points is a critical problem in epipolar geometry. In this paper, a global fundamental matrix estimation method based on inlier updating is proposed. Firstly, the coplanar constraint was incorporated into the solution of the fundamental matrix to reduce the number of parameters to be solved. Subsequently, an inlier updating matrix was introduced according to the threshold of the epipolar geometry distance to eliminate the potential outliers and obtain a reliable initial value of the fundamental matrix. On this basis, we employed a four-point iterative method to estimate the fundamental matrix and make it satisfy the rank constraint at the same time. Finally, the epipolar geometry in binocular vision was extended to triple-view, and the fundamental matrix obtained in the previous step was globally optimized by minimizing the coordinate deviation between the intersection point and feature point in each group of images. The experiments show that the proposed fundamental matrix estimation method is robust to noise and outliers. In the attitude measurement, the maximum static error was 0.104° and dynamic measurement error was superior to 0.273°, which improved the reconstruction accuracy of feature points. Indoor images were further used to test the method, and the mean rotation angle error was 0.362°. The results demonstrate that the estimation method proposed in this paper has a good practical application prospect in multi-view 3D reconstruction and visual localization.

This paper introduces a real-time 3D reconstruction of a human captured using a depth sensor and has integrated it with a holographic telepresence application. Holographic projection is widely recognized as one of the most promising 3D display technologies, and it is expected to become more widely available in the near future. This technology may also be deployed in various ways, including holographic prisms and Z-Hologram, which this research has used to demonstrate the initial results by displaying the reconstructed 3D representation of the user. The realization of a stable and inexpensive 3D data acquisition system is a problem that has yet to be solved. When we involve multiple sensors we need to compress and optimize the data so that it can be sent to a server for a telepresence. Therefore the paper presents the processes in real-time 3D reconstruction, which consists of data acquisition, background removal, point cloud extraction, and a surface generation which applies a marching cube algorithm to finally form an isosurface from the set of points in the point cloud which later texture mapping is applied on the isosurface generated. The compression results has been presented in this paper, and the results of the integration process after sending the data over the network also have been discussed.

Nuclear industry in Russia plays an important role in total power generation. At the same time, it is considered to be dangerous in terms of high potential risk in a case of any failure occurrence. Therefore, constant monitoring and quality control is essential on every stage of energy production process, as well as maintenance of the technical quality of the exploited components. For that reason, specified regulatory documents are developed. They provide quality requirements for each component type and regulate inspection procedures. In this paper, welded joints were considered as the controlled object. It is represented that standard quality control methods based on the manual visual inspection are not accurate enough. Therefore, this paper suggests an advanced method of automated optical scanning for misalignment evaluation of welded parts based on structural light technique. Precision improvement was achieved by implementation of a robotic manipulator, which led to the development of the specific calibration technique. Considering that there are no established methodologies for such method the validation experiments were performed. The ability to detect the minimum displacement in accordance with nuclear industry regulatory documents was studied. The results demonstrated that misalignment of 0.47 mm can be measured, and it proves that proposed method can be further implemented for a practical application in nuclear industry

Dietary calorie management has been an important topic in recent years, and various methods and applications on image-based food calorie estimation have been published in the multimedia community. Most of the existing methods of estimating food calorie amounts use 2D-based image recognition. On the other hand, in this paper, we would like to make inferences based on 3D volume for more accurate estimation. We performed 3D reconstruction of a dish (food and plate) and a plate (without foods), from a single image. We succeeded in restoring the 3D shape with high accuracy while maintaining the consistency between a plate part of an estimated 3D dish and an estimated 3D plate. To achieve this, the following contributions were made in this paper. (1) Proposal of "Hungry Networks," a new network that generates two kinds of 3D volumes from a single image. (2) Introduction of plate consistency loss that matches the shapes of the plate parts of the two reconstructed models. (3) Creating a new dataset of 3D food models that are 3D scanned of actual foods and plates. We also conducted an experiment to infer the volume of only the food region from the difference of the two reconstructed volumes. As a result, it was shown that the introduced new loss function not only matches the 3D shape of the plate, but also contributes to obtaining the volume with higher accuracy. Although there are some existing studies that consider 3D shapes of foods, this is the first study to generate a 3D mesh volume from a single dish image.

BackgroundBioimaging techniques offer a robust tool for studying molecular pathways and morphological phenotypes of cell populations subjected to various conditions. As modern high resolution 3D microscopy provides access to an ever-increasing amount of high quality images, there arises a need for their analysis in an automated, unbiased and simple way.Segmentation of structures within cell nucleus, which is the focus of this paper, presents a new layer of complexity in the form of dense packing and significant signal overlap.At the same time the available segmentation tools provide a steep learning curve for new users with limited technical background. This is especially apparent in bulk processing of image sets, which requires the use of some form of programming notation.ResultsIn this paper, we present PartSeg, a tool for segmentation and reconstruction of 3D microscopy images, optimised for the study of cell nucleus. PartSeg integrates refined versions of several state-of-the-art algorithms, including a new multi-scale approach for segmentation and quantitative analysis of 3D microscopy images.The features and user-friendly interface of PartSeg were carefully planned with biologists in mind, based on analysis of multiple use cases and difficulties encountered with other tools, to offer ergonomic interface with a minimal entry barrier. Bulk processing in an ad-hoc manner is possible without the need for programmer support. As the size of datasets of interest grows, such bulk processing solutions become essential for proper statistical analysis of results.Advanced users can use PartSeg components as a library within Python data processing and visualisation pipelines, for example within Jupyter notebooks. The tool is extensible so that new functionality and algorithms can be added by the use of plugins.For biologists the utility of PartSeg is presented in several scenarios, showing the quantitative analysis of nuclear structures.ConclusionsIn this paper, we have presented PartSeg which is a tool for precise and verifiable segmentation and reconstruction of 3D microscopy images. PartSeg is optimised for cell nucleus analysis and offers multiscale segmentation algorithms best-suited for this task. PartSeg can also be used for bulk processing of multiple images and its components can be reused in other systems or computational experiments.Contactg.bokota@cent.uw.edu.pl, a.magalska@nencki.edu.pl, d.plewczynski@cent.uw.edu.pl

Significance Three-dimensional refinement is a critical component of cryo-EM single-particle reconstruction. In this paper, we report the development of a computational method, OPUS-SSRI, and its application to seven real cryo-EM datasets. Our data clearly demonstrated that OPUS-SSRI can improve the final resolutions and structural details in cryo-EM single-particle analysis., In this paper, we present a refinement method for cryo-electron microscopy (cryo-EM) single-particle reconstruction, termed as OPUS-SSRI (Sparseness and Smoothness Regularized Imaging). In OPUS-SSRI, spatially varying sparseness and smoothness priors are incorporated to improve the regularity of electron density map, and a type of real space penalty function is designed. Moreover, we define the back-projection step as a local kernel regression and propose a first-order method to solve the resulting optimization problem. On the seven cryo-EM datasets that we tested, the average improvement in resolution by OPUS-SSRI over that from RELION 3.0, the commonly used image-processing software for single-particle cryo-EM, was 0.64 Å, with the largest improvement being 1.25 Å. We expect OPUS-SSRI to be an invaluable tool to the broad field of cryo-EM single-particle analysis. The implementation of OPUS-SSRI can be found at https://github.com/alncat/cryoem.

Since diverse and complex emotions need to be expressed by different facial deformation and appearances, facial animation has become a serious and on-going challenge for computer animation industry. Face reconstruction techniques based on 3D morphable face model and deep learning provide one effective solution to reuse existing databases and create believable animation of new characters from images or videos in seconds, which greatly reduce heavy manual operations and a lot of time. In this paper, we review the databases and state-of-the-art methods of 3D face reconstruction from a single RGB image. First, we classify 3D reconstruction methods into three categories and review each of them. These three categories are: Shape-from-Shading (SFS), 3D Morphable Face Model (3DMM), and Deep Learning (DL) based 3D face reconstruction. Next, we introduce existing 2D and 3D facial databases. After that, we review 10 methods of deep learning-based 3D face reconstruction and evaluate four representative ones among them. Finally, we draw conclusions of this paper and discuss future research directions.

Mixed Reality (MR) is a technology which enable to bring a virtual element into the real-world environment. MR intends to improve reality on the virtual world immerse onto real-world space. Occasionally the MR has been improved as the display technologies advanced progressively. In MR collaborative interface context, the local and remote user work together on collaborative task while sense the immersive environment in the cooperative application. User telepresence is an immersive telepresence, where the reconstruction of a human appears in a real-life. Up till now, producing full telepresence of the life-size human body may require a high transmission bandwidth of the internet. Therefore, this paper explores on a robust real-time 3D reconstruction method for MR telepresence. This paper discusses the previous works on the reconstruction method of a full-body human and the existing research works that have proposed the reconstruction methods for telepresence. Besides the 3D reconstruction method, this paper also enlightens our recent finding on the MR framework to transport a full-body human from a local location to a remote location. The MR telepresence will be discussed, as well as the robust 3D reconstruction method which has been implemented with user telepresence feature where the user experiences an accurate 3D representation of a remote person. The paper ends with the discussion and results, MR telepresence with robust 3D reconstruction method to execute user telepresence.

Scene flow is the dense 3D reconstruction of motion and geometry of a scene. Most state-of-the-art methods use a pair of stereo images as input for full scene reconstruction. These methods depend a lot on the quality of the RGB images and perform poorly in regions with reflective objects, shadows, ill-conditioned light environment and so on. LiDAR measurements are much less sensitive to the aforementioned conditions but LiDAR features are in general unsuitable for matching tasks due to their sparse nature. Hence, using both LiDAR and RGB can potentially overcome the individual disadvantages of each sensor by mutual improvement and yield robust features which can improve the matching process. In this paper, we present DeepLiDARFlow, a novel deep learning architecture which fuses high level RGB and LiDAR features at multiple scales in a monocular setup to predict dense scene flow. Its performance is much better in the critical regions where image-only and LiDAR-only methods are inaccurate. We verify our DeepLiDARFlow using the established data sets KITTI and FlyingThings3D and we show strong robustness compared to several state-of-the-art methods which used other input modalities. The code of our paper is available at https://github.com/dfki-av/DeepLiDARFlow., This paper is accepted to IROS2020

Refereed/Peer-reviewed Estimating depth from RGB images is a long-standing ill-posed problem, which has been explored for decades by the computer vision, graphics, and machine learning communities. Among the existing techniques, stereo matching remains one of the most widely used in the literature due to its strong connection to the human binocular system. Traditionally, stereo-based depth estimation has been addressed through matching hand-crafted features across multiple images. Despite the extensive amount of research, these traditional techniques still suffer in the presence of highly textured areas, large uniform regions, and occlusions. Motivated by their growing success in solving various 2D and 3D vision problems, deep learning for stereo-based depth estimation has attracted a growing interest from the community, with more than 150 papers published in this area between 2014 and 2019. This new generation of methods has demonstrated a significant leap in performance, enabling applications such as autonomous driving and augmented reality. In this paper, we provide a comprehensive survey of this new and continuously growing field of research, summarize the most commonly used pipelines, and discuss their benefits and limitations. In retrospect of what has been achieved so far, we also conjecture what the future may hold for deep learning-based stereo for depth estimation research.

Knowing the amounts of energy and nutrients in an individual’s diet is important for maintaining health and preventing chronic diseases. As electronic and AI technologies advance rapidly, dietary assessment can now be performed using food images obtained from a smartphone or a wearable device. One of the challenges in this approach is to computationally measure the volume of food in a bowl from an image. This problem has not been studied systematically despite the bowl being the most utilized food container in many parts of the world, especially in Asia and Africa. In this paper, we present a new method to measure the size and shape of a bowl by adhering a paper ruler centrally across the bottom and sides of the bowl and then taking an image. When observed from the image, the distortions in the width of the paper ruler and the spacings between ruler markers completely encode the size and shape of the bowl. A computational algorithm is developed to reconstruct the three-dimensional bowl interior using the observed distortions. Our experiments using nine bowls, colored liquids, and amorphous foods demonstrate high accuracy of our method for food volume estimation involving round bowls as containers. A total of 228 images of amorphous foods were also used in a comparative experiment between our algorithm and an independent human estimator. The results showed that our algorithm overperformed the human estimator who utilized different types of reference information and two estimation methods, including direct volume estimation and indirect estimation through the fullness of the bowl.

Objective: This paper aims to propose a 3D laparoscopic imaging system that can realize dense 3D reconstruction in real time. Methods: Based on the active stereo technique which yields high-density, accurate and robust 3D reconstruction by combining structured light and stereo vision, we design a laparoscopic system consisting of two image feedback channels and one pattern projection channel. Remote high-speed image acquisition and pattern generation lay the foundation for the real-time dense 3D surface reconstruction and enable the miniaturization of the laparoscopic probe. To enhance the reconstruction efficiency and accuracy, we propose a novel active stereo method by which the dense 3D point cloud is obtained using only five patterns, while most existing multiple-shot structured light techniques require $\text{10--40}$ patterns. In our method, dual-frequency phase-shifting fringes are utilized to uniquely encode the pixels of the measured targets, and a dual-codeword matching scheme is developed to simplify the matching procedure and achieve high-precision reconstruction. Results: Compared with the existing structured light techniques, the proposed method shows better real-time efficiency and accuracy in both quantitative and qualitative ways. Ex-vivo experiments demonstrate the robustness of the proposed method to different biological organs and the effectiveness to lesions and deformations of the organs. Feasibility of the proposed system for real-time dense 3D reconstruction is verified in dynamic experiments. According to the experimental results, the system acquires 3D point clouds with a speed of 12 frames per second. Each frame contains more than 40,000 points, and the average errors tested on standard objects are less than 0.2 mm. Significance: This paper provides a new real-time dense 3D reconstruction method for 3D laparoscopic imaging. The established prototype system has shown good performance in reconstructing surface of biological tissues.