Search

We propose WS-DeepSfT, a novel deep learning-based approach to the Shape-from-Template (SfT) problem, which aims at reconstructing the 3D shape of a deformable object from a single RGB image and a template. WS-DeepSfT addresses the limitations of existing SfT techniques by combining a weakly-supervised deep neural network (DNN) for registration and a classical As-Rigid-As-Possible (ARAP) algorithm for 3D reconstruction. Unlike previous deep learning-based SfT methods, which require extensive synthetic data and depth sensors for training, WS-DeepSfT only requires regular RGB video of the deforming object and a segmentation mask to discriminate the object from the background. The registration model is trained without synthetic data, using videos where the object undergoes deformations, while ARAP does not require training and infers the 3D shape in real-time with minimal overhead. We show that WS-DeepSfT outperforms the state-of-the-art, in both accuracy and robustness, without requiring depth sensors or synthetic data generation. WS-DeepSfT thus offers a robust, efficient, and scalable approach to SfT, bringing it closer to applications such as augmented reality.

Shape-from-Template (SfT) solves the registration and 3D reconstruction of a deformable 3D object, represented by the template, from a single image. Recently, methods based on deep learning have been able to solve SfT for the wide-baseline case in real-time, clearly surpassing classical methods. However, the main limitation of current methods is the need for fine tuning of the neural models to a specific geometry and appearance represented by the template texture map. We propose the first texture-generic deep learning SfT method which adapts to new texture maps at run-time, without the need for texture specific fine tuning. We achieve this by dividing the problem into a segmentation step and a registration and reconstruction step, both solved with deep learning. We include the template texture map as one of the neural inputs in both steps, training our models to adapt to different ones. We show that our method obtains comparable or better results to previous deep learning models, which are texture specific. It works in challenging imaging conditions, including complex deformations, occlusions, motion blur and poor textures. Our implementation runs in real-time, with a low-cost GPU and CPU.

Dense motion field estimation is a key computer vision problem. Many solutions have been proposed to compute small or large displacements, narrow or wide baseline stereo disparity, or non‐rigid surface registration, but a unified methodology is still lacking. The authors introduce a general framework that robustly combines direct and feature‐based matching. The feature‐based cost is built around a novel robust distance function that handles keypoints and weak features such as segments. It allows us to use putative feature matches to guide dense motion estimation out of local minima. The authors’ framework uses a robust direct data term. It is implemented with a powerful second‐order regularisation with external and self‐occlusion reasoning. Their framework achieves state‐of‐the‐art performance in several cases (standard optical flow benchmarks, wide‐baseline stereo and non‐rigid surface registration). Their framework has a modular design that customises to specific application needs.

Background: Optical Coherence Tomography (OCT) is an emerging medical imaging technology. It is well suited to various medical applications requiring tissue imaging with micrometer resolution and millimeter penetration depth such as in ophthalmology and dermatology. Despite its numerous advantages, OCT has a long acquisition time for high-resolution images or volumes. This paper deals with the development of a Compressed, Sensing (CS) paradigm for faster 3-dimensional OCT image acquisition. Methods: The proposed framework includes three main steps: 1) defining a random-like and parameterizable and continuous scanning trajectories that must be compatible with a smooth mechanical scan, 2) rasterizing the scanning trajectory to make it achievable by a physical system (i.e. galvanometer mirrors), and 3) incorporating a high sparsifying data technique so-called 3D shearlet transform into the compressed sensing scheme. Actually, shearlet transform is mathematically optimal for multidimensional data decomposition and has been proven more efficient than classical ones such as those obtained by wavelet or curvelet transforms. Actually, shearlet system provides a very efficient tool for encoding anisotropic features (such as edges in images) in multivariate problem classes. Results: Numerical simulations and ex vivo experiments were carried out. The obtained results showed the ability of the proposed method to recover OCT images and volumes with high fidelity for different subsampling rates and scanning schemes, demonstrating the relevance of the proposed approach.

Time‐of‐flight (TOF) sensors provide real‐time depth information at high frame‐rates. One issue with TOF sensors is the usual high level of noise (i.e. the depth measure's repeatability within a static setting). However, until now, TOF sensors’ noise has not been well studied. The authors show that the commonly agreed hypothesis that noise depends only on the amplitude information is not valid in practice. They empirically establish that the noise follows a signal‐dependent Gaussian distribution and varies according to pixel position, depth and integration time. They thus consider all these factors to model noise in two new noise models. Both models are evaluated, compared and used in the two following applications: depth noise removal by depth filtering and uncertainty (repeatability) estimation in three‐dimensional measurement.