Search

Non-Rigid Structure-from-Motion (NRSfM) reconstructs a deformable 3D object from the correspondences established between monocular 2D images. Current NRSfM methods lack statistical robustness, which is the ability to cope with correspondence errors.This prevents one to use automatically established correspondences, which are prone to errors, thereby strongly limiting the scope of NRSfM. We propose a three-step automatic pipeline to solve NRSfM robustly by exploiting isometry. Step 1 computes the optical flow from correspondences, step 2 reconstructs each 3D point's normal vector using multiple reference images and integrates them to form surfaces with the best reference and step 3 rejects the 3D points that break isometry in their local neighborhood. Importantly, each step is designed to discard or flag erroneous correspondences. Our contributions include the robustification of optical flow by warp estimation, new fast analytic solutions to local normal reconstruction and their robustification, and a new scale-independent measure of 3D local isometric coherence. Experimental results show that our robust NRSfM method consistently outperforms existing methods on both synthetic and real datasets., Accepted in TPAMI 2021

We give an effective solution to the regularized optimization problem $g (\boldsymbol{x}) + h (\boldsymbol{x})$, where $\boldsymbol{x}$ is constrained on the unit sphere $\Vert \boldsymbol{x} \Vert_2 = 1$. Here $g (\cdot)$ is a smooth cost with Lipschitz continuous gradient within the unit ball $\{\boldsymbol{x} : \Vert \boldsymbol{x} \Vert_2 \le 1 \}$ whereas $h (\cdot)$ is typically non-smooth but convex and absolutely homogeneous, \textit{e.g.,}~norm regularizers and their combinations. Our solution is based on the Riemannian proximal gradient, using an idea we call \textit{proxy step-size} -- a scalar variable which we prove is monotone with respect to the actual step-size within an interval. The proxy step-size exists ubiquitously for convex and absolutely homogeneous $h(\cdot)$, and decides the actual step-size and the tangent update in closed-form, thus the complete proximal gradient iteration. Based on these insights, we design a Riemannian proximal gradient method using the proxy step-size. We prove that our method converges to a critical point, guided by a line-search technique based on the $g(\cdot)$ cost only. The proposed method can be implemented in a couple of lines of code. We show its usefulness by applying nuclear norm, $\ell_1$ norm, and nuclear-spectral norm regularization to three classical computer vision problems. The improvements are consistent and backed by numerical experiments., 19 pages

International audience; Purpose: We propose to learn a 3D keypoint descriptor which we use to match keypoints extracted from full-body CT scans. Our methods are inspired by 2D keypoint descriptor learning, which was shown to outperform hand-crafted descriptors. Adapting these to 3D images is challenging because of the lack of labelled training data and high memory requirements.Method: We generate semi-synthetic training data. For that, we first estimate the distribution of local affine inter-subject transformations using labelled anatomical landmarks on a small subset of the database. We then sample a large number of transformations and warp unlabelled CT scans, for which we can subsequently establish reliable keypoint correspondences using guided matching. These correspondences serve as training data for our descriptor, which we represent by a CNN and train using the triplet loss with online triplet mining.Results: We carry out experiments on a synthetic data reliability benchmark and a registration task involving 20 CT volumes with anatomical landmarks used for evaluation purposes. Our learned descriptor outperforms the 3D-SURF descriptor on both benchmarks while having a similar runtime.Conclusion: We propose a new method to generate semi-synthetic data and a new learned 3D keypoint descriptor. Experiments show improvement compared to a hand-crafted descriptor. This is promising as literature has shown that jointly learning a detector and a descriptor gives further performance boost.

Registration maps or warps form a key element in Shape-from-Template (SfT). They relate the template with the input image, which contains the projection of the deformed surface. Recently, it was shown that isometric SfT can be solved analytically if the warp and its first-order derivatives are known. In practice, the warp is recovered by interpolating a set of discrete template-to-image point correspondences. This process relies on smoothness priors but ignores the 3D geometry. This may produce errors in the warp and poor reconstructions. In contrast, we propose to create a 3D consistent warp, which technically is a very challenging task, as the 3D shape variables must be eliminated from the isometric SfT equations to find differential constraints for the warp only. Integrating these constraints in warp estimation yields the isowarp, a warp 3D consistent with isometric SfT. Experimental results show that incorporating the isowarp in the SfT pipeline allows the analytic solution to outperform non-convex 3D shape refinement methods and the recent DNN-based SfT methods. The isowarp can be properly initialized with convex methods and its hyperparameters can be automatically obtained with cross-validation. The isowarp is resistant to 3D ambiguities and less computationally expensive than existing 3D shape refinement methods. The isowarp is thus a theoretical and practical breakthrough in SfT.

3D reconstruction of deformable objects using inter-image visual motion from monocular images has been studied under Shape-from-Template (SfT) and Non-Rigid Structure-from-Motion (NRSfM). Most methods have been developed for simple deformation models, primarily isometry. They may treat a surface as a discrete set of points and draw constraints from the points only or they may use a non-parametric representation and use both points and differentials to express constraints. We propose a differential framework based on Cartan's theory of connections and moving frames. It is applicable to SfT and NRSfM, and to deformation models other than isometry. It utilises infinitesimal-level assumptions on the surface's geometry and mappings. It has the following properties. 1) It allows one to derive existing solutions in a simpler way. 2) It models SfT and NRSfM in a unified way. 3) It allows us to introduce a new skewless deformation model and solve SfT and NRSfM for it. 4) It facilitates a generic solution to SfT which does not require deformation modeling. Our framework is complete: it solves deformable 3D reconstruction for a whole class of algebraic deformation models including isometry. We compared our solutions with the state-of-the-art methods and show that ours outperform in terms of both accuracy and computation time.

International audience; Purpose. The registration of a preoperative 3D model, reconstructed for example from MRI, to intraoperative laparoscopy 2D images, is the main challenge to achieve augmented reality in laparoscopy. The current systems have a major limitation: they require that the surgeon manually marks the occluding contours during surgery. This requires the surgeon to fully comprehend the non-trivial concept of occluding contours and surgeon time, directly impacting acceptance and usability. To overcome this limitation, we propose a complete framework for object-class occluding contour detection (OC2D), with application to uterus surgery. Methods. Our first contribution is a new distance-based evaluation score complying with all the relevant performance criteria. Our second contribution is a loss function combining cross-entropy and two new penalties designed to boost 1-pixel thickness responses. This allows us to train a U-Net end-to-end, outperforming all competing methods, which tends to produce thick responses. Our third contribution is a dataset of 3818 carefully labelled laparoscopy images of the uterus, which was used to train and evaluate our detector. Results. Evaluation shows that the proposed detector has a similar false negative rate to existing methods but substantially reduces both false positive rate and response thickness. Finally, we ran a user-study to evaluate the impact of OC2D against manually marked occluding contours in augmented laparoscopy. We used 10 recorded gynecologic laparoscopies and involved 5 surgeons. Using OC2D led to a reduction of 3 minutes and 53 seconds in surgeon time without sacrificing registration accuracy. Conclusions. We provide a new set of criteria and a distance-based measure to evaluate an OC2D method. We propose an OC2D method which outperforms the state of the art methods. The results obtained from the user study indicate that fully automatic augmented laparoscopy is feasible. 2 T. François et al. Registration Preoperative 3D model with internal structures Occluding contours deformation of the 3D model Occluding contours Occlusion boundaries Uterus Connection contours Fig. 1 (left) In augmented laparoscopy, the preoperative 3D model is registered by fitting the occluding contours of the organ in laparoscopy images. The current systems require the surgeon to mark these contours manually during surgery. (right) An occluding contour arises at an organ boundary where the organ occludes another structure, as opposed to an occlusion boundary where the organ is occluded by another structure. The set of occluding contours is the silhouette. OC2D is the task of detecting the occluding contours for a specific object, here the uterus. It forms a task of semantic detection far more challenging than organ segmentation.

Generalized Procrustes Analysis (GPA) is the problem of bringing multiple shapes into a common reference by estimating transformations. GPA has been extensively studied for the Euclidean and affine transformations. We introduce GPA with deformable transformations, which forms a much wider and difficult problem. We specifically study a class of transformations called the Linear Basis Warps (LBWs), which contains the affine transformation and most of the usual deformation models, such as the Thin-Plate Spline (TPS). GPA with deformations is a nonconvex underconstrained problem. We resolve the fundamental ambiguities of deformable GPA using two shape constraints requiring the eigenvalues of the shape covariance. These eigenvalues can be computed independently as a prior or posterior. We give a closed-form and optimal solution to deformable GPA based on an eigenvalue decomposition. This solution handles regularization, favoring smooth deformation fields. It requires the transformation model to satisfy a fundamental property of free-translations, which asserts that the model can implement any translation. We show that this property fortunately holds true for most common transformation models, including the affine and TPS models. For the other models, we give another closed-form solution to GPA, which agrees exactly with the first solution for models with free-translation. We give pseudo-code for computing our solution, leading to the proposed DefGPA method, which is fast, globally optimal and widely applicable. We validate our method and compare it to previous work on six diverse 2D and 3D datasets, with special care taken to choose the hyperparameters from cross-validation., Published on International journal of computer vision (IJCV) 2022

Shape-from-Template (SfT) is the problem of using a shape template to infer the shape of a deformable object observed in an image. The usual case of SfT is ‘Surface’ SfT, where the shape is a 2D surface embedded in 3D, and the image is a 2D perspective projection. We introduce ‘Curve’ SfT, comprising two new cases of SfT where the shape is a 1D curve. The first new case is when the curve is embedded in 2D and the image a 1D perspective projection. The second new case is when the curve is embedded in 3D and the image a 2D perspective projection. We present a thorough theoretical study of these new cases for isometric deformations, which are a good approximation of ropes, cables and wires. Unlike Surface SfT, we show that Curve SfT is only ever solvable up to discrete ambiguities. We present the necessary and sufficient conditions for solvability with critical point analysis. We further show that unlike Surface SfT, Curve SfT cannot be solved locally using exact non-holonomic Partial Differential Equations. Our main technical contributions are two-fold. First, we give a stable, global reconstruction method that models the problem as a discrete Hidden Markov Model. This can generate all candidate solutions. Second, we give a non-convex refinement method using a novel angle-based deformation parameterization. We present quantitative and qualitative results showing that real curve shaped objects such as a necklace can be successfully reconstructed with Curve SfT.

International audience

International audience; A major research area in Computer Assisted Intervention (CAI) is to aid laparoscopic surgery teams with Augmented Reality (AR) guidance. This involves registering data from other modalities such as MR and fusing it with the laparoscopic video in real-time, to reveal the location of hidden critical structures. We present the first system for AR guided laparoscopic surgery of the uterus. This works with pre-operative MR or CT data and monocular laparoscopes, without requiring any additional interventional hardware such as optical trackers. We present novel and robust solutions to two main sub-problems: the initial registration, which is solved using a short exploratory video, and update registration, which is solved with real-time tracking-by-detection. These problems are challenging for the uterus because it is a weakly-textured, highly mobile organ that moves independently of surrounding structures. In the broader context, our system is the first that has successfully performed markerless real-time registration and AR of a mobile human organ with monocular laparoscopes in the OR.

The vast majority of mesh-based modelling applications iteratively transform the mesh vertices under prescribed geometric conditions. This occurs in particular in methods cycling through the constraint set such as Position-Based Dynamics (PBD). A common case is the approximate local area preservation of triangular 2D meshes under external editing constraints. At the constraint level, this yields the nonconvex optimal triangle projection under prescribed area problem, for which there does not currently exist a direct solution method. In current PBD implementations, the area preservation constraint is linearised. The solution comes out through the iterations, without a guarantee of optimality, and the process may fail for degenerate inputs where the vertices are colinear or colocated. We propose a closed-form solution method and its numerically robust algebraic implementation. Our method handles degenerate inputs through a two-case analysis of the problem’s generic ambiguities. We show in a series of experiments in area-based 2D mesh editing that using optimal projection in place of area constraint linearisation in PBD speeds up and stabilises convergence.

International audience; We address the monocular visual shape servoing problem. This pushes the challenging visual servoing problem one step further from rigid object manipulation towards deformable object manipulation. Explicitly, it implies deforming the object towards a desired shape in 3D space by robots using monocular 2D vision. We specifically concentrate on a scheme capable of controlling large isometric deformations. Two important open subproblems arise for implementing such a scheme. (P1) Since it is concerned with large deformations, perception requires tracking the deformable object's 3D shape from monocular 2D images which is a severely underconstrained problem. (P2) Since rigid robots have fewer degrees of freedom than a deformable object, the shape control becomes underactuated. We propose a template-based shape servoing scheme in which we solve these two problems. The template allows us to both infer the object's shape using an improved Shape-from-Template algorithm and steer the object's deformation by means of the robots' movements. We validate the scheme via simulations and real experiments.

We propose an original approach to absolute pose and structure-from-motion (SfM) which handles rolling shutter (RS) effects. Unlike most existing methods which either augment global shutter projection with velocity parameters or impose continuous time and motion through pose interpolation, we use local differential constraints. These are established by drawing analogies with non-rigid 3D vision techniques, namely shape-from-template and non-rigid SfM (NRSfM). The proposed idea is to interpret the images of a rigid surface acquired by a moving RS camera as those of a virtually deformed surface taken by a GS camera. These virtually deformed surfaces are first recovered by relaxing the RS constraint using SfT or NRSfM. Then we upgrade the virtually deformed surface to the actual rigid structure and compute the camera pose and ego-motion by reintroducing the RS constraint. This uses a new 3D-3D registration procedure that minimizes a cost function based on the Euclidean 3D point distance. This is more stable and physically meaningful than the reprojection error or the algebraic distance used in previous work. Experimental results obtained with synthetic and real data show that the proposed methods outperform existing ones in terms of accuracy and stability, even in the known critical configurations.

Background and objective:Surgical tool detection, segmentation, and 3D pose estimation are crucial components in Computer-Assisted Laparoscopy (CAL). The existing frameworks have two main limitations. First, they do not integrate all three components. Integration is critical; for instance, one should not attempt computing pose if detection is negative. Second, they have highly specific requirements, such as the availability of a CAD model. We propose an integrated and generic framework whose sole requirement for the 3D pose is that the tool shaft is cylindrical. Our framework makes the most of deep learning and geometric 3D vision by combining a proposed Convolutional Neural Network (CNN) with algebraic geometry. We show two applications of our framework in CAL: tool-aware rendering in Augmented Reality (AR) and tool-based 3D measurement. Methods:We name our CNN as ART-Net (Augmented Reality Tool Network). It has a Single Input Multiple Output (SIMO) architecture with one encoder and multiple decoders to achieve detection, segmentation, and geometric primitive extraction. These primitives are the tool edge-lines, mid-line, and tip. They allow the tool’s 3D pose to be estimated by a fast algebraic procedure. The framework only proceeds if a tool is detected. The accuracy of segmentation and geometric primitive extraction is boosted by a new Full resolution feature map Generator (FrG). We extensively evaluate the proposed framework with the EndoVis and new proposed datasets. We compare the segmentation results against several variants of the Fully Convolutional Network (FCN) and U-Net. Several ablation studies are provided for detection, segmentation, and geometric primitive extraction. The proposed datasets are surgery videos of different patients. Results:In detection, ART-Net achieves 100.0 % in both average precision and accuracy. In segmentation, it achieves 81.0 % in mean Intersection over Union (mIoU) on the robotic EndoVis dataset (articulated tool), where it outperforms both FCN and U-Net, by 4.5 p p and 2.9 p p , respectively. It achieves 88.2 % in mIoU on the remaining datasets (non-articulated tool). In geometric primitive extraction, ART-Net achieves 2.45 ∘ and 2.23 ∘ in mean Arc Length (mAL) error for the edge-lines and mid-line, respectively, and 9.3 pixels in mean Euclidean distance error for the tool-tip. Finally, in terms of 3D pose evaluated on animal data, our framework achieves 1.87 mm, 0.70 mm, and 4.80 mm mean absolute errors on the X , Y , and Z coordinates, respectively, and 5 . 94 ∘ angular error on the shaft orientation. It achieves 2.59 mm and 1.99 mm in mean and median location error of the tool head evaluated on patient data. Conclusions:The proposed framework outperforms existing ones in detection and segmentation. Compared to separate networks, integrating the tasks in a single network preserves accuracy in detection and segmentation but substantially improves accuracy in geometric primitive extraction. Overall, our framework has similar or better accuracy in 3D pose estimation while largely improving robustness against the very challenging imaging conditions of laparoscopy. The source code of our framework and our annotated dataset will be made publicly available at https://github.com/kamruleee51/ART-Net .

In the recent past, algorithms based on Convolutional Neural Networks (CNNs) have achieved significant milestones in object recognition. With large examples of each object class, standard datasets train well for inter-class variability. However, gathering sufficient data to train for a particular instance of an object within a class is impractical. Furthermore, quantitatively assessing the imaging conditions for each image in a given dataset is not feasible. By generating sufficient images with known imaging conditions, we study to what extent CNNs can cope with hard imaging conditions for instance-level recognition in an active learning regime. Leveraging powerful rendering techniques to achieve instance-level detection, we present results of training three state-of-the-art object detection algorithms namely, Fast R-CNN, Faster R-CNN and YOLO9000, for hard imaging conditions imposed into the scene by rendering. Our extensive experiments produce a mean Average Precision score of 0.92 on synthetic images and 0.83 on real images using the best performing Faster R-CNN. We show for the first time how well detection algorithms based on deep architectures fare for each hard imaging condition studied.

International audience; "Background: Optical Coherence Tomography (OCT) is an emerging medical imaging technology. It is well suited to various medical applications requiring tissue imaging with micrometer resolution and millimeter penetration depth such as in ophthalmology and dermatology. Despite its numerous advantages, OCT has a long acquisition time for high-resolution images or volumes. This paper deals with the development of a Compressed, Sensing (CS) paradigm for faster 3-dimensional OCT image acquisition.Methods: The proposed framework includes three main steps: 1) defining a random-like and parameterizable and continuous scanning trajectories that must be compatible with a smooth mechanical scan, 2) rasterizing the scanning trajectory to make it achievable by a physical system (i.e. galvanometer mirrors), and 3) incorporating a high sparsifying data technique so-called 3D shearlet transform into the compressed sensing scheme. Actually, shearlet transform is mathematically optimal for multidimensional data decomposition and has been proven more efficient than classical ones such as those obtained by wavelet or curvelet transforms. Actually, shearlet system provides a very efficient tool for encoding anisotropic features (such as edges in images) in multivariate problem classes.Results: Numerical simulations and ex vivo experiments were carried out. The obtained results showed the ability of the proposed method to recover OCT images and volumes with high fidelity for different subsampling rates and scanning schemes, demonstrating the relevance of the proposed approach."

Augmented reality (AR) is a surgical guidance technology that allows key hidden subsurface structures to be visualized by endoscopic imaging. We report here 2 cases of patients with adenomyoma selected for the AR technique. The adenomyomas were localized using AR during laparoscopy. Three-dimensional models of the uterus, uterine cavity, and adenomyoma were constructed before surgery from T2-weighted magnetic resonance imaging, allowing an intraoperative 3-dimensional shape of the uterus to be obtained. These models were automatically aligned and “fused” with the laparoscopic video in real time, giving the uterus a semitransparent appearance and allowing the surgeon in real time to both locate the position of the adenomyoma and uterine cavity and rapidly decide how best to access the adenomyoma. In conclusion, the use of our AR system designed for gynecologic surgery leads to improvements in laparoscopic adenomyomectomy and surgical safety.

Augmented reality is a technology that allows a surgeon to see key hidden subsurface structures in an endoscopic video in real-time. This works by overlaying information obtained from preoperative imaging and fusing it in real-time with the endoscopic image. Magnetic resonance diffusion tensor imaging (DTI) and fiber tractography are known to provide additional information to that obtained from standard structural magnetic resonance imaging (MRI). Here, we report the first 2 cases of the use of real-time augmented reality during laparoscopic myomectomies with visualization of uterine muscle fibers after DTI tractography–MRI to help the surgeon decide the starting point incision. In the first case, a 31-year-old patient was undergoing laparoscopic surgery for a 6-cm FIGO type V myoma. In the second case, a 38-year-old patient was undergoing a laparoscopic myomectomy for a unique 6-cm FIGO type VI myoma. Signed consent forms were obtained from both patients, which included clauses of no modification of the surgery. Before surgery, MRI was performed. The external surface of the uterus, the uterine cavity, and the surface of the myomas were delimited on the basis of the findings of preoperative MRI. A fiber tracking algorithm was used to extrapolate the uterine muscle fibers’ architecture. The aligned models were blended with each video frame to give the impression that the uterus is almost transparent, enabling the surgeon to localize the myomas and uterine cavity exactly. The uterine muscle fibers were also displayed, and their visualization helped us decide the starting incision point for the myomectomies. Then, myomectomies were performed using a classic laparoscopic technique. These case reports show that augmented reality and DTI fiber tracking in a uterus with myomas are possible, providing fiber direction and helping the surgeon visualize and decide the starting incision point for laparoscopic myomectomy. Respecting the fibers’ orientation could improve the quality of the scar and decrease the architectural disorganization of the uterus.

Machine learning based methods achieves impressive results in object classification and detection. Utilizing representative data of the visual world during the training phase is crucial to achieve good performance with such data driven approaches. However, it not always possible to access bias-free datasets thus, robustness to biased data is a desirable property for a learning system. Capsule Networks have been introduced recently and their tolerance to biased data has received little attention. This paper aims to fill this gap and proposes two experimental scenarios to assess the tolerance to imbalanced training data and to determine the generalization performance of a model with unfamiliar affine transformations of the images. This paper assesses dynamic routing and EM routing based Capsule Networks and proposes a comparison with Convolutional Neural Networks in the two tested scenarios. The presented results provide new insights into the behaviour of capsule networks.

We propose a novel learning approach, in the form of a fully-convolutional neural network (CNN), which automatically and consistently removes specular highlights from a single image by generating its diffuse component. To train the generative network, we define an adversarial loss on a discriminative network as in the GAN framework and combined it with a content loss. In contrast to existing GAN approaches, we implemented the discriminator to be a multi-class classifier instead of a binary one, to find more constraining features. This helps the network pinpoint the diffuse manifold by providing two more gradient terms. We also rendered a synthetic dataset designed to help the network generalize well. We show that our model performs well across various synthetic and real images and outperforms the state-of-the-art in consistency.

Image warps -or just warps- capture the geometric deformation existing between two images of a deforming surface. The current approach to enforce a warp's smoothness is to penalize its second order partial derivatives (Bookstein in IEEE Trans Pattern Anal Mach Intell 11:567---585, 1989; Rueckert et al. in IEEE Trans Med Imaging 18:712---721, 1999). Because this favors locally affine warps, this fails to capture the local projective component of the image deformation. This may have a negative impact on applications such as image registration and deformable 3D reconstruction. We propose a novel penalty designed to smooth the warp while capturing the deformation's local projective structure. Our penalty is based on equivalents to the Schwarzian derivatives, which are projective differential invariants exactly preserved by homographies. We propose a methodology to derive a set of partial differential equations with only homographies as solutions. We call this system the Schwarzian equations and we explicitly derive them for 2D functions using differential properties of homographies. We name as Schwarp a warp which is estimated by penalizing the residual of Schwarzian equations. Experimental evaluation shows that Schwarps outperform existing warps in modeling and extrapolation power, and lead to better results in three deformable reconstruction methods, namely, shape reconstruction in shape-from-template, camera calibration in Shape-from-Template and Non-Rigid Structure-from-Motion.

We study a problem that we call Shape-from-Template , which is the problem of reconstructing the shape of a deformable surface from a single image and a 3D template. Current methods in the literature address the case of isometric deformations, and relax the isometry constraint to the convex inextensibility constraint, solved using the so-called maximum depth heuristic. We call these methods zeroth-order since they use image point locations (the zeroth-order differential structure) to solve the shape inference problem from a perspective image. We propose a novel class of methods that we call first-order. The key idea is to use both image point locations and their first-order differential structure. The latter can be easily extracted from a warp between the template and the input image. We give a unified problem formulation as a system of PDEs for isometric and conformal surfaces that we solve analytically . This has important consequences. First, it gives the first analytical algorithms to solve this type of reconstruction problems. Second, it gives the first algorithms to solve for the exact constraints . Third, it allows us to study the well-posedness of this type of reconstruction: we establish that isometric surfaces can be reconstructed unambiguously and that conformal surfaces can be reconstructed up to a few discrete ambiguities and a global scale. In the latter case, the candidate solution surfaces are obtained analytically. Experimental results on simulated and real data show that our isometric methods generally perform as well as or outperform state of the art approaches in terms of reconstruction accuracy, while our conformal methods largely outperform all isometric methods for extensible deformations.

A specific study of the orthographic, weak-perspective and paraperspective cameras.An algebraic procedure solving the affine correction problem for each camera model.A closed-form solution for each camera model.A full characterization of the affine corrections' generic ambiguities for each camera model.An experimental evaluation comparing the algebraic procedures to global polynomial optimization and an interior-point method. Given a general affine camera, we study the problem of finding the closest metric affine camera, where the latter is one of the orthographic, weak-perspective and paraperspective projection models. This problem typically arises in stratified Structure-from-Motion methods such as factorization-based methods. For each type of metric affine camera, we give a closed-form solution and its implementation through an algebraic procedure. Using our algebraic procedure, we can then provide a complete analysis of the problem's generic ambiguity space. This also gives the means to generate the other solutions if any.

Augmented reality for monocular laparoscopy from a preoperative volume such as CT is achieved in two steps. The first step is to segment the organ in the preoperative volume and reconstruct its 3D model. The second step is to register the preoperative 3D model to an initial intraoperative laparoscopy image. To date, there does not exist an automatic initial registration method to solve the second step for the liver in the de facto operating room conditions of monocular laparoscopy. Existing methods attempt to solve for both deformation and pose simultaneously, leading to nonconvex problems with no optimal solution algorithms. We propose in contrast to break the problem down into two parts, solving for (i) deformation and (ii) pose. Part (i) simulates biomechanical deformations from the preoperative to the intraoperative state to predict the liver’s unknown intraoperative shape by modeling gravity, the abdominopelvic cavity’s pressure and boundary conditions. Part (ii) rigidly registers the simulated shape to the laparoscopy image using contour cues. Our formulation leads to a well-posed problem, contrary to existing methods. This is because it exploits strong environment priors to complement the weak laparoscopic visual cues. Quantitative results with in silico and phantom experiments and qualitative results with laparosurgery images for two patients show that our method outperforms the state-of-the-art in accuracy and registration time.

We present a global and convex formulation for the template-less 3D reconstruction of a deforming object with the perspective camera. We show for the first time how to construct a Second-Order Cone Programming (SOCP) problem for Non-Rigid Structure-from-Motion (NRSfM) using the Maximum-Depth Heuristic (MDH). In this regard, we deviate strongly from the general trend of using affine cameras and factorization-based methods to solve NRSfM, which do not perform well with complex nonlinear deformations. In MDH, the points' depths are maximized so that the distance between neighbouring points in camera space are upper bounded by the geodesic distance. In NRSfM both geodesic and camera space distances are unknown. We show that, nonetheless, given point correspondences and the camera's intrinsics the whole problem can be solved with SOCP. This is the first convex formulation for NRSfM with physical constraints. We further present how robustness and temporal continuity can be included in the formulation to handle outliers and decrease the problem size, respectively. We show with extensive experiments that our methods accurately reconstruct quasi-isometric objects from partial views under articulated and strong deformations. Compared to the previous methods, our approach gives better or similar accuracy. It naturally handles missing correspondences, non-smooth objects and is very simple to implement compared to previous methods, with only one free parameter (the neighbourhood size).

We study Isometric Non-Rigid Shape-from-Motion (Iso-NRSfM): given multiple intrinsically calibrated monocular images, we want to reconstruct the time-varying 3D shape of a thin-shell object undergoing isometric deformations. We show that Iso-NRSfM is solvable from local warps, the inter-image geometric transformations. We propose a new theoretical framework based on the Riemmanian manifold to represent the unknown 3D surfaces as embeddings of the camera's retinal plane. This allows us to use the manifold's metric tensor and Christoffel Symbol (CS) fields. These are expressed in terms of the first and second order derivatives of the inverse-depth of the 3D surfaces, which are the unknowns for Iso-NRSfM. We prove that the metric tensor and the CS are related across images by simple rules depending only on the warps. This forms a set of important theoretical results. We show that current solvers cannot solve for the first and second order derivatives of the inverse-depth simultaneously. We thus propose an iterative solution in two steps. 1) We solve for the first order derivatives assuming that the second order derivatives are known. We initialise the second order derivatives to zero, which is an infinitesimal planarity assumption. We derive a system of two cubics in two variables for each image pair. The sum-of-squares of these polynomials is independent of the number of images and can be solved globally, forming a well-posed problem for $N\geq 3$ images. 2) We solve for the second order derivatives by initialising the first order derivatives from the previous step. We solve a linear system of $4N-4$ equations in three variables. We iterate until the first order derivatives converge. The solution for the first order derivatives gives the surfaces’ normal fields which we integrate to recover the 3D surfaces. The proposed method outperforms existing work in terms of accuracy and computation cost on synthetic and real datasets.

We introduce a 3D deformable model-based framework for the retrieval of near-isometric flattenable objects using keypoints and BoVW (Bag-of-Visual-Words). By 3D deformable model we mean a texturemapped 3D shape which may deform isometrically. We assume that such a model is available for each object in the database. We exploit the 3D deformable models at the training and the retrieval phases. For our first contribution, we exploit the possibility of generating synthetic data from the 3D deformable models to define a new BoVW model for the database object representation. Our model chooses an optimal per-object representation by maximizing each object’s mean average precision. The maximization is done over multiple candidate representations which are generated using the criteria of keypoint repeatability, weight discriminance and stability. Our second contribution is the use of SfT (Shape-from-Template) to facilitate geometric verification at the retrieval phase, for a few objects hypothesized using the new BoVW model. Existing methods use a rigid model, such as the fundamental matrix, or a simple deformable model based on semi-local constraints. SfT however is a physics-based method which uses an object’s 3D deformable model to reconstruct its isometric 3D deformation from a single input image. The output of SfT thus directly provides a geometric verification score. A byproduct of our work is to extend the scope of SfT. The proposed object retrieval framework is used to provide SfT with a few object hypotheses which may be quickly tested for the 3D deformable object selection. Performance evaluation on synthetic and real images reveals the benefits of our retrieval framework using a database with size varying between 20 and 1000 objects. The use of the new BoVW model and SfT versus the BoVW baseline and a rigid model improves the retrieval performance by 4.2% and 11.3% with p -values of 5 × 10 − 6 and 7 × 10 − 30 respectively.

The Shape-from-Template (SfT) problem is to recover the 3D shape of a deformable object from a single image, given a 3D template and a deformation constraint. We propose Particle-SfT, a new SfT algorithm which handles isometric and non-isometric deformations. We build Particle-SfT upon a particle system guided by deformation and reprojection constraint projections. Reconstruction is achieved by evolving particles to a globally attractive equilibrium, while taking observable external forces such as gravity into account, if any. Particle-SfT may be used to refine an existing initial shape. However, in practice we simply use the template as initial guess. This is because, as opposed to the existing refining methods, Particle-SfT has an extremely wide convergence basin. Particle-SfT is also faster than the existing refining methods. This is because it moves pieces of the shape's mesh independently to achieve larger step size by optimal constraint projection. We proved its convergence to a fixed-point. We experimented it with synthetic and real data. It has the same accuracy as the best performing isometric method and consistently outperforms all existing elastic methods in almost all cases, while being much faster.

Planar Structure-from-Motion (SfM) is the problem of reconstructing a planar object or surface from a set of 2D images using motion information. The problem is well-understood with the perspective camera model and can be solved with Homography Decomposition (HD). However when the structure is small and/or viewed far from the camera the perspective effects diminish, and in the limit the projections become affine. In these situations HD fails because the problem itself becomes ill-posed. We propose a stable alternative using affine camera models. These have been used extensively to reconstruct non-planar structures, however a general, accurate and closed-form method for planar structures has been missing. The problem is fundamentally different with planar structures because the types of affine camera models one can use are more restricted and it is inherently more ambiguous and non-linear. We provide a closed-form method for the orthographic camera model that solves the general problem (three or more views with three or more correspondences and missing correspondences) and returns all metric structure solutions and corresponding camera poses. The method does not require initialisation, and optimises an objective function that is very similar to the reprojection error. In fact there is no clear benefit in refining its solutions with bundle adjustment, which is a remarkable result. We also present a new theoretical analysis that deepens our understanding of the problem. The main result is the necessary and sufficient geometric conditions for the problem to be degenerate with the orthographic camera. We also show there can exist up to two solutions for metric structure with four or more views (previously it was assumed to be unique), and we give the necessary and sufficient geometric conditions for disambiguation. Other theoretical results include showing that in the case of three images the optimal reconstruction (with respect to reprojection error) can usually be found in closed-form, and additional prior knowledge needed to solve with non-orthographic affine cameras.

Objective To report the use of augmented reality (AR) in gynecology. Design AR is a surgical guidance technology that enables important hidden surface structures to be visualized in endoscopic images. AR has been used for other organs, but never in gynecology and never with a very mobile organ like the uterus. We have developed a new AR approach specifically for uterine surgery and demonstrated its use for myomectomy. Setting Tertiary university hospital. Patient(s) Three patients with one, two, and multiple myomas, respectively. Intervention(s) AR was used during laparoscopy to localize the myomas. Main Outcome Measure(s) Three-dimensional (3D) models of the patient's uterus and myomas were constructed before surgery from T2-weighted magnetic resonance imaging. The intraoperative 3D shape of the uterus was determined. These models were automatically aligned and "fused" with the laparoscopic video in real time. Result(s) The live fused video made the uterus appear semitransparent, and the surgeon can see the location of the myoma in real time while moving the laparoscope and the uterus. With this information, the surgeon can easily and quickly decide on how best to access the myoma. Conclusion(s) We developed an AR system for gynecologic surgery and have used it to improve laparoscopic myomectomy. Technically, the software we developed is very different to approaches tried for other organs, and it can handle significant challenges, including image blur, fast motion, and partial views of the organ.

Dense motion field estimation is a key computer vision problem. Many solutions have been proposed to compute small or large displacements, narrow or wide baseline stereo disparity, or non-rigid surface registration, but a unified methodology is still lacking. The authors introduce a general framework that robustly combines direct and feature-based matching. The feature-based cost is built around a novel robust distance function that handles keypoints and weak features such as segments. It allows us to use putative feature matches to guide dense motion estimation out of local minima. The authors’ framework uses a robust direct data term. It is implemented with a powerful second-order regularisation with external and self-occlusion reasoning. Their framework achieves state-of-the-art performance in several cases (standard optical flow benchmarks, wide-baseline stereo and non-rigid surface registration). Their framework has a modular design that customises to specific application needs.

We propose a new monocular 3-D reconstruction method adapted for reconstructing organs in the abdominal cavity. It combines both motion and shading cues. The former uses a conformal deformation prior and the latter the Cook-Torrance reflectance model. Our method runs in two phases: first, a 3-D geometric and photometric template of the organ at rest is reconstructed in vivo. The geometric shape is reconstructed using rigid shape-from-motion while the surgeon is exploring-but not deforming-structures in the abdominal cavity. This geometric template is then used to retrieve the photometric properties. A nonparametric model of the light's direction of the laparoscope and the Cook-Torrance reflectance model of the organ's tissue are estimated. Second, the surgeon manipulates and deforms the environment. Here, the 3-D template is conformally deformed to globally match a set of few correspondences between the 2-D image data provided by the monocular laparoscope and the 3-D template. Then, the coarse 3-D shape is refined using shading cues to obtain a final 3-D deformed shape. This second phase only relies on a single image. Therefore, it copes with both sequential processing and self-recovery from tracking failure. The proposed approach has been validated using 1) ex vivo and in vivo data with ground-truth, and 2) in vivo laparoscopic videos of a patient's uterus. Our experimental results illustrate the ability of our method to reconstruct natural 3-D deformations typical in real surgical procedures.

The deformable registration of a preoperative organ volume to an intraoperative laparoscopy image is required to achieve augmented reality in laparoscopy. This is an extremely challenging objective for the liver. This is because the preoperative volume is textureless, and the liver is deformed and only partially visible in the laparoscopy image. We solve this problem by modeling the preoperative volume as a Neo-Hookean elastic model, which we evolve under shading and contour cues. The contour cues combine the organ’s silhouette and a few curvilinear anatomical landmarks. The problem is difficult because the shading cue is highly nonconvex and the contour cues give curve-level (and not point-level) correspondences. We propose a convergent alternating projections algorithm, which achieves a \(4\%\) registration error.

International audience; We have developed a technique to study how good computers can be at diagnosing gastrointestinal lesions from regular (white light and narrow banded) colonoscopic videos compared to two levels of clinical knowledge (expert and beginner). Our technique includes a novel tissue classification approach which may save clinician's time by avoiding chromoendoscopy, a time-consuming staining procedure using indigo carmine. Our technique also discriminates the severity of individual lesions in patients with many polyps, so that the gastroenterologist can directly focus on those requiring polypectomy. Technically, we have designed and developed a framework combining machine learning and computer vision algorithms, which performs a virtual biopsy of hyperplastic lesions, serrated adenomas and adenomas. Serrated adenomas are very difficult to classify due to their mixed/hybrid nature and recent studies indicate that they can lead to colorectal cancer through the alternate serrated pathway. Our approach is the first step to avoid systematic biopsy for suspected hyperplastic tissues. We also propose a database of colonoscopic videos showing gastrointestinal lesions with ground truth collected from both expert image inspection and histology. We not only compare our system with the expert predictions, but we also study if the use of 3D shape features improves classification accuracy, and compare our technique's performance with three competitor methods.

Shape-from-Template (SfT) reconstructs the shape of a deforming surface from a single image, a 3D template and a deformation prior. For isometric deformations, this is a well-posed problem. However, previous methods which require no initialization break down when the perspective effects are small, which happens when the object is small or viewed from larger distances. That is, they do not handle all projection geometries. We propose stable SfT methods that accurately reconstruct the 3D shape for all projection geometries. We follow the existing approach of using first-order differential constraints and obtain local analytical solutions for depth and the first-order quantities: the depth-gradient or the surface normal. Previous methods use the depth solution directly to obtain the 3D shape. We prove that the depth solution is unstable when the projection geometry tends to affine, while the solution for the first-order quantities remain stable for all projection geometries. We therefore propose to solve SfT by first estimating the first-order quantities (either depth-gradient or surface normal) and integrating them to obtain shape. We validate our approach with extensive synthetic and real-world experiments and obtain significantly more accurate results compared to previous initialization-free methods. Our approach does not require any optimization, which makes it very fast.

Generalized procrustes analysis computes the best set of transformations that relate matched shape data. In shape analysis the transformations are usually chosen as similarities, while in general statistical data analysis other types of transformation groups such as the affine group may be used. Generalized procrustes analysis has a nonlinear and nonconvex formulation. The classical approach alternates the computation of a so-called reference shape and the computation of transformations relating this reference shape to each shape datum in turn. We propose the stratified approach to generalized procrustes analysis. It first uses the affine transformation group to analyze the data and then upgrades the solution to the sought after group, whether Euclidean or similarity. We derive a convex formulation for each of these two steps, and efficient practical algorithms that gracefully handle missing data (incomplete shapes). Extensive experimental results show that our approaches perform well on simulated and real data. In particular our closed-form solution gives very accurate results for generalized procrustes analysis of Euclidean data.

This paper presents a method for detecting a textured deformed surface in an image. It uses (wide-baseline) point matches between a template and the input image. The main contribution of the paper is twofold. First, we propose a robust method based on local surface smoothness capable of discarding outliers from the set of point matches. Our method handles large proportions of outliers (beyond 70% with less than 15% of false positives) even when the surface self-occludes. Second, we propose a method to estimate a self-occlusion resistant warp from point matches. Our method allows us to realistically retexture the input image. A pixel-based (direct) registration approach is also proposed. Bootstrapped by our robust point-based method, it finely tunes the warp parameters using the value (intensity or color) of all the visible surface pixels. The proposed framework was tested with simulated and real data. Convincing results are shown for the detection and retexturing of deformed surfaces in challenging images.

The direct registration problem for images of a deforming surface has been well studied. Parametric flexible warps based, for instance, on the Free-Form Deformation or a Radial Basis Function such as the Thin-Plate Spline, are often estimated using additive Gauss-Newton-like algorithms. The recently proposed compositional framework has been shown to be more efficient, but cannot be directly applied to such non-groupwise warps. Our main contribution in this paper is the Feature-Driven framework. It makes possible the use of compositional algorithms for most parametric warps such as those above mentioned. Two algorithms are proposed to demonstrate the relevance of our Feature-Driven framework: the Feature-Driven Inverse Compositional and the Feature-Driven Learning-based algorithms. As another contribution, a detailed derivation of the Feature-Driven warp parameterization is given for the Thin-Plate Spline and the Free-Form Deformation. We experimentally show that these two types of warps have a similar representational power. Experimental results show that our Feature-Driven registration algorithms are more efficient in terms of computational cost, without loss of accuracy, compared to existing methods.

The prediction sum of squares is a useful statistic for comparing different models. It is based on the principle of leave-one-out or ordinary cross-validation, whereby every measurement is considered in turn as a test set, for the model parameters trained on all but the held out measurement. As for linear least squares problems, there is a simple well-known non-iterative formula to compute the prediction sum of squares without having to refit the model as many times as the number of measurements. We extend this formula to cases where the problem has multiple parameter or measurement sets. We report experimental results on the fitting of a warp between two images, for which the number of deformation centres is automatically selected, based on one of the proposed non-iterative formulae.

Image registration consists in estimating geometric and photometric transformations that align two images as best as possible. The direct approach consists in minimizing the discrepancy in the intensity or color of the pixels. The inverse compositional algorithm has been recently proposed by Baker et al. for the direct estimation of groupwise geometric transformations. It is efficient in that it performs several computationally expensive calculations at a pre-computation phase. Photometric transformations act on the value of the pixels. They account for effects such as lighting change. Jointly estimating geometric and photometric transformations is thus important for many tasks such as image mosaicing. We propose an algorithm to jointly estimate groupwise geometric and photometric transformations while preserving the efficient pre-computation based design of the original inverse compositional algorithm. It is called the dual inverse compositional algorithm. It uses different approximations than the simultaneous inverse compositional algorithm and handles groupwise geometric and global photometric transformations. Its name stems from the fact that it uses an inverse compositional update rule for both the geometric and the photometric transformations. We demonstrate the proposed algorithm and compare it to previous ones on simulated and real data. This shows clear improvements in computational efficiency and in terms of convergence.