Search

Establishing correct correspondences between two images should consider both local and global spatial context. Given putative correspondences of feature points in two views, in this paper, we propose Order-Aware Network, which infers the probabilities of correspondences being inliers and regresses the relative pose encoded by the essential or fundamental matrix. Specifically, this proposed network is built hierarchically and comprises three operations. First, to capture the local context of sparse correspondences, the network clusters unordered input correspondences by learning a soft assignment matrix. These clusters are in canonical order and invariant to input permutations. Next, the clusters are spatially correlated to encode the global context of correspondences. After that, the context-encoded clusters are interpolated back to the original size and position to build a hierarchical architecture. We intensively experiment on both outdoor and indoor datasets. The accuracy of the two-view geometry and correspondences are significantly improved over the state-of-the-arts. Besides, based on the proposed method and advanced local feature, we won the first place in CVPR 2019 image matching workshop challenge and also achieve state-of-the-art results in the Visual Localization benchmark. Code is available at https://github.com/zjhthu/OANet. [ABSTRACT FROM AUTHOR]

Objective: Two-dimensional fluoroscopy is the standard guidance imaging method for closed endovascular intervention. However, two-dimensional fluoroscopy lacks depth perception for the intervention catheter and causes radiation exposure for both surgeons and patients. In this paper, we extend our previous study and develop the improved three-dimensional (3D) catheter shape estimation using ultrasound imaging. In addition, we perform further quantitative evaluations of endovascular navigation. Method: First, the catheter tracking accuracy in ultrasound images is improved by adjusting the state vector and adding direction information. Then, the 3D catheter points from the catheter tracking are further optimized based on the 3D catheter shape optimization with a high-quality sample set. Finally, the estimated 3D catheter shapes from ultrasound images are overlaid with preoperative 3D tissue structures for the intuitive endovascular navigation. Results: the tracking accuracy of the catheter increased by 24.39%, and the accuracy of the catheter shape optimization step also increased by approximately 17.34% compared with our previous study. Furthermore, the overall error of catheter shape estimation was further validated in the catheter intervention experiment of in vitro cardiovascular tissue and in a vivo swine, and the errors were 2.13 mm and 3.37 mm, respectively. Conclusion: Experimental results demonstrate that the improved catheter shape estimation using ultrasound imaging is accurate and appropriate for endovascular navigation. Significance: Improved navigation reduces the radiation risk because it decreases use of X-ray imaging. In addition, this navigation method can also provide accurate 3D catheter shape information for endovascular surgery. [ABSTRACT FROM AUTHOR]

Augmented reality (AR) surgical navigation systems based on image overlay have been used in minimally invasive surgery. However, conventional systems still suffer from a limited viewing zone, a shortage of intuitive three-dimensional (3D) image guidance and cannot be moved freely. To fuse the 3-D overlay image with the patient in situ, it is essential to track the overlay device while it is moving. A direct line-of-sight should be maintained between the optical markers and the tracker camera. In this study, we propose a moving-tolerant AR surgical navigation system using autostereoscopic image overlay, which can avoid the use of the optical tracking system during the intraoperative period. The system captures binocular image sequences of environmental change in the operation room to locate the overlay device, rather than tracking the device directly. Therefore, it is no longer required to maintain a direct line-of-sight between the tracker and the tracked devices. The movable range of the system is also not limited by the scope of the tracker camera. Computer simulation experiments demonstrate the reliability of the proposed moving-tolerant AR surgical navigation system. We also fabricate a computer-generated integral photography-based 3-D overlay AR system to validate the feasibility of the proposed moving-tolerant approach. Qualitative and quantitative experiments demonstrate that the proposed system can always fuse the 3-D image with the patient, thus, increasing the feasibility and reliability of traditional 3-D overlay image AR surgical navigation systems. [ABSTRACT FROM AUTHOR]

Automatic femur segmentation from computed tomography volume is a crucial but challenging task for computer-aided diagnosis in orthopedic surgeries. The main obstacles are weak bone boundaries, narrowness of joint space, variations in femur density and shape, as well as diverse leg postures. In this paper, we presented a novel 3-D feature-enhanced network to address these challenges. The novelty of our approach lies in two feature enhancement modules, including the edge detection task and the multi-scale features fusion. First, the edge detection task was embedded into femur segmentation from computed tomography volume to solve the problems of narrow joint space and weak femur boundary. Crucially, a task-specific edge detector was used to optimize the performance of femur segmentation in an end-to-end trainable system. Second, the multi-scale features fusion provided both local and global contexts to handle the problems of large variations in leg postures as well as femur shape and density. The results demonstrated that accurate 3-D femur segmentation with a high Dice similarity coefficient of 96.88% was achieved using the developed method, and the segmentation of computed tomography volume took 0.93 s on an average. [ABSTRACT FROM AUTHOR]

We propose a computer generated integral photography (CGIP) method that employs a lens based rendering (LBR) algorithm for super-multiview displays to achieve higher frame rates and better image quality without pixel resampling or view interpolation. The algorithm can utilize both fixed and programmable graphics pipelines to accelerate CGIP rendering and inter-perspective antialiasing. Two hardware prototypes were fabricated with two high-resolution liquid crystal displays and micro-lens arrays (MLA). Qualitative and quantitative experiments were performed to evaluate the feasibility of the proposed algorithm. To the best of our knowledge, the proposed LBR method outperforms state-of-the-art CGIP algorithms relative to rendering speed and image quality with our super-multiview hardware configurations. A demonstration experiment was also conducted to reveal the interactivity of a super-multiview display utilizing the proposed algorithm. [ABSTRACT FROM AUTHOR]

Intramedullary (IM) nail implantation is currently the standard treatment for femoral intertrochanteric fractures. However, individual differences in femur cavity bring a challenge in designing well-matched IM nails and cause difficulties in IM nail implantation. Therefore, there is an intense need to analyze femur cavities to predict difficulties in IM nail implantation to assist the design of IM nails. This study proposed a method to automatically identify subtypes of femur cavities that exhibit differences in potential difficulties in nail implantation by clustering the morphological features of femur models. The unsupervised subtype extraction method offers a scientific approach to stratify patients for designing and choosing well-matched IM nails. First, the quantitative morphological features of 422 femur cavities were extracted from computed tomography patient models. Second, 422 femur cavities were clustered into three distinct subtypes using a density peak-based k-means clustering method to provide a possible solution for the scientific design of IM nails. The effectiveness of the identified subtypes was validated by comparing subtype differences associated with IM nail implantation and the natural attributes of the patient. Quantitative evaluation of the mismatch degree and real clinical cases confirmed that the clustering results were clinically effective, with clear differences in the subtypes. Therefore, particular IM nails designed from the identified subtypes will potentially facilitate IM nail implantation and reduce complications. Compared with state-of-the-art methods, we used the largest scale dataset and unsupervised clustering to achieve subtype identification of femur cavities with clinical significance. [ABSTRACT FROM AUTHOR]

We develop a spatial position measurement system using three-dimensional (3-D) image marker-based tracking tools targeted at surgical navigation in minimally invasive surgery. We generate 3-D image markers with spatial information encoded to 2-D images, design tracking tools with the 3-D image markers, and analyze the tracking tools’ theoretical spatial errors, which are primarily limited by the spatial distribution of reconstructed fiducial 3-D markers. A pattern analysis-based positional measurement algorithm is developed to calculate the tool's spatial information using its spatial configuration. Evaluation experiments were conducted to demonstrate the accuracy and effectiveness of the proposed system. Furthermore, surgical navigation feasibility studies were performed. With a patient-image registration algorithm, a navigation interface that shows preoperative medical data and intraoperative information about the tool can intuitively and accurately assist surgeons. The results demonstrate that the proposed tracking tools, which have compact volume and spatial positional information, are of potential use in minimally invasive surgery in a limited space. [ABSTRACT FROM AUTHOR]

Objective: Precise minimally invasive surgery (MIS) has significant advantages over traditional open surgery in clinic. Although pre-/intraoperative diagnosis images can provide necessary guidance for therapy, hand–eye discoordination occurs when guidance information is displayed away from the surgical area. In this study, we introduce a real three-dimensional (3-D) see-through guidance system for precision surgery. Methods: To address the resolution and viewing angle limitation as well as the accuracy degradation problems of autostereoscopic 3-D display, we design a high quality and high accuracy 3-D integral videography (IV) medical image display method. Furthermore, a novel see-through microscopic device is proposed to assist surgeons with the superimposition of real 3-D guidance onto the surgical target is magnified by an optical visual magnifier module. Results: Spatial resolutions of 3-D IV image in different depths have been increased 50%∼70%, viewing angles of different image sizes have been increased 9%∼19% compared with conventional IV display methods. Average accuracy of real 3-D guidance superimposed on surgical target was 0.93 mm ± 0.41 mm. Preclinical studies demonstrated that our system could provide real 3-D perception of anatomic structures inside the patient's body. Conclusion: The system showed potential clinical feasibility to provide intuitive and accurate in situ see-through guidance for microsurgery without restriction on observers’ viewing position. Significance: Our system can effectively improve the precision and reliability of surgical guidance. It will have wider applicability in surgical planning, microscopy, and other fields. [ABSTRACT FROM PUBLISHER]

In endovascular catheter interventions, the determination of the three-dimensional (3D) catheter shape can increase navigation information and help reduce trauma. This study describes a shape determination method for a flexible interventional catheter using ultrasound scanning and a two-step particle filter without X-ray fluoroscopy. First, we propose a multi-feature, multi-template particle filter algorithm for accurate catheter tracking from ultrasound images. Second, we model the mechanical behavior of the catheter and apply a particle filter shape optimization algorithm to refine the results from the first step. Finally, the acquired catheter’s 3D shapes are displayed together with the preoperative 3D images of the cardiac structures to provide intuitive endovascular navigation. We validated our method using ultrasound scanning of the straight and curved catheters in a water tank, and the shape determination errors were 1.44 ± 0.38 mm and 1.95 ± 0.46 mm, respectively. Further, endovascular catheter shape determination was validated in a catheter intervention experiment with a heart phantom. The error of the acquired endovascular catheter shape was 2.23 ± 0.87 mm. These results demonstrate that our two-step method is both accurate and effective. Using ultrasound scanning for shape determination of a flexible catheter will be helpful in endovascular interventions, reducing exposure to radiation and providing rich navigation information. [ABSTRACT FROM AUTHOR]

A 360° three-dimensional (3D) display is one of the most attractive displays for full parallax in horizontal direction. In this paper, we proposed two 360° crosstalk-free 3D autostereoscopic display methods based on pyramidal mirrors-based 3D light field multiplexing and diaphragms. Display performances of proposed displays were analyzed. We fabricated two 360° display prototypes based on the n-gonal pyramidal mirrors assemblage and flat 3D displays. Moreover, we evaluated our proposed methods by simulations and experiments. Experimental results as well as simulation results were consistent with theoretical analysis based on the geometric optics. Furthermore, the proposed schemes and theories may provide a guideline for light field multiplexing and wide viewing angle augmented reality. [ABSTRACT FROM PUBLISHER]

Quality of three-dimensional (3D) displays depends on optical apparatus setups and image generation algorithms. We present a 3D quantitative evaluation method and develop a corresponding system to evaluate the performance of 3D autostereoscopic display. The evaluation system consists of a set of dedicated 3D evaluation reference patterns, a 3D image acquisition platform and a personal computer for auto-control, image processing, and quantitative analysis. Using the proposed method and system, we evaluate resolution, viewing angle, accuracy of image depth, and quality of displayed image of the 3D autostereoscopic display. Furthermore, we perform the evaluation experiments using two integral photography-based 3D displays. The quantificational evaluation results of main parameters are largely consistent with results of theoretical analysis of evaluated displays and well verify feasibility and availability of the evaluation system. An application experiment using one lenticular-based 3D display is also conducted to verify the universality of our proposed evaluation system. [ABSTRACT FROM PUBLISHER]

Computer-assisted oral and maxillofacial surgery (OMS) has been rapidly evolving since the last decade. State-of-the-art surgical navigation in OMS still suffers from bulky tracking sensors, troublesome image registration procedures, patient movement, loss of depth perception in visual guidance, and low navigation accuracy. We present an augmented reality navigation system with automatic marker-free image registration using 3-D image overlay and stereo tracking for dental surgery. A customized stereo camera is designed to track both the patient and instrument. Image registration is performed by patient tracking and real-time 3-D contour matching, without requiring any fiducial and reference markers. Real-time autostereoscopic 3-D imaging is implemented with the help of a consumer-level graphics processing unit. The resulting 3-D image of the patient’s anatomy is overlaid on the surgical site by a half-silvered mirror using image registration and IP-camera registration to guide the surgeon by exposing hidden critical structures. The 3-D image of the surgical instrument is also overlaid over the real one for an augmented display. The 3-D images present both stereo and motion parallax from which depth perception can be obtained. Experiments were performed to evaluate various aspects of the system; the overall image overlay error of the proposed system was 0.71 mm. [ABSTRACT FROM PUBLISHER]