Search

In this paper, we study the indirect relationship between the adoption of Computer-aided Detection or Diagnostic (CADe or CADx) systems for Hepatic Resection (HR) and the patient’s health post-surgery. We vary the number, actual size, and the estimated size of tumors along with model parameters of tumor growth over 1000 simulations of HR according to predefined statistical distributions of parameter values. The average time (t) taken by the tumors to relapse is assessed for the non-adoption of CAD (Case 1), the adoption of semi-automatic CAD (Case 2) and the adoption of automatic CAD (Case 3) in HR. In this study, we have simulated 126 automatic CAD algorithms (Case 3). For tumor volumes (TV) less than 50 cm, if administration of bevacizumab, a post-operative therapy, is (not) adopted in the simulation, t is found to be 646, 84, and 60 days (40, 24, and 17 days) for Case 1, Case 2, and Case 3, respectively. For TV greater than 50 cm, and with (without) bevacizumab, t is found to be 86, 1, and 6 days (28, 6, and 3 days) for Case 1, Case 2, and Case 3, respectively. For with (without) bevacizumab treatment and for all tumor volumes, t is found to be 260, 90, and 104 days (38, 13, and 11 days) for Case 1, Case 2, and Case 3, respectively. We have observed that the tumors relapsed quickly in those cases where CAD was adopted.

Segmentation of abdominal Computed Tomography (CT) scan is essential for analyzing, diagnosing, and treating visceral organ diseases (e.g., hepatocellular carcinoma). This paper proposes a novel neural network (Res-PAC-UNet) that employs a fixed-width residual UNet backbone and Pyramid Atrous Convolutions, providing a low disk utilization method for precise liver CT segmentation. The proposed network is trained on medical segmentation decathlon dataset using a modified surface loss function. Additionally, we evaluate its quantitative and qualitative performance; the Res16-PAC-UNet achieves a Dice coefficient of 0.950 ± 0.019 with less than half a million parameters. Alternatively, the Res32-PAC-UNet obtains a Dice coefficient of 0.958 ± 0.015 with an acceptable parameter count of approximately 1.2 million.

Introduction: Percutaneous thermal ablation is widely adopted as a curative treatment approach for unresectable liver neoplasms. Accurate immediate assessment of therapeutic response post-ablation is critical to achieve favourable outcomes. The conventional technique of side-by-side comparison of pre- and post-ablation scans is challenging and hence there is a need for improved methods, which will accurately evaluate the immediate post-therapeutic response. Objectives and Significance: This review summarizes the findings of studies investigating the feasibility and efficacy of the fusion imaging systems in the immediate post-operative assessment of the therapeutic response to thermal ablation in liver neoplasms. The findings could potentially empower the clinicians with updated knowledge of the state-of-the-art in the assessment of treatment response for unresectable liver neoplasms. Methods and Analysis: A rapid review will be performed on publicly available major electronic databases to identify articles reporting the feasibility and efficacy of the fusion imaging systems in the immediate assessment of the therapeutic response to thermal ablation in liver neoplasms. The risk of bias and quality of articles will be assessed using the Cochrane risk of bias tool 2.0 and Newcastle Ottawa tool. Ethics and Dissemination: Being a review, we do not anticipate the need for any approval from the Institutional Review Board. The outcomes of this study will be published in a peer-reviewed journal. Highlights Evaluation of the therapeutic response in liver neoplasms immediately post-ablation is critical to achieve favourable patient outcomes. We will examine the feasibility and technical efficacy of different fusion imaging systems in assessing the immediate treatment response post-ablation. The findings are expected to guide the clinicians with updated knowledge on the state-of-the-art when assessing the immediate treatment response for unresectable liver neoplasms.

Recent tele-mentoring technologies for minimally invasive surgery (MIS) augments the operative field with movements of virtual surgical instruments as visual cues. The objective of this work is to assess different user-interfaces that effectively transfer mentor's hand gestures to the movements of virtual surgical instruments.A user study was conducted to assess three different user-interface devices (Oculus-Rift, SpaceMouse, Touch Haptic device) under various scenarios. The devices were integrated with a MIS tele-mentoring framework for control of both manual and robotic virtual surgical instruments.The user study revealed that Oculus Rift is preferred during robotic scenarios, whereas the touch haptic device is more suitable during manual scenarios for tele-mentoring.A user-interface device in the form of a stylus controlled by fingers for pointing in 3D space is more suitable for manual MIS, whereas a user-interface that can be moved and oriented easily in 3D space by wrist motion is more suitable for robotic MIS.

Medical images (e.g., magnetic resonance imaging (MRI) and computed tomography (CT)) provide critical information to the clinicians in order to diagnose pathology and plan interventions. Image segmentation is the first and foremost step taken by the clinicians while optimizing analytic diagnosis and treatment planning for interventions (e.g., transplantation and complete resection) and therapeutic procedures (e.g., radiotherapy, PVE, and embolization approaches), especially in hepatocellular carcinoma. Thus, segmentation techniques certainly impact the diagnosis and treatment outcomes. This paper studies the literature during the year 2012 until 2021 and reviews the segmentation methods classifying them into three categories based on their clinical utility (i.e., surgical and radiological interventions). The classification is based on the parameters such as precision, accuracy, location, liver condition, and other clinical considerations.

Mixed reality-based medical holography is expected to have powerful applications in many surgical fields through easy and intuitive interaction with anatomic holograms using gestures. However, interaction with nested holograms using default gesture interaction, as is the case in visualized complex anatomic structures, is found to be challenging. This work develops, implements, and compares two modes of interaction with nested holograms, using native gestures and 3D SpaceMouse. Each of this mode uses ray casting and either Box Collider (BC) or Mesh Collider (MC) techniques to select target hologram. The use of SpaceMouse is found to have superior control over the selection of nested holograms irrespective of the mode of interaction, while a combination of native gestures and BC mode seems to most improve the efficiency of interaction of a selected nested hologram.

The blood-brain barrier (BBB) is a prime focus for clinicians to maintain the homeostatic function in health and deliver the theranostics in brain cancer and number of neurological diseases. The structural hierarchy and in situ biochemical signaling of BBB neurovascular unit have been primary targets to recapitulate into the in vitro modules. The microengineered perfusion systems and development in 3D cellular and organoid culture have given a major thrust to BBB research for neuropharmacology. In this review, we focus on revisiting the nanoparticles based bimolecular engineering to enable them to maneuver, control, target, and deliver the theranostic payloads across cellular BBB as nanorobots or nanobots. Subsequently we provide a brief outline of specific case studies addressing the payload delivery in brain tumor and neurological disorders (e.g., Alzheimer's disease, Parkinson's disease, multiple sclerosis, etc.). In addition, we also address the opportunities and challenges across the nanorobots' development and design. Finally, we address how computationally powered machine learning (ML) tools and artificial intelligence (AI) can be partnered with robotics to predict and design the next generation nanorobots to interact and deliver across the BBB without causing damage, toxicity, or malfunctions. The content of this review could be references to multidisciplinary science to clinicians, roboticists, chemists, and bioengineers involved in cutting-edge pharmaceutical design and BBB research.

Training and skills assessment for robotic surgeries has received unprecedented attention in technological research in the last few years due to the gradual adoption of minimally invasive robotic interventions. Robot-assisted radical prostatectomy (RARP) is one of these interventions that help cure cancer by completely removing the prostate. Trainers currently use numerous training methodologies to teach novice surgeons macro RARP skills. However, limited resources for teaching and assessing micro-skills, such as the optimal urethra dissection ranges given basic variables: prostate size and prostate cancer location, are used in practice. Therefore, we built an interactive prototype to teach these skills. This paper validates the prototype using content validity and Questionnaire for User Interaction Satisfaction (QUIS) with five surgeons. The results demonstrate high content validity and increased end-user satisfaction.

Background User interfaces play a vital role in the planning and execution of an interventional procedure. The objective of this study is to investigate the effect of using different user interfaces for planning transrectal robot-assisted MR-guided prostate biopsy (MRgPBx) in an augmented reality (AR) environment. Method End-user studies were conducted by simulating an MRgPBx system with end- and side-firing modes. The information from the system to the operator was rendered on HoloLens as an output interface. Joystick, mouse/keyboard, and holographic menus were used as input interfaces to the system. Results The studies indicated that using a joystick improved the interactive capacity and enabled operator to plan MRgPBx in less time. It efficiently captures the operator's commands to manipulate the augmented environment representing the state of MRgPBx system. Conclusions The study demonstrates an alternative to conventional input interfaces to interact and manipulate an AR environment within the context of MRgPBx planning.

Accurate organ segmentation is nowadays considered indispensable in typical surgical pipelines. Over the last couple of decades, researchers and clinicians have been driven to achieve the most efficient segmentation technique that would allow radiologists and surgeons to have easy access to organ measurements and visualization. Although several implementations and systems have been implemented, there still lies plenty of space in creating a more robust, accurate, reliable, and fast segmentation algorithm. Since automated segmentation algorithms have less user interaction (minimizing the probability of performance variation) and are usually preferred by the clinicians, the aim of this paper is to provide a state-of-art automated solution for accurately segmenting the liver and liver tumor from CT volumes. The implementation utilizes a deep learning hybrid network that blends features extracted from both 2D and 3D convolutions to enhance the overall performance of the algorithm, in addition to some pre-processing and postprocessing. We tested this network on both public and private data and it showed promising results that would be highly applicable in surgical setups. Finally, we briefly discuss how these results are relevant in a typical surgical application.

Purpose: Cerebral aneurysms are one of the prevalent cerebrovascular disorders in adults worldwide and caused by a weakness in the brain artery. The most impressive treatment for a brain aneurysm is interventional radiology treatment, which is extremely dependent on the skill level of the radiologist. Hence, accurate detection and effective therapy for cerebral aneurysms still remain important clinical challenges. In this work, we have introduced a pipeline for cerebral blood flow simulation and real-time visualization incorporating all aspects from medical image acquisition to real-time visualization and steering. Methods: We have developed and employed an improved version of HemeLB as the main computational core of the pipeline. HemeLB is a massive parallel lattice-Boltzmann fluid solver optimized for sparse and complex geometries. The visualization component of this pipeline is based on the ray marching method implemented on CUDA capable GPU cores. Results: The proposed visualization engine is evaluated comprehensively and the reported results demonstrate that it achieves significantly higher scalability and sites updates per second, indicating higher update rate of geometry sites' values, in comparison with the original HemeLB. This proposed engine is more than two times faster and capable of 3D visualization of the results by processing more than 30 frames per second. Conclusion: A reliable modeling and visualizing environment for measuring and displaying blood flow patterns in vivo, which can provide insight into the hemodynamic characteristics of cerebral aneurysms, is presented in this work. This pipeline increases the speed of visualization and maximizes the performance of the processing units to do the tasks by breaking them into smaller tasks and working with GPU to render the images. Hence, the proposed pipeline can be applied as part of clinical routines to provide the clinicians with the real-time cerebral blood flow-related information. 2020, CARS. This study was made possible by a National Priorities Research Program (NPRP) Grant No. 5-792-2-328 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors. Scopus

Segmentation of brain aneurysm is of paramount importance in aneurysm treatment planning. However, the segmentation of intensity varying and low-contrast cerebral blood vessels is an extremely challenging task. In this paper, we present an approach to segmenting the brain vasculature in low contrast computed tomography angiography and magnetic resonance angiography. The main contributions are: (1) a stochastic resonance based methodology in discrete Hartley transform domain is developed to enhance the contrast of a selected angiographic image for patch placement, and (2) a multi-scale adaptive principal component analysis based method is proposed that estimates the phase map of input images. Level-set method is applied to the phase-map data in order to segment the vasculature. We have tested the algorithm on real datasets obtained from two sources, CIBC institute and Hamad Medical corporation. The average Dice coefficient (in %) is found to be $$94.1\pm 1.2$$ (value indicates the mean and standard deviation) whereas average false positive ratio, false negative ratio, and specificity are found to be 0.019, $$7.55\times 10^{-3}$$ , and 0.75, respectively. Average Hausdorff distance between segmented contour and ground truth is determined to be 2.97 mm. The qualitative and quantitative results show promising segmentation accuracy reflecting the potential of the proposed method.

Background and objectives Surgical procedures such as laparoscopic and robotic surgeries are popular since they are invasive in nature and use miniaturized surgical instruments for small incisions. Tracking of the instruments (graspers, needle drivers) and field of view from the stereoscopic camera during surgery could further help the surgeons to remain focussed and reduce the probability of committing any mistakes. Tracking is usually preferred in computerized video surveillance, traffic monitoring, military surveillance system, and vehicle navigation. Despite the numerous efforts over the last few years, object tracking still remains an open research problem, mainly due to motion blur, image noise, lack of image texture, and occlusion. Most of the existing object tracking methods are time-consuming and less accurate when the input video contains high volume of information and more number of instruments. Methods This paper presents a variational framework to track the motion of moving objects in surgery videos. The key contributions are as follows: (1) A denoising method using stochastic resonance in maximal overlap discrete wavelet transform is proposed and (2) a robust energy functional based on Bhattacharyya coefficient to match the target region in the first frame of the input sequence with the subsequent frames using a similarity metric is developed. A modified affine transformation-based registration is used to estimate the motion of the features following an active contour-based segmentation method to converge the contour resulted from the registration process. Results and conclusion The proposed method has been implemented on publicly available databases; the results are found satisfactory. Overlap index (OI) is used to evaluate the tracking performance, and the maximum OI is found to be 76% and 88% on private data and public data sequences.

Cerebral aneurysms are among most prevalent and devastating cerebrovascular diseases of adult population worldwide. The resulting sequelae of untimely/inadequate therapeutic intervention include sub-arachnoid hemorrhage. Geometric modeling of aneurysm being the first step in the treatment planning, the scientists therefore focus more on segmentation of aneurysm rather than its detection. A successful aneurysm detection among the bunch of vessels would certainly facilitate and ease the segmentation process. In this work, we present a novel method for aneurysm detection; the key contributions are: contrast enhancement of input image using stochastic resonance concept in wavelet domain, adaptive thresholding, and modified Hough Circle Transform. Experimental results show that the proposed method is efficient in detecting the location and type of aneurysm. This work was partly supported by NPRP Grant #NPRP 5-792-2-328 from the Qatar National Research Fund (a member of the Qatar Foundation). Scopus

Despite great advances in medical image segmentation, the accurate and automatic segmentation of endoscopic scenes remains a challenging problem. Two important aspects have to be considered in segmenting an endoscopic scene: (1) noise and clutter due to light reflection and smoke from cutting tissue, and (2) structure occlusion (e.g. vessels occluded by fat, or endophytic tumours occluded by healthy kidney tissue). In this paper, we propose a variational technique to augment a surgeon’s endoscopic view by segmenting visible as well as occluded structures in the intraoperative endoscopic view. Our method estimates the 3D pose and deformation of anatomical structures segmented from 3D preoperative data in order to align to and segment corresponding structures in 2D intraoperative endoscopic views. Our preoperative to intraoperative alignment is driven by, first, spatio-temporal, signal processing based vessel pulsation cues and, second, machine learning based analysis of colour and textural visual cues. To our knowledge, this is the first work that utilizes vascular pulsation cues for guiding preoperative to intraoperative registration. In addition, we incorporate a tissue-specific (i.e. heterogeneous) physically based deformation model into our framework to cope with the non-rigid deformation of structures that occurs during the intervention. We validated the utility of our technique on fifteen challenging clinical cases with 45 % improvements in accuracy compared to the state-of-the-art method. A new technique for localizing both visible and occluded structures in an endoscopic view was proposed and tested. This method leverages both preoperative data, as a source of patient-specific prior knowledge, as well as vasculature pulsation and endoscopic visual cues in order to accurately segment the highly noisy and cluttered environment of an endoscopic video. Our results on in vivo clinical cases of partial nephrectomy illustrate the potential of the proposed framework for augmented reality applications in minimally invasive surgeries.

We present a new method to segment low contrast liver CT images with high noise level.We utilize the noise constructively to enhance the input image contrast.The segmentation method is based on cellular automata.Level sets are used to generate segmentation of intermediate slices.The results show good segmentation accuracy when compared with ground truth images. Liver segmentation continues to remain a major challenge, largely due to its intensity complexity with surrounding anatomical structures (stomach, kidney, and heart), high noise level and lack of contrast in pathological computed tomography data. In this paper, we present an approach to reconstructing the liver surface in low contrast computed tomography. The main contributions are: (1) a stochastic resonance based methodology in discrete cosine transform domain is developed to enhance the contrast of pathological liver images, (2) a new formulation is proposed to prevent the object boundary, resulted by cellular automata method, from leaking into the surrounding areas of similar intensity, and (3) a level-set method is suggested to generate intermediate segmentation contours from two segmented slices distantly located in a subject sequence. We have tested the algorithm on real datasets obtained from two sources, Hamad General Hospital and MICCAI Grand Challenge workshop. Both qualitative and quantitative evaluation performed on liver data show promising segmentation accuracy when compared with ground truth data reflecting the potential of the proposed method.

In image-guided robotic surgery, segmenting the endoscopic video stream into meaningful parts provides important contextual information that surgeons can exploit to enhance their perception of the surgical scene. This information provides surgeons with real-time decision-making guidance before initiating critical tasks such as tissue cutting. Segmenting endoscopic video is a challenging problem due to a variety of complications including significant noise attributed to bleeding and smoke from cutting, poor appearance contrast between different tissue types, occluding surgical tools, and limited visibility of the objects' geometries on the projected camera views. In this paper, we propose a multi-modal approach to segmentation where preoperative 3D computed tomography scans and intraoperative stereo-endoscopic video data are jointly analyzed. The idea is to segment multiple poorly visible structures in the stereo/multichannel endoscopic videos by fusing reliable prior knowledge captured from the preoperative 3D scans. More specifically, we estimate and track the pose of the preoperative models in 3D and consider the models' non-rigid deformations to match with corresponding visual cues in multi-channel endoscopic video and segment the objects of interest. Further, contrary to most augmented reality frameworks in endoscopic surgery that assume known camera parameters, an assumption that is often violated during surgery due to non-optimal camera calibration and changes in camera focus/zoom, our method embeds these parameters into the optimization hence correcting the calibration parameters within the segmentation process. We evaluate our technique on synthetic data, ex vivo lamb kidney datasets, and in vivo clinical partial nephrectomy surgery with results demonstrating high accuracy and robustness.

Background The aim of this study was to enhance the visual feedback of surgeons, during robotic surgeries, by designing and developing an actuated 2D imaging probe, which is used in conjunction with the traditional stereoscopic camera of the da Vinci surgical system. The probe provides the surgeon with additional visual cues, overcoming visualization constraints encountered during certain scenarios of robot-assisted minimally invasive surgery. Methods The actuated imaging probe is implemented as a master–slave tele-manipulated system, and it is designed to be compatible with the da Vinci surgical system. The detachable probe design enables it to be mounted on any of the EndoWrist® instruments of the robot and is controlled by the surgeon using a custom-made pedal system. The image from the 2D probe is rendered along with the stereoscopic view on the surgeon’s console. Results The experimental results demonstrate the effectiveness of the proposed actuated imaging probe when used as an additional visualization channel and in surgical scenarios presenting visual problems due to tissue occlusion. Conclusion The study shows the potential benefits of an additional actuated imaging probe when used in conjunction with traditional surgical instruments to perform surgical tasks requiring visualization from multiple orientations and workspaces. Acknowledgments This work was partially supported by the National Priorities Research Program (NPRP) under Grant NPRP 09-776-2-298 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors. We would like to thank Dr. Jean-Marc Peyrat for his assistance in the experiments. Scopus

Hilar dissection is an important and delicate stage in partial nephrectomy, during which surgeons remove connective tissue surrounding renal vasculature. Serious complications arise when the occluded blood vessels, concealed by fat, are missed in the endoscopic view and as a result are not appropriately clamped. Such complications may include catastrophic blood loss from internal bleeding and associated occlusion of the surgical view during the excision of the cancerous mass (due to heavy bleeding), both of which may compromise the visibility of surgical margins or even result in a conversion from a minimally invasive to an open intervention. To aid in vessel discovery, we propose a novel automatic method to segment occluded vasculature from labeling minute pulsatile motion that is otherwise imperceptible with the naked eye. Our segmentation technique extracts subtle tissue motions using a technique adapted from phase-based video magnification, in which we measure motion from periodic changes in local phase information albeit for labeling rather than magnification. Based on measuring local phase through spatial decomposition of each frame of the endoscopic video using complex wavelet pairs, our approach assigns segmentation labels by detecting regions exhibiting temporal local phase changes matching the heart rate. We demonstrate how our technique is a practical solution for time-critical surgical applications by presenting quantitative and qualitative performance evaluations of our vessel detection algorithms with a retrospective study of fifteen clinical robot-assisted partial nephrectomies.

Needle–based image–guided interventions (NB-IGI) are well established practices that are rapidly expanding to a wider range of therapeutic and diagnostic interventions due to their impact on clinical outcomes. In parallel, a number of robotic manipulators are emerging to increase accuracy, decrease inter-operator variabilities, and reduce the duration of the procedure. Most current systems are application–specific, thereby are appropriate for a limited number of procedures and further exacerbating the already rising costs of healthcare. In order to expand clinical usage of robotic NB-IGI, and eventually reduce the burden on healthcare costs, this paper proposes a modular, customizable robotic systems concept. The concept includes simplified hardware and control techniques, and initial results demonstrate that accuracy is within clinically acceptable limits. Preliminary designs, implementations, and assessments of the proposed system demonstrate its potential and clinical value.

Despite advanced medical imaging technologies being available, image segmentation still continues to represent a major bottleneck in practical applications due to high noise level and lack of contrast. This paper presents a semi-automatic algorithm that uses stochastic resonance as a pre-processing step for enhancing the contrast of input magnetic resonance (MR) images followed by a cellular automata (CA)-based mechanism for surface reconstruction of the left ventricle (LV) from a few selected MR slices. The main contribution, in this work, is a new formulation for preventing the CA method to leak into surrounding areas of similar intensity. Another contribution is the use of level sets for segmenting the rest of the slices of a subject automatically between the selected slices segmented by the CA to build the LV surface. Both qualitative and quantitative evaluations performed on York and MICCAI Grand Challenge workshop database of MR images show promising segmentation accuracy when compared with the grou...

Despite increased image quality including medical imaging, image segmentation continues to represent a major bottleneck in practical applications due to noise and lack of contrast. In this paper, we present a new methodology to segment noisy, low contrast medical images, with a view to developing practical applications. Firstly, the contrast of the image is enhanced and then a modified graph-based method is followed. This paper has mainly two contributions: (1) a contrast enhancement stage performed by suitably utilizing the noise present in the medical data. This step is achieved through stochastic resonance theory applied in the wavelet domain and (2) a new weighting function is proposed for traditional graph-based approaches. Both qualitative (by our clinicians/radiologists) and quantitative evaluation performed on publicly available computed tomography (CT) (MICCAI 2007 Grand Challenge workshop database) and cardiac magnetic resonance (CMR) databases reflect the potential of the proposed method even in the presence of tumors/papillary muscles.

Open posterior spinal procedures involve extensive soft tissue disruption, increased hospital length of stay, and disfiguring scars. Our aim was to demonstrate the feasibility of using robotic-assistance for minimally invasive exposure of the posterolateral spine with and without carbon dioxide (COSheep specimens underwent minimally invasive subperiosteal dissection of the spine during three trials. The da Vinci S Surgical system was used for access with and without working space support via COWithout insufflation, a sub-paraspinal muscle tunnel measuring 16 cm was developed between two 5 cm incisions. With insufflation, the one-sided tunnel length was 12.5 cm but without the soft tissue trauma and obstructed visualization experienced without COThe use of robot-assistance for minimally invasive access to the posterior spine appears to be feasible. The use of CO

© 1982-2012 IEEE.Reconstruction of the anterior cruciate ligament (ACL) through arthroscopy is one of the most common procedures in orthopaedics. It requires accurate alignment and drilling of the tibial and femoral tunnels through which the ligament graft is attached. Although commercial computer-Assisted navigation systems exist to guide the placement of these tunnels, most of them are limited to a fixed pose without due consideration of dynamic factors involved in different knee flexion angles. This paper presents a new model for intraoperative guidance of arthroscopic ACL reconstruction with reduced error particularly in the ligament attachment area. The method uses 3D preoperative data at different flexion angles to build a subject-specific statistical model of knee pose. To circumvent the problem of limited training samples and ensure physically meaningful pose instantiation, homogeneous transformations between different poses and local-deformation finite element modelling are used to enlarge the training set. Subsequently, an anatomical geodesic flexion analysis is performed to extract the subject-specific flexion characteristics. The advantages of the method were also tested by detailed comparison to standard Principal Component Analysis (PCA), nonlinear PCA without training set enlargement, and other state-of-The-Art articulated joint modelling methods. The method yielded sub-millimetre accuracy, demonstrating its potential clinical value.

Background and ObjectiveOften times, clinicians use a three-dimensional set of medical images to diagnose and plan treatments, which typically include visual identification of structures such as bones and tissues [1]. This can be a challenging task as anatomical structures of interest can contain significant noise, and easily blend with neighboring tissues. We propose to tackle 2 cases: (a) treatment planning of pelvic fractures where a small size ring formed by the fused bones of the ischium, ilium, and pubis attached to the sacrum contains vital structures (including major blood vessels, nerves, digestive and reproductive organs) and should be carefully delineated, and (b) liver cancer treatment where malignant tissue has to be carefully removed. The pixel intensity of tumor is similar to those of healthy tissue and proper delineation is of utmost important before proceeding to plan therapy.To address the aforementioned challenges, in this work we present a soft-clustering technique using Enhanced Fuzzy C-Means (EFCM) along with a bilateral filter to detect the region of interest. The key feature of the proposed algorithm combines domain and range filtering allowing the filter to maintain balance between preservation of relevant details and the degree of noise reduction. The approach allows traditional Fuzzy C-Means not only to exploit useful spatial information, but also to dynamically minimize clustering errors caused by common noise in medical images.MethodologyA three-step workflow is used to process the medical images:Step 1: After MR/CT images are acquired; clinician initially draws a rough outline around the region of interest (where the fracture or tumor is present) on the two-dimensional image slices. The manual input reduces the computational time to determine the desired tissue cluster by providing the region of interest instead of scanning/processing the entire image.Step 2: In this step, a bilateral filter is used to remove noise while preserving details of the edges [2]. Linear filters, such as Gaussian, compute a weighted average of pixel values in the neighborhood. The weights decrease with distance from the neighborhood center. This works well for images where local neighboring pixels have similar values (slow spatial variation). As the noise that corrupts these neighboring pixels is mutually less correlated than the signal, the noise is averaged away while signal is preserved. However, the assumption of slow spatial variations fails at edges, which are consequently blurred by linear low-pass filtering. In this context, we use a non-linear/bilateral filter that combines both domain and range filtering. In smooth regions, the pixel values within a local neighborhood are similar to each other, and the normalized similarity function is close to one. Consequently, the bilateral filter acts essentially as a standard domain filter, averages away the weakly correlated differences between pixel values caused by noise, and preserves edge details.Step 3: As a last step, EFCM clustering algorithm is applied to the noise-filtered image. Fuzzy C-Means clustering works by assigning membership to each data point with respect to the cluster centers [3]. A distance is computed between the cluster center and the data point. The membership of the data towards a particular cluster center varies linearly as per the distance. Closer the data to a cluster center, higher is its membership. The summation of membership of each data point across different clusters is equal to one. An objective function based on the Euclidean metric is then used to update the membership and cluster centers iteratively. However, the parameter estimation resulting from the described objective function may not be robust in a noisy environment. Therefore in this work, we develop an algorithm that uses a modified Euclidean term (described in Table I) that is robust against noise and allows meaningful clustering of compact pixels for image analysis by the clinicians (Fig. 1).Results and ConclusionThe method proposed in this work was evaluated using two datasets: (a) CT images of pelvic fracture (two subjects) publicly available online for research purposes, on OsiriX website (http://www.osirix-viewer.com/datasets). The image acquisition details are as followed: Slice Thickness: 2 mm, Pixel Spacing: 0.29 mm × 0.29 mm, Bit-depth: 12, and Acquisition Matrix: 512 × 512. (b) CT images containing liver with tumor from five anonymized subjects, obtained from Hamad Medical Corporation, Doha, Qatar. The image acquisition details are as followed: Slice Thickness: 3 mm, Pixel Spacing: 0.32 mm × 0.32 mm, Bit-depth: 16, and Acquisition Matrix: 512 × 512.The algorithms were implemented on MATLAB R2013a running on a workstation with 16 GB RAM and 2.8 GHz Intel processor. The average time required to perform segmentation was recorded as 8 ± 1.5 minutes. However the computational time could be further reduced, as implementation was done without optimization of the internal function calls. An initial assessment of the experimental results has shown satisfactory outcomes in both the cases to detect pelvic fracture and liver tumor (Fig. 1). The use of traditional linear filters on the datasets has failed to identify clusters with similar pixel intensity values. The use of bilateral filter with Euclidean modification proposed in this work has lead to desired soft clustering identifying the required anatomical structures in the images.In future, we plan to optimize and validate the method extensively on different tissue-types using multiple imaging modalities.References[1] Vona G. et al., “Impact of cyto-morphological detection of circulating tumor cells in patients with liver cancer,” Hepatology, 39, 792–797, 2004.[2] Sugimoto K. et al., “Compressive Bilateral Filtering,” IEEE Trans. on Image Processing, 24, 3357–3369, 2015.[3] Havens T. C., “Fuzzy c-Means Algorithms for Very Large Data,” IEEE Trans. on Fuzzy Systems, 20, 1130–1146, 2012.

Laparoscopic ultrasound (US) is often used during partial nephrectomy surgeries to identify tumour boundaries within the kidney. However, visual identification is challenging as tumour appearance varies across patients and US images exhibit significant noise levels. To address these challenges, we present the first fully automatic method for detecting the presence of kidney tumour in free-hand laparoscopic ultrasound sequences in near real-time. Our novel approach predicts the probability that a frame contains tumourous tissue using random forests and encodes this probability combined with a regularization term within a graph. Using Dijkstra's algorithm we find a globally optimal labelling (tumour vs. non-tumour) of each frame. We validate our method on a challenging clinical dataset composed of five patients, with a total of 2025 2D ultrasound frames, and demonstrate the ability to detect the presence of kidney tumour with a sensitivity and specificity of 0.774 and 0.916, respectively.

Synergistic fusion of pre-operative (pre-op) and intraoperative (intra-op) imaging data provides surgeons with invaluable insightful information that can improve their decision-making during minimally invasive robotic surgery. In this paper, we propose an efficient technique to segment multiple objects in intra-op multi-view endoscopic videos based on priors captured from pre-op data. Our approach leverages information from 3D pre-op data into the analysis of visual cues in the 2D intra-op data by formulating the problem as one of finding the 3D pose and non-rigid deformations of tissue models driven by features from 2D images. We present a closed-form solution for our formulation and demonstrate how it allows for the inclusion of laparoscopic camera motion model. Our efficient method runs in real-time on a single core CPU making it practical even for robotic surgery systems with limited computational resources. We validate the utility of our technique on ex vivo data as well as in vivo clinical data from laparoscopic partial nephrectomy surgery and demonstrate its robustness in segmenting stereo endoscopic videos.

Hilar dissection is an important and delicate stage in partial nephrectomy during which surgeons remove connective tissue surrounding renal vasculature. Potentially serious complications arise when vessels occluded by fat are missed in the endoscopic view and are not appropriately clamped. To aid in vessel discovery, we propose an automatic method to localize and label occluded vasculature. Our segmentation technique is adapted from phase-based video magnification, in which we measure subtle motion from periodic changes in local phase information albeit for labeling rather than magnification. We measure local phase through spatial decomposition of each frame of the endoscopic video using complex wavelet pairs. We then assign segmentation labels based on identifying responses of regions exhibiting temporal local phase changes matching the heart rate frequency. Our method is evaluated with a retrospective study of eight real robot-assisted partial nephrectomies demonstrating utility for surgical guidance that could potentially reduce operation times and complication rates.

Tumour identification is a critical step in robot-assisted partial nephrectomy (RAPN) during which the surgeon determines the tumour localization and resection margins. To help the surgeon in achieving this step, our research work aims at leveraging both pre- and intra-operative imaging modalities (CT, MRI, laparoscopic US, stereo endoscopic video) to provide an augmented reality view of kidney-tumour boundaries with uncertainty-encoded information. We present herein the progress of this research work including segmentation of preoperative scans, biomechanical simulation of deformations, stereo surface reconstruction from stereo endoscopic camera, pre-operative to intra-operative data registration, and augmented reality visualization.

Cerebral aneurysms are one of the prevalent and devastating cerebrovascular diseases of adult population worldwide. In most cases, it causes rupturing of cerebral vessels inside the brain, which leads to subarachnoid hemorrhage. In this work, we present a clinical workflow model to assist endovascular interventionists to select the type of stent-related treatment for cerebral aneurysms. The model pipeline includes data-acquisition, processing of aneurysm geometry and the simulation of blood flow. The preliminary results show the potential clinical value of the proposed computational workflow.

INTRODUCTION: Minimally-invasive robotic surgery benefits the surgeon with increased dexterity and precision, more comfortable seating, and depth perception. Indeed, the stereo-endoscopic camera of the daVinci robot provides the surgeon with a high-resolution 3D view of the surgical scene inside the patient body. To leverage this depth information using advanced computational tools (such as augmented reality or collision detection), we need a fast and accurate stereo matching algorithm, which computes the disparity (pixel shift) map between left and right images. To improve this trade-off between speed and accuracy, we propose an efficient multi-scale approach that overcomes standard multi-scale limitations due to interpolation artifacts when upsampling intermediate disparity results from coarser to finer scale. METHODS: Standard stereo matching algorithms perform an exhaustive search of the most similar patch between the reference and target images (along the same horizontal line when images are rectified). This requires a wide search range in the target image to ensure finding the corresponding pixel in the reference image (Figure 1). To optimize this search, we propose a multi-scale approach that uses the disparity map resulting from previous iteration at lower resolution. Instead of directly using the pixel position in the reference image to place the search region in the target image, we shift it by the corresponding disparity value from previous iteration and reduce the width of the search region as it is expected to be closer to the optimal solution. We also add two additional search regions shifted by disparity values at left and right adjacent pixel positions (Figure 2) to avoid errors typically related to interpolation artifacts when resizing disparity map. To avoid important overlaps between different search regions, we only add them where the disparity map has strong gradients. MATERIAL: We used stereo images from the Middlebury dataset (http://vision.middlebury.edu/stereo/data/) and stereo-endoscopic images captured at full HD 1080i resolution using a daVinci S/Si HD Surgical System. Experiments were performed with a GPU implementation on a workstation with 128GB RAM, an Intel Xeon Processor E5-2690, and an NVIDIA Tesla C2075. RESULTS: We compared the accuracy and speed between standard and proposed methods using ten images from the Middlebury dataset that has the advantage to provide ground truth disparity maps. We used the sum of square difference (SSD) as a similarity metric between patches of size 3x3 in left and right rectified images, resized to half their original size (665x555). For the standard method, we set the search range offset and width to respectively -25 and 64 pixels. For the proposed method, we initialize the disparity to 0 followed by five iterations with a search range width of 16. Results in Table 1 show that we managed to improve the average accuracy by 27% without affecting the average computation time of 120ms. CONCLUSION: We proposed an efficient multi-scale stereo matching algorithm that significantly improves accuracy without compromising speed. In future work, we will investigate the benefits of a similar approach using temporal consistency between successive frames and its use in more advanced computational tools for image-guided surgery.

Background & Objective: Blood vessel segmentation has various applications such as proper diagnosis, surgical planning, and simulation. However, the common challenges faced in blood vessel segmentation are mainly vessels of varying width and contrast changes. In this paper, we propose a segmentation algorithm, where: (1) a histogram-based approach is proposed to determine the initial patch (seeds) and (2) on this patch, a Gauss- Hermite quadrature filter is applied across different scales to handle vessels of varying width with high precision. Subsequently, a level set method is used to perform segmentation on the filter output. Methods: In spatial domain, a Gauss-Hermite quadrature filter is basically a complex filter pair, where the real component is a line filter that can detect linear structures, and the imaginary component is an edge filter that can detect edge structures; the filter pair is used for combined line-edge detection. The local phase is the argument of the complex filter response that determines the type of structure (line/edge), and the magnitude of the response determines the strength of the structure. Since the filter is applied in different directions, all filter responses are then combined to produce an orientation invariant phase map by summing filter responses for all directions. We use 6 filters with center frequency pi/2. To handle vessels of varying width, a multi-scale integration approach is implemented. Vessels of different width appear both as lines and edges across different scales. These scales are combined to produce a global phase map that is used for segmentation. The resulting global phase map contains detailed information about line and edge structures. For blood vessel segmentation, a local phase of 90 degree indicates edge structures. Therefore, it is necessary to consider only the real part of the quadrature filter response. Edges will be found at zero crossing whereas positive and negative values will be obtained for inside and outside of line structures. Therefore, level set (LS) approach is utilized that uses the real part of the phase map as a speed function to drive the deforming contour towards the vessel boundary. In this way, the blood vessel boundary gets extracted. An initial patch on the desired image object is a requirement in this algorithm to start calculating the local phase map. It is obtained by first selecting a few possible partitions using peaks (local maxima) in the intensity histogram. Then, optimal number of seeds is obtained by an iterative clustering of these peaks using their histogram heights and grey scale difference. The seeds around the object form the patch. Results & Conclusion: The proposed method has been tested on 6 subjects of Head MRT Angiography having resolution 416×512×112. We use 6 filters of size 7×7×7 and 4 scales in this experiment. The average time required by MATLAB R14 to perform segmentation is 3 m for one subject by a 2 GB RAM and core2duo processor (without optimization). The resulted segmentation is promising and robust in terms of boundary leakage as can be observed from the Figure.

INTRODUCTION: The advent of da Vinci surgical robot (Intuitive Surgical, California, USA) has allowed complex surgical procedures in urology, gynecology, cardiothoracic, and pediatric to be performed with better clinical outcomes. The end effectors of these robots exhibits enhanced dexterity with improved range of motion leading to better access and precise control during the surgery. Understanding the design and kinematics of these end effectors (imitating surgical instruments' tooltips) would assist in replication of their complex motion in a computer-generated environment. This would further support the development of computer-aided robotic surgical applications. The aim of this work is to develop a software framework comprising of the geometric three-dimensional models of the surgical robot tool-tips along with their kinematic analysis. METHODS: The geometric models of the surgical tooltips were designed based on the EndoWristTM instruments of the da Vinci surgical robot. Shapes of the link and inter-link distances of the EndoWristTM instrument were measured in detail. A three-dimensional virtual model was then recreated using CAD software (Solidworks, Dassault Systems, Massachusetts, USA). The kinematic analysis was performed considering trocar as the base-frame for actuation. The actuation mechanism of the tool composed of a prismatic joint (T1) followed by four revolute joints (Ri ; i = 1 to 4) in tandem (Figure 1). The relationship between the consequent joints was expressed in form of transformation matrices using Denavit-Hartenberg (D-H) convention. Equations corresponding to the forward and the inverse kinematics were then computed using D-H parameters and applying geometrical approach. The kinematics equations of the designed tooltips were implemented through a modular cross-platform software framework developed using C/C++. In the software, the graphical rendering was performed using openGL and a multi-threaded environment was implemented using Boost libraries. RESULTS AND DISCUSSION: Five geometric models simulating the articulated motion of the EndoWristTM instruments were designed (Figure 2). These models were selected based on the five basic interactions of the surgical tooltip with the anatomical structures, which included: Cauterization of the tissue, Stitching using needles, Applying clips on vascular structures, Cutting using scissors, and Grasping of the tissues. The developed software framework, which includes kinematics computation and graphical rendering of the designed components, was evaluated for applicability in two scenarios (Figure 3). The first scenario demonstrates the integration of the software with a patient-specific simulator for pre-operative surgical rehearsal and planning (Figure 3a). The second scenario shows the applicability of the software in generation of virtual overlays of the tooltips superimposed with the stereoscopic video stream and rendered on the surgeon's console of the surgical robot (Figure 3b). This would further assist in development of vision-based guidance for the tooltips. CONCLUSION: The geometrical modeling and kinematic analysis allowed the generation of the motion of the tooltips in a virtual space that could be used for both pre-operatively and intra-operatively, before and during the surgery, respectively. The resulting framework can also be used to simulate and test new tooltip designs.

INTRODUCTION: Robotic surgery allows minimally invasive procedures to be performed with greater precision, higher dexterity, and ergonomic comfort. The widely used daVinci surgical robot (Intuitive Surgical, California, USA) consists of a central stereoscopic camera and three robotic surgical arms controlled by the surgeon using a console. Though the stereoscopic camera provides superior visualization of the surgical site, it faces problems in certain surgical scenarios. These include visual problems with depth perception along view direction, occlusion by tissues, and low resolution at farther distance. One possible solution is to augment the understanding of the surgical site by addition of an extra visualization channel during the surgery. This could be achieved by inclusion of an additional camera probe. In this paper, we explore the preliminary design of an actuated probe with a camera alongside instruments to be used in a robotic surgery and demonstrate its functionality in three modes of operation. DESIGN METHODOLOGY: The probe consists of three tubular segments in tandem: telescopic arm, actuated spring, and camera (Figure-1). The probe is inserted along with the surgical instrument through a trocar. The design of the trocar is modified to have an additional insertion port alongside the instrument. Although this requires shifting of remote-centre-of-motion for the surgical-robot, it could be implemented in the robot control software as an additional feature without any change in the hardware. The telescopic arm allows insertion and retraction of the probe. The actuated spring is used to control the angulation of the probe. The angulation is achieved using a cable driven active system that combines pull and release action inside the spring. At the distal end of the probe, a camera is fixed to visualize the surgical site. Earlier prototypes used a straight camera that looked directly in front relative to the probe. This required two angulations in the spring: first to make the probe move away from the surgical instrument, and second to redirect the camera onto the surgical instrument. To simplify the mechanism while achieving the same results, we used an orthogonal camera in lieu of a straight camera. The video-stream captured through the camera is rendered to the surgeon's console. PRELIMINARY RESULTS: The preliminary design of the probe was implemented in CAD software. The probe design exhibited two-degree of freedom resulting in three modes of operation during the surgery (Figure-2). Mode 1: The 'insertion and retraction mode' would be used to insert and retract the tool. Mode 2: The 'endoscopic mode' would allow close visualization of the tool-tip (Figure-3a). Since this mode increases the field-of-view of the tissue to be operated, it could be use for surgical subtasks requiring higher level of precision such as clipping, stapling, or making a cut with vital tissues in the vicinity. Mode 3: The 'exploration mode' is used to explore hard-to-reach and occluded anatomies inside the patient's body, for example exploring through abdominal adhesion during a robotic abdominal surgery (Figure-3b). The future work will focus on fabrication of the probe and testing the modes in a clinical setting.

Clinically, segmentation has many benefits for effective patient management, both in terms of pre-operative planning and post-operative assessment. Volumetric image segmentation of medical data still remains as a major challenge, largely due to the complexities of in-vivo anatomical structures, cross-subject and cross-modality variations. This correspondence presents a semiautomatic segmentation algorithm that is based on graph and chaos theory. Also, we introduce a new weighting function in the method for accurate delineation of regions of interest in medical images that contain regional inhomogeneities; the preliminary results show the potential of the proposed technique.

Hilar dissection is an important and delicate stage in partial nephrectomy during which surgeons remove connective tissue surrounding renal vasculature. Potentially serious complications arise when vessels occluded by fat are missed in the endoscopic view and are not appropriately clamped. To aid in vessel discovery, we propose an automatic method to localize and label occluded vasculature. Our segmentation technique is adapted from phase-based video magnification, in which we measure subtle motion from periodic changes in local phase information albeit for labeling rather than magnification. We measure local phase through spatial decomposition of each frame of the endoscopic video using complex wavelet pairs. We then assign segmentation labels based on identifying responses of regions exhibiting temporal local phase changes matching the heart rate frequency. Our method is evaluated with a retrospective study of eight real robot-assisted partial nephrectomies demonstrating utility for surgical guidance that could potentially reduce operation times and complication rates.

Background & Objective: Cerebral aneurysms are one of the prevalent and devastating cerebrovascular diseases of adult population worldwide. The resulting effect is subarachnoid hemorrhage, intra- cerebral hematoma and other complications leading to a high mortality rate. When the aneurysm is fusiform, having wide neck or is large in shape, deploying stent in the parent artery to bypass aneurysm is considered as the most suitable treatment. The stent graft is designed to seal tightly with your artery above and below the aneurysm. The graft is stronger than the weakened artery and it allows your blood to pass through it without pushing on the bulge. So that blood cannot flow through the aneurysm to cause any future complication including rupture. Therefore, separation of parent artery from the aneurysm is immensely desired. This paper presents a method to separate parent artery from the aneurysm. Method: It has been challenging to distinguish the parent artery from the aneurysm geometry using a computer algorithm [1]. To date, only a few approaches to accomplish this task have been proposed. In our method, an initial surface mesh of the parent artery with aneurysm is first generated. Then the following steps are subsequently performed to separate the artery from the aneurysm. Step 1. The user specifies foreground and background on the mesh by placing centerline (Figure (a)) on the parent artery and aneurysm; it is also useful for generating Voronoi diagram (Figure (b) and (c)). Step 2. A feature preserving harmonic field based on the user specification is generated (Figure (d)). The resultant harmonic (i.e., "intensity") field over the artery geometry contains large variations not only at these concave and high curvature regions but also at the borders between the normal parent artery and the aneurysm. Since the parent artery centerline is nominally influenced by the presence of an aneurysm, the parent centerline is reconstructed; deviation is recorded and used while finalizing the average isoline or cutting boundary. Step 4. The isolines are generated with the help of Voronoi diagram (from which the average isoline is extracted); see Figure (e). We consider the isolines of the resultant harmonic field as the potential cutting boundary of the parent artery from the aneurysm. Along an isoline, the field variation is minimum. Step 5. A graph-based technique [2] is applied on the harmonic field to segment the parent artery from the aneurysm utilizing the average isoline and distance metric, where we define the energy function according to the harmonic field on the mesh. Results & Conclusion: For testing the method, we collected CTA slices with average thickness of 0.29mm, pixel spacing of 0.29mm x 0.29mm, and matrix size 512x512 on five subjects at the Hamad Medical Corporation using the Siemens Axiom Artis Interventional suite. The average time required by MATLAB R14 to perform segmentation is 2 m for one subject by a 2 GB RAM and core2duo processor (without optimization). Experimental results have shown satisfactory results for meshes with either simple or complicated model.

Synergistic fusion of pre-operative (pre-op) and intraoperative (intra-op) imaging data provides surgeons with invaluable insightful information that can improve their decision-making during minimally invasive robotic surgery. In this paper, we propose an efficient technique to segment multiple objects in intra-op multi-view endoscopic videos based on priors captured from pre-op data. Our approach leverages information from 3D pre-op data into the analysis of visual cues in the 2D intra-op data by formulating the problem as one of finding the 3D pose and non-rigid deformations of tissue models driven by features from 2D images. We present a closed-form solution for our formulation and demonstrate how it allows for the inclusion of laparoscopic camera motion model. Our efficient method runs in real-time on a single core CPU making it practical even for robotic surgery systems with limited computational resources. We validate the utility of our technique on ex vivo data as well as in vivo clinical data from laparoscopic partial nephrectomy surgery and demonstrate its robustness in segmenting stereo endoscopic videos.

Laparoscopic ultrasound (US) is often used during partial nephrectomy surgeries to identify tumour boundaries within the kidney. However, visual identification is challenging as tumour appearance varies across patients and US images exhibit significant noise levels. To address these challenges, we present the first fully automatic method for detecting the presence of kidney tumour in free-hand laparoscopic ultrasound sequences in near real-time. Our novel approach predicts the probability that a frame contains tumourous tissue using random forests and encodes this probability combined with a regularization term within a graph. Using Dijkstra’s algorithm we find a globally optimal labelling (tumour vs. non-tumour) of each frame. We validate our method on a challenging clinical dataset composed of five patients, with a total of 2025 2D ultrasound frames, and demonstrate the ability to detect the presence of kidney tumour with a sensitivity and specificity of 0.774 and 0.916, respectively.

Robotic assistance in minimally invasive surgical interventions has gained substantial popularity over the past decade. Surgeons perform such operations by remotely manipulating laparoscopic tools whose motion is executed by the surgical robot. One of the main tools deployed is an endoscopic binocular camera that provides stereoscopic vision of the operated scene. Such surgeries have notably garnered wide interest in renal surgeries such as partial nephrectomy, which is the focus of our work. This operation consists of the localization and removal of tumorous tissue in the kidney. During this procedure, the surgeon would greatly benefit from an augmented reality view that would display additional information from the different imaging modalities available, such as pre-operational CT and intra-operational ultrasound. In order to fuse and visualize these complementary data inputs in a pertinent way, they need to be accurately registered to a 3D reconstruction of the imaged surgical scene topology captured by the binocular camera. In this paper we propose a simple yet powerful approach for dense matching between the two stereoscopic camera views and for reconstruction of the 3D scene. Our method adaptively and accurately finds the optimal correspondence between each pair of images according to three strict confidence criteria that efficiently discard the majority of outliers. Using experiments on clinical in-vivo stereo data, including comparisons to two state-of-the-art 3D reconstruction techniques in minimally invasive surgery, our results illustrate superior robustness and better suitability of our approach to realistic surgical applications.

Introduction: Surgery remains one of the primary methods for terminating cancerous tumours. Minimally-invasive robotic surgery, in particular, provides several benefits, such as filtering of hand tremor, offering more complex and flexible manipulation capabilities that lead to increased dexterity and higher precisions, and more comfortable seating for the surgeon. All in turn lead to reduced blood loss, lower infection and complication rates, less post-operative pain, shorter hospital stays, and better overall surgical outcomes. Pre-operative 3D medical imaging modalities, mainly magnetic resonance imaging (MRI) and computed tomography (CT) are used for surgical planning, in which tumour excision margins are identified for maximal sparing of healthy tissue. However, transferring such plans from the pre-operative frame-of-reference to the dynamic intra-operative scene remains a necessary yet largely unsolved problem. We summarize our team's progress towards addressing this problem focusing on partial nephrectomy (RAPN) performed with a daVinci surgical robot. Method: We perform pre-operative 3D image segmentation of the tumour and surrounding healthy tissue using interactive random walker image segmentation, which provides an uncertainty-encoding segmentation used to construct a 3D model of the segmented patient anatomy. We reconstruct the 3D geometry of the surgical scene from the stereo endoscopic video, regularized by the patient-specific shape prior. We process the endoscopic images to detect tissue boundaries and other features. Then we align, first via rigid then via deformable registration, the pre-operative segmentation to the 3D reconstructed scene and the endoscopic image features. Finally, we present to the surgeon an augmented reality view showing an overlay of the tumour resection targets on top of the endoscopic view, in a way that depicts uncertainty in localizing the tumour boundary. Material: We collected pre-operative and intra-operative patient data in the context of RAPN including stereo endoscopic video at full HD 1080i (da Vinci S HD Surgical System), CT images (Siemens CT Sensation 16 and 64 slices), MR images (Siemens MRI Avanto 1.5T), and US images (Ultrasonix SonixTablet with a flexible laparoscopic linear probe). We also acquired CT images and stereo video from in-silico phantoms and ex-vivo lamb kidneys with artificial tumours for test and validation purposes. Results and Discussion: We successfully developed a novel proof-of-concept framework for prior and uncertainty encoded augmented reality system that fuses pre-operative patient specific information into the intra-operative surgical scene. Preliminary studies and initial surgeons' feedback on the developed augmented reality system are encouraging. Our future work will focus on investigating the use of intra-operative US data in our system to leverage all imaging modalities available during surgeries. Before a full system integration of these components, improving accuracy and speed of aforementioned algorithms, and the intuitiveness of the augmented reality visualization, remain active research projects for our team.