Search

Robot-assisted surgery is rapidly developing in the medical field, and the integration of augmented reality shows the potential to improve the operation performance of surgeons by providing more visual information. In this paper, we proposed a markerless augmented reality framework to enhance safety by avoiding intra-operative bleeding, which is a high risk caused by collision between surgical instruments and delicate blood vessels (arteries or veins). Advanced stereo reconstruction and segmentation networks are compared to find the best combination to reconstruct the intra-operative blood vessel in 3D space for registration with the pre-operative model, and the minimum distance detection between the instruments and the blood vessel is implemented. A robot-assisted lymphadenectomy is emulated on the da Vinci Research Kit in a dry lab, and ten human subjects perform this operation to explore the usability of the proposed framework. The result shows that the augmented reality framework can help the users to avoid the dangerous collision between the instruments and the delicate blood vessel while not introducing an extra load. It provides a flexible framework that integrates augmented reality into the medical robotic platform to enhance safety during surgery., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 The Author(s). Published by Elsevier Ltd.. All rights reserved.)

Background and Objective: Safety of robotic surgery can be enhanced through augmented vision or artificial constraints to the robotl motion, and intra-operative depth estimation is the cornerstone of these applications because it provides precise position information of surgical scenes in 3D space. High-quality depth estimation of endoscopic scenes has been a valuable issue, and the development of deep learning provides more possibility and potential to address this issue., Methods: In this paper, a deep learning-based approach is proposed to recover 3D information of intra-operative scenes. To this aim, a fully 3D encoder-decoder network integrating spatio-temporal layers is designed, and it adopts hierarchical prediction and progressive learning to enhance prediction accuracy and shorten training time., Results: Our network gets the depth estimation accuracy of MAE 2.55±1.51 (mm) and RMSE 5.23±1.40 (mm) using 8 surgical videos with a resolution of 1280×1024, which performs better compared with six other state-of-the-art methods that were trained on the same data., Conclusions: Our network can implement a promising depth estimation performance in intra-operative scenes using stereo images, allowing the integration in robot-assisted surgery to enhance safety., Competing Interests: Declaration of Competing Interest The authors declare no conflict of interest relevant to this work., (Copyright © 2023 Elsevier B.V. All rights reserved.)

Robotic rehabilitation has demonstrated slight positive effects compared to traditional care, but there is still a lack of targeted high-level control strategies in the current state-of-the-art for minimizing pathological motor behaviors. In this study, we analyzed upper-limb motion capture data from healthy subjects performing a pick-and-place task to identify task-specific variability in postural patterns. The results revealed consistent behaviors among subjects, presenting an opportunity to develop a novel extraction method for variable volume references based solely on observations from healthy individuals. These human-centered references were tested on a simulated 4 degrees-of-freedom upper-limb exoskeleton, showing its compliant adaptation to the path considering the variance in healthy subjects' motor behavior.

Purpose: Advanced developments in the medical field have gradually increased the public demand for surgical skill evaluation. However, this assessment always depends on the direct observation of experienced surgeons, which is time-consuming and variable. The introduction of robot-assisted surgery provides a new possibility for this evaluation paradigm. This paper aims at evaluating surgeon performance automatically with novel evaluation metrics based on different surgical data., Methods: Urologists ([Formula: see text]) from a hospital were requested to perform a simplified neobladder reconstruction on an ex vivo setup twice with different camera modalities ([Formula: see text]) randomly. They were divided into novices and experts ([Formula: see text], respectively) according to their experience in robot-assisted surgeries. Different performance metrics ([Formula: see text]) are proposed to achieve the surgical skill evaluation, considering both instruments and endoscope. Also, nonparametric tests are adopted to check if there are significant differences when evaluating surgeons performance., Results: When grouping according to four stages of neobladder reconstruction, statistically significant differences can be appreciated in phase 1 ([Formula: see text]) and phase 2 ([Formula: see text]) with normalized time-related metrics and camera movement-related metrics, respectively. On the other hand, considering experience grouping shows that both metrics are able to highlight statistically significant differences between novice and expert performances in the control protocol. It also shows that the camera-related performance of experts is significantly different ([Formula: see text]) when handling the endoscope manually and when it is automatic., Conclusion: Surgical skill evaluation, using the approach in this paper, can effectively measure surgical procedures of surgeons with different experience. Preliminary results demonstrate that different surgical data can be fully utilized to improve the reliability of surgical evaluation. It also demonstrates its versatility and potential in the quantitative assessment of various surgical operations., (© 2022. The Author(s).)

A major challenge during autonomous navigation in endovascular interventions is the complexity of operating in a deformable but constrained workspace with an instrument. Simulation of deformations for it can provide a cost-effective training platform for path planning. Aim of this study is to develop a realistic, auto-adaptive, and visually plausible simulator to predict vessels' global deformation induced by the robotic catheter's contact and cyclic heartbeat motion. Based on a Position-based Dynamics (PBD) approach for vessel modeling, Particle Swarm Optimization (PSO) algorithm is employed for an auto-adaptive calibration of PBD deformation parameters and of the vessels movement due to a heartbeat. In-vitro experiments were conducted and compared with in-silico results. The end-user evaluation results were reported through quantitative performance metrics and a 5-Point Likert Scale questionnaire. Compared with literature, this simulator has an error of 0.23±0.13% for deformation and 0.30±0.85mm for the aortic root displacement. In-vitro experiments show an error of 1.35±1.38mm for deformation prediction. The end-user evaluation results show that novices are more accustomed to using joystick controllers, and cardiologists are more satisfied with the visual authenticity. The real-time and accurate performance of the simulator make this framework suitable for creating a dynamic environment for autonomous navigation of robotic catheters., Competing Interests: Declaration of Competing Interest The authors have no conflicts of interest to disclose., (Copyright © 2022. Published by Elsevier Ltd.)

Purpose: Automation of sub-tasks during robotic surgery is challenging due to the high variability of the surgical scenes intra- and inter-patients. For example, the pick and place task can be executed different times during the same operation and for distinct purposes. Hence, designing automation solutions that can generalise a skill over different contexts becomes hard. All the experiments are conducted using the Pneumatic Attachable Flexible (PAF) rail, a novel surgical tool designed for robotic-assisted intraoperative organ manipulation., Methods: We build upon previous open-source surgical Reinforcement Learning (RL) training environment to develop a new RL framework for manipulation skills, rlman. In rlman, contextual RL agents are trained to solve different aspects of the pick and place task using the PAF rail system. rlman is implemented to support both low- and high-dimensional state information to solve surgical sub-tasks in a simulation environment., Results: We use rlman to train state of the art RL agents to solve four different surgical sub-tasks involving manipulation skills using the PAF rail. We compare the results with state-of-the-art benchmarks found in the literature. We evaluate the ability of the agent to be able to generalise over different aspects of the targeted surgical environment., Conclusion: We have shown that the rlman framework can support the training of different RL algorithms for solving surgical sub-task, analysing the importance of context information for generalisation capabilities. We are aiming to deploy the trained policy on the real da Vinci using the dVRK and show that the generalisation of the trained policy can be transferred to the real world., (© 2022. The Author(s).)

An autonomous robotic laparoscopic surgical technique is capable of tracking tissue motion and offers consistency in suturing for the anastomosis of the small bowel.

Purpose: Both safety and accuracy are of vital importance for surgical operation procedures. An efficient way to avoid the singularity of the surgical robot concerning safety issues is to maximize its manipulability in robot-assisted surgery. The goal of this work was to validate a dynamic neural network optimization method for manipulability optimization control of a 7-degree of freedom (DoF) robot in a surgical operation., Methods: Three different paths, a circle, a sinusoid and a spiral were chosen to simulate typical surgical tasks. The dynamic neural network-based manipulability optimization control was implemented on a 7-DoF robot manipulator. During the surgical operation procedures, the manipulability of the robot manipulator and the accuracy of the surgical operation are recorded for performance validation., Results: By comparison, the dynamic neural network-based manipulability optimization control achieved optimized manipulability but with a loss of the accuracy of trajectory tracking (the global error was 1 mm compare to the 0.5 mm error of non-optimized method)., Conclusions: The method validated in this work achieved optimized manipulability with a loss of error. Future works should be introduced to improve the accuracy of the surgical operation., (© 2020 John Wiley & Sons Ltd.)

Objective: Virtual Reality (VR) simulators represent a remarkable educational opportunity in order to acquire and refine surgical practical skills. Nevertheless, there exists no consensus regarding a standard curriculum of simulation-based training. This study introduces an automatic, adaptive curriculum where the training session is real-time scheduled on the basis of the trainee's performances., Methods: An experimental study using the master console of the da Vinci Research Kit (Intuitive Surgical Inc., Sunnyvale, US) was carried out to test this approach. Tasks involving fundamental skills of robotic surgery were designed and simulated in VR. Twelve participants without medical background along with twelve medical residents were randomly and equally divided into two groups: a control group, self-managing the training session, and an experimental group, undergoing the proposed adaptive training., Results: The performances of the experimental users were significantly better with respect to the ones of the control group after training (non-medical: p < 0.01; medical: p = 0.02). This trend was analogous in the non-medical and medical populations and no significant difference was identified between these two classes (even in the baseline assessment)., Conclusion: The analysis of the learning of the involved surgical skills highlighted how the proposed adaptive training managed to better identify and compensate for the trainee's gaps. The absence of initial difference between the non-medical and medical users underlines that robotic surgical devices require specific training before clinical practice., Significance: This feasibility study could pave the way towards the improvement of simulation-based training curricula.

In this paper, an improved recurrent neural network (RNN) scheme is proposed to perform the trajectory control of redundant robot manipulators using remote center of motion (RCM) constraints. Firstly, learning by demonstration is implemented to model the surgical operation skills in the Cartesian space. After that, considering the kinematic constraints associated with the optimization control of redundant manipulators, we propose a novel RNN-based approach to facilitate accurate task tracking based on the general quadratic performance index, which includes managing the constraints on RCM joint angle, and joint velocity, simultaneously. The results of the conducted theoretical analysis confirm that the RCM constraint has been established successfully, and accordingly. The corresponding end-effector tracking errors asymptotically converge to zero. Finally, demonstration experiments are conducted in a laboratory setup environment using KUKA LWR4+ to validate the effectiveness of the proposed control strategy., Competing Interests: Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2020 Elsevier Ltd. All rights reserved.)

Purpose: Surgical workflow recognition and context-aware systems could allow better decision making and surgical planning by providing the focused information, which may eventually enhance surgical outcomes. While current developments in computer-assisted surgical systems are mostly focused on recognizing surgical phases, they lack recognition of surgical workflow sequence and other contextual element, e.g., "Instruments." Our study proposes a hybrid approach, i.e., using deep learning and knowledge representation, to facilitate recognition of the surgical workflow., Methods: We implemented "Deep-Onto" network, which is an ensemble of deep learning models and knowledge management tools, ontology and production rules. As a prototypical scenario, we chose robot-assisted partial nephrectomy (RAPN). We annotated RAPN videos with surgical entities, e.g., "Step" and so forth. We performed different experiments, including the inter-subject variability, to recognize surgical steps. The corresponding subsequent steps along with other surgical contexts, i.e., "Actions," "Phase" and "Instruments," were also recognized., Results: The system was able to recognize 10 RAPN steps with the prevalence-weighted macro-average (PWMA) recall of 0.83, PWMA precision of 0.74, PWMA F1 score of 0.76, and the accuracy of 74.29% on 9 videos of RAPN., Conclusion: We found that the combined use of deep learning and knowledge representation techniques is a promising approach for the multi-level recognition of RAPN surgical workflow.

Training with simulation systems has become a primary alternative for learning the fundamental skills of robotic surgery. However, there exists no consensus regarding a standard training curriculum: sessions defined a priori by expert trainers or self-directed by the trainees feature lack of consistency. This study proposes an adaptive approach that structures the curriculum on the basis of an objective assessment of the trainee's performance. The work comprised an experimental session with 12 participants performing training on virtual reality tasks with the da Vinci Research Kit surgical console. Half of the subjects self-managed their training session, while the others underwent the adaptive training. The final performance of the latter trainees was found to be higher compared to the former (p=0.002), showing how outcome-based, dynamic designs could constitute a promising advance in robotic surgical training.

Despite the benefits introduced by robotic systems in abdominal Minimally Invasive Surgery (MIS), major complications can still affect the outcome of the procedure, such as intra-operative bleeding. One of the causes is attributed to accidental damages to arteries or veins by the surgical tools, and some of the possible risk factors are related to the lack of sub-surface visibilty. Assistive tools guiding the surgical gestures to prevent these kind of injuries would represent a relevant step towards safer clinical procedures. However, it is still challenging to develop computer vision systems able to fulfill the main requirements: (i) long term robustness, (ii) adaptation to environment/object variation and (iii) real time processing. The purpose of this paper is to develop computer vision algorithms to robustly track soft tissue areas (Safety Area, SA), defined intra-operatively by the surgeon based on the real-time endoscopic images, or registered from a pre-operative surgical plan. We propose a framework to combine an optical flow algorithm with a tracking-by-detection approach in order to be robust against failures caused by: (i) partial occlusion, (ii) total occlusion, (iii) SA out of the field of view, (iv) deformation, (v) illumination changes, (vi) abrupt camera motion, (vii), blur and (viii) smoke. A Bayesian inference-based approach is used to detect the failure of the tracker, based on online context information. A Model Update Strategy (MUpS) is also proposed to improve the SA re-detection after failures, taking into account the changes of appearance of the SA model due to contact with instruments or image noise. The performance of the algorithm was assessed on two datasets, representing ex-vivo organs and in-vivo surgical scenarios. Results show that the proposed framework, enhanced with MUpS, is capable of maintain high tracking performance for extended periods of time ( ≃ 4 min - containing the aforementioned events) with high precision (0.7) and recall (0.8) values, and with a recovery time after a failure between 1 and 8 frames in the worst case., (Copyright © 2017 Elsevier B.V. All rights reserved.)

Background: Scene supervision is a major tool to make medical robots safer and more intuitive. The paper shows an approach to efficiently use 3D cameras within the surgical operating room to enable for safe human robot interaction and action perception. Additionally the presented approach aims to make 3D camera-based scene supervision more reliable and accurate., Methods: A camera system composed of multiple Kinect and time-of-flight cameras has been designed, implemented and calibrated. Calibration and object detection as well as people tracking methods have been designed and evaluated., Results: The camera system shows a good registration accuracy of 0.05 m. The tracking of humans is reliable and accurate and has been evaluated in an experimental setup using operating clothing. The robot detection shows an error of around 0.04 m., Conclusions: The robustness and accuracy of the approach allow for an integration into modern operating room. The data output can be used directly for situation and workflow detection as well as collision avoidance.

Robotic surgery is transforming the current surgical practice, not only by improving the conventional surgical methods but also by introducing innovative robot-enhanced approaches that broaden the capabilities of clinicians. Being mainly of man-machine collaborative type, surgical robots are seen as media that transfer pre- and intraoperative information to the operator and reproduce his/her motion, with appropriate filtering, scaling, or limitation, to physically interact with the patient. The field, however, is far from maturity and, more critically, is still a subject of controversy in medical communities. Limited or absent haptic feedback is reputed to be among reasons that impede further spread of surgical robots. In this paper, objectives and challenges of deploying haptic technologies in surgical robotics are discussed, and a systematic review is performed on works that have studied the effects of providing haptic information to the users in major branches of robotic surgery. It attempts to encompass both classical works and the state-of-the-art approaches, aiming at delivering a comprehensive and balanced survey both for researchers starting their work in this field and for the experts.

During hands-on cooperative surgery, the use of a redundant robot allows to address encumbrance issues in the Operating Room (OR), which can occur due to the presence of large medical instrumentation, such as the surgical microscope. This work presents a new Null Space Optimization (NSO) strategy to constraint the position of the manipulator's elbow within predefined range of motions, according to the spatial requirements of the specific procedure, also taking into account the physical joint limits of the robotic assistant. The proposed strategy was applied to the 7 degrees of freedom (dof) lightweight robot LWR4+. The performance of the NSO was compared to two state-of-the-art null space optimization strategies, i.e. damped posture and fixed optimal posture, over a pool of three non-expert users in both strict (20deg) and negligible (100deg) angular encumbrance limitations. The NSO strategy was proved versatile in providing wide elbow mobility together with safe distance from relevant continuity null space boundaries, guaranteeing smooth guidance trajectories. Future works would be performed in order to evaluate the potential feasibility of NSO in a real surgical scenario.

Micron is a fully handheld active micromanipulator that helps to improve position accuracy and precision in microsurgery by cancelling hand tremor. This work describes adaptation, tuning, and testing of the Micron system for stapedotomy, a microsurgical procedure performed in the middle ear to restore hearing that requires accurate manipulation in narrow spaces. Two end-effectors, a handle, and a brace (or rest) were designed and prototyped. The control system was adapted for the new hardware. The system was tested ex vivo in stapedotomy procedure comparing manually-performed and Micron-assisted surgical tasks. Tremor amplitude was found to be reduced significantly. Further testing is needed in order to obtain statistically significant results regarding other parameters dealing with regularity of the fenestra shape.

The research of stereotactic apparatus to guide surgical devices began in 1908, yet a major part of today's stereotactic neurosurgeries still rely on stereotactic frames developed almost half a century ago. Robots excel at handling spatial information, and are, thus, obvious candidates in the guidance of instrumentation along precisely planned trajectories. In this review, we introduce the concept of stereotaxy and describe a standard stereotactic neurosurgery. Neurosurgeons' expectations and demands regarding the role of robots as assistive tools are also addressed. We list the most successful robotic systems developed specifically for or capable of executing stereotactic neurosurgery. A critical review is presented for each robotic system, emphasizing the differences between them and detailing positive features and drawbacks. An analysis of the listed robotic system features is also undertaken, in the context of robotic application in stereotactic neurosurgery. Finally, we discuss the current perspective, and future directions of a robotic technology in this field. All robotic systems follow a very similar and structured workflow despite the technical differences that set them apart. No system unequivocally stands out as an absolute best. The trend of technological progress is pointing toward the development of miniaturized cost-effective solutions with more intuitive interfaces.

During hands-on robotic surgery it is advisable to know how and when to provide the surgeon with different assistance levels with respect to the current performed activity. Gesteme-based on-line classification requires the definition of a complete set of primitives and the observation of large signal percentage. In this work an on-line, gesteme-free activity recognition method is addressed. The algorithm models the guidance forces and the resulting trajectory of the manipulator with 26 low-level components of a Gaussian Mixture Model (GMM). Temporal switching among the components is modeled with a Hidden Markov Model (HMM). Tests are performed in a simplified scenario over a pool of 5 non-surgeon users. Classification accuracy resulted higher than 89% after the observation of a 300 ms-long signal. Future work will address the use of the current detected activity to on-line trigger different strategies to control the manipulator and adapt the level of assistance.

Background: Force feedback in robotic minimally invasive surgery allows the human operator to manipulate tissues as if his/her hands were in contact with the patient organs. A force sensor mounted on the probe raises problems with sterilization of the overall surgical tool. Also, the use of off-axis gauges introduces a moment that increases the friction force on the bearing, which can easily mask off the signal, given the small force to be measured., Methods: This work aims at designing and testing two methods for estimating the resistance to the advancement (force) experienced by a standard probe for brain biopsies within a brain-like material. The further goal is to provide a neurosurgeon using a master-slave tele-operated driver with direct feedback on the tissue mechanical characteristics. Two possible sensing methods, in-axis strain gauge force sensor and position-position error (control-based method), were implemented and tested, both aimed at device miniaturization. The analysis carried out was aimed at fulfilment of the psychophysics requirements for force detection and delay tolerance, also taking into account safety, which is directly related to the last two issues. Controller parameters definition is addressed and consideration is given to development of the device with integration of a haptic interface., Results: Results show better performance of the control-based method (RMSE < 0.1 N), which is also best for reliability, sterilizability, and material dimensions for the application addressed., Conclusions: The control-based method developed for force estimation is compatible with the neurosurgical application and is also capable of measuring tissue resistance without any additional sensors. Force feedback in minimally invasive surgery allows the human operator to manipulate tissues as if his/her hands were in contact with the patient organs., (Copyright © 2011 John Wiley & Sons, Ltd.)