Search

Limited or absent haptic feedback is reported as a factor hindering the continued adoption of surgical robots. This article presents a proof of concept for vibrotactile feedback integrated into a continuum robot to explore whether such feedback improves spatial perception in surgical settings. The robot is equipped with a capacitive sensor for noncontact endoscope localization, enabling spatial awareness of the robot’s tool center point (TCP) within the surgical environment. The data from the sensor is processed and transmitted to a bracelet worn by the user, which generates vibrotactile feedback. The bracelet contains four vibration motors providing tactile cues for navigation and positioning of the robot’s TCP. All subsystems are integrated into a unified system to deliver vibrotactile feedback to the user. When the user maneuvers the TCP of the robot near an object, they receive vibrotactile feedback via the bracelet. Thereby, the intensity of vibration increases as the TCP approaches the object, and the direction of the obstacle is mapped on the bracelet. Initial functional tests were performed and prove the functionality of the proposed system.

Automatic speech recognition (ASR), powered by deep learning techniques, is crucial for enhancing humancomputer interaction. However, its full potential remains unrealized in diverse real-world environments, with challenges such as dialects, accents, and domain-specific jargon, particularly in fields like surgery, persisting. Here, we investigate the potential of large language models (LLMs) as error correction modules for ASR.We leverage Whisper-medium or ASRLibriSpeech for speech recognition, and GPT-3.5 or GPT-4 for error correction.We employ various prompting methods, from zero-shot to few-shot with leading questions and sample medical terms to correct wrong transcriptions. Results, measured by word error rate (WER), reveal Whisper’s superior transcription accuracy over ASR-LibriSpeech, with a WER of 11.93% compared to 32.09%. GPT-3.5, with the few-shot with medical terms prompting method, further enhances performance, achieving a 64.29% and 37.83% WER-reduction for Whisper and ASR-LibriSpeech, respectively. Additionally, Whisper exhibits faster execution speed. Substituting GPT-3.5 with GPT- 4 further enhances transcription accuracy. Despite some few challenges, our approach demonstrates the potential of leveraging domain-specific knowledge through LLM prompting for accurate transcription, particularly in sophisticated domains like surgery.

Robotic assistance systems are being used more and more frequently in the operating room, with the goal to support surgeons and to automate parts of a procedure. The laparoscopic cholecystectomy is one of the most common procedures in Germany.We aim to automate the assistant grasp task in this procedure. To achieve this goal, first the grasp points on the gallbladder need to be determined. In this work, we therefore present a statistical shape model fitting to the gallbladder for grasp point determination. Gallbladder and liver point clouds are utilized as inputs. A registration algorithm is used to fit the shape model to the gallbladder mesh. The process is evaluated on three different datasets achieving a successful grasping point identification of 90% for artificially created gallbladders, 100% for our silicon phantom model, and 90% for ex-vivo organs.

Endotracheal intubation is a medical procedure to secure a patient’s airways, which can be challenging in emergency situations due to an individual’s anatomy. Intubation blades can benefit from an adjustable design with shape memory alloys. We present a novel blade design with two degrees of freedom, continuously transforming a straight into a curved blade. Our demonstrators reached mean angular deflections up to 16∘ without exceeding outer blade temperatures of 40∘C. Mean forces of almost 24N were applicable. This approach has the potential to reduce the complexity of intubation procedures and increase the success rate, especially in time-critical emergency situations.

Training and prototype development in robotassisted surgery requires appropriate and safe environments for the execution of surgical procedures. Current dry lab laparoscopy phantoms often lack the ability to mimic complex, interactive surgical tasks. This work presents an interactive surgical phantom for the cholecystectomy. The phantom enables the removal of the gallbladder during cholecystectomy by allowing manipulations and cutting interactions with the synthetic tissue. The force-displacement behavior of the gallbladder is modelled based on retraction demonstrations. The force model is compared to the force model of ex-vivo porcine gallbladders and evaluated on its ability to estimate retraction forces.

Machine learning-based solutions rely heavily on the quality and quantity of the training data. In the medical domain, the main challenge is to acquire rich and diverse annotated datasets for training. We propose to decrease the annotation efforts and further diversify the dataset by introducing an annotation-efficient learning workflow. Instead of costly pixel-level annotation, we require only image-level labels as the remainder is covered by simulation. Thus, we obtain a large-scale dataset with realistic images and accurate ground truth annotations. We use this dataset for the instrument localization activity task together with a studentteacher approach. We demonstrate the benefits of our workflow compared to state-of-the-art methods in instrument localization that are trained only on clinical datasets, which are fully annotated by human experts.

The manipulation of flexible endoscopes is a procedure that requires great dexterity since it requires the synchronization and use of both hands in parallel. Imprecise handling during gastroendoscopy could harm the digestive tract. Our solution allows the physician to use only one hand to wirelessly control the forward, backward, and tip bending motion. The proposed system provides endoscopic vision and tactile impact force sensing at the tip to detect the force applied to tissue and thus avoid damage. We experimentally evaluate the handling of the robotic system in open space and inside a medical phantom. The results revealed a training effect with less time demand for task completion and reduction of average impact force after only 5 runs. The proposed system was successfully controlled using one hand and, together with the force information, could enhance the physician’s experience during endoscopy. Future work will address axial control and an intensive user study with clinical experts.

Minimally invasive surgery is the standard for many abdominal interventions, with an increasing use of telemanipulated robots. As collaborative robots enter the field of medical interventions, their intuitive control needs to be addressed. Augmented reality can thereby support a surgeon by representing the surgical scene in a natural way. In this work, an augmented reality based robot control for laparoscopic cholecystectomy is presented. A user can interact with the virtual scene to clip the cystic duct and artery as well as to manipulate the deformable gallbladder. An evaluation was performed based on the SurgTLX and system usability scale.

Cardiovascular diseases are the main cause of death worldwide. State-of-the-art treatment often includes the process of navigating endovascular instruments through the vasculature. Automation of the procedure receives much attention lately and may increase treatment quality and unburden clinicians. However, in order to ensure the patient’s safety the endovascular device needs to be steered carefully through the body. In this work, we present a collection of medical criteria that are considered by physicians during an intervention and that can be evaluated automatically enabling a highly objective assessment. Additionally, we trained an autonomous controller with deep reinforcement learning to gently navigate within a 2D simulation of an aortic arch. Among others, the controller reduced the maximum and mean contact force applied to the vessel walls by 43% and 29%, respectively.

Intraoperative imaging can assist neurosurgeons to define brain tumours and other surrounding brain structures. Interventional ultrasound (iUS) is a convenient modality with fast scan times. However, iUS data may suffer from noise and artefacts which limit their interpretation during brain surgery. In this work, we use two deep learning networks, namely UNet and TransUNet, to make automatic and accurate segmentation of the brain tumour in iUS data. Experiments were conducted on a dataset of 27 iUS volumes. The outcomes show that using a transformer with UNet is advantageous providing an efficient segmentation modelling long-range dependencies between each iUS image. In particular, the enhanced TransUNet was able to predict cavity segmentation in iUS data with an inference rate of more than 125 FPS. These promising results suggest that deep learning networks can be successfully deployed to assist neurosurgeons in the operating room.

Purpose Computerized medical imaging processing assists neurosurgeons to localize tumours precisely. It plays a key role in recent image-guided neurosurgery. Hence, we developed a new open-source toolkit, namely Slicer-DeepSeg, for efficient and automatic brain tumour segmentation based on deep learning methodologies for aiding clinical brain research. Methods Our developed toolkit consists of three main components. First, Slicer-DeepSeg extends the 3D Slicer application and thus provides support for multiple data input/ output data formats and 3D visualization libraries. Second, Slicer core modules offer powerful image processing and analysis utilities. Third, the Slicer-DeepSeg extension provides a customized GUI for brain tumour segmentation using deep learning-based methods. Results The developed Slicer- DeepSeg was validated using a public dataset of high-grade glioma patients. The results showed that our proposed platform’s performance considerably outperforms other 3D Slicer cloud-based approaches. Conclusions Developed Slicer-DeepSeg allows the development of novel AIassisted medical applications in neurosurgery. Moreover, it can enhance the outcomes of computer-aided diagnosis of brain tumours. Open-source Slicer-DeepSeg is available at github.com/razeineldin/Slicer-DeepSeg.

Towards computer-assisted neurosurgery, scene understanding algorithms for microscope video data are required. Previous work utilizes optical flow to extract spatiotemporal context from neurosurgical video sequences. However, to select an appropriate optical flow method, we need to analyze which algorithm yields the highest accuracy for the neurosurgical domain. Currently, there are no benchmark datasets available for neurosurgery. In our work, we present an approach to generate synthetic data for optical flow evaluation on the neurosurgical domain. We simulate image sequences and thereby take into account domainspecific visual conditions such as surgical instrument motion. Then, we evaluate two optical flow algorithms, Farneback and PWC-Net, on our synthetic data. Qualitative and quantitative assessments confirm that our data can be used to evaluate optical flow for the neurosurgical domain. Future work will concentrate on extending the method by modeling additional effects in neurosurgery such as elastic background motion.

Robots from material-based actuators offer high potential for small-scale robots with abilities hardly achievable by classical methods like electric motors. Besides excellent scaling to minimally invasive systems, allowing for omission of metallic components, such robots can be applied in imaging modalities such as MRI or CT. To allow for higher accessibility in this field of research, a facile method for fabrication of such soft actuators was developed. It comprises only two materials: graphene oxide and silicone elastomer. The facile fabrication method does not require specialized equipment. The resulting actuator is biocompatible and controllable by light mediated heat. The bending motion can be controlled by the intensity of applied infrared light and the actuator was experimentally shown to move five times its own weight. Thus, providing capabilities for a medical soft robotic actuator.

Endovascular surgical robotics requires a facile and realistic testbed to validate control algorithms. This work compares methods to manufacture such a testbed. Utilizing animal tissue, a mannequin, and a low-voltage flow pump it is possible to perform catheter-based interventions with X-ray feedback for less than €300. The aim of this paper is to lower the entry hurdle for validating endovascular surgical robots by providing a method to build a facile and low-budget testbed.

Smart medical phantoms for training and evaluation of endovascular procedures ought to measure impact forces on the vessel walls worth protecting to provide feedback to clinicians and articulated soft robots. Recent commercial smart phantoms are expensive, usually not customizable to different applications and lack accessibility for integrated development. This work investigates piezoresistive films as highly integratable flexible sensors to be used in arbitrary soft vessel phantom anatomies over large surfaces and curved shapes providing quantitative measurement in the force range up to 1 N with 0.1 N resolution. First results show promising performance at the point of calibration and in a 5 mm range around it, with absolute measuring error of 28 mN and a standard deviation of ±10 mN and response times

Intraoperative brain deformation, so-called brain shift, affects the applicability of preoperative magnetic resonance imaging (MRI) data to assist the procedures of intraoperative ultrasound (iUS) guidance during neurosurgery. This paper proposes a deep learning-based approach for fast and accurate deformable registration of preoperative MRI to iUS images to correct brain shift. Based on the architecture of 3D convolutional neural networks, the proposed deep MRI-iUS registration method has been successfully tested and evaluated on the retrospective evaluation of cerebral tumors (RESECT) dataset. This study showed that our proposed method outperforms other registration methods in previous studies with an average mean squared error (MSE) of 85. Moreover, this method can register three 3D MRI-US pair in less than a second, improving the expected outcomes of brain surgery.

Patient tracking is an essential part in a surgical augmented reality system for correct hologram to patient registration. Augmented reality can support a surgeon with visual assistance to navigate more precisely during neurosurgical interventions. In this work, a system for patient tracking based on infrared markers is proposed. These markers are widely used in medical applications and meet the special medical requirements such as sterilizability. A tracking accuracy of 0.76 mm is achieved when using the near field reflectivity and depth sensor of the HoloLens. On the HoloLens a performance of 55–60 fps is reached, which grants a sufficiently stable placement of the holograms in the operating room.

Semantic segmentation of organs and tissue types is an important sub-problem in image based scene understanding for laparoscopic surgery and is a prerequisite for context-aware assistance and cognitive robotics. Deep Learning (DL) approaches are prominently applied to segmentation and tracking of laparoscopic instruments. This work compares different combinations of neural networks, loss functions, and training strategies in their application to semantic segmentation of different organs and tissue types in human laparoscopic images in order to investigate their applicability as components in cognitive systems. TernausNet-11 trained on Soft-Jaccard loss with a pretrained, trainable encoder performs best in regard to segmentation quality (78.31% mean Intersection over Union [IoU]) and inference time (28.07 ms) on a single GTX 1070 GPU.

The treatment of cerebro- and cardiovascular diseases requires complex and challenging navigation of a catheter. Previous attempts to automate catheter navigation lack the ability to be generalizable. Methods of Deep Reinforcement Learning show promising results and may be the key to automate catheter navigation through the tortuous vascular tree. This work investigates Deep Reinforcement Learning for guidewire manipulation in a complex and rigid vascular model in 2D. The neural network trained by Deep Deterministic Policy Gradients with Hindsight Experience Replay performs well on the low-level control task, however the high-level control of the path planning must be improved further.