Search

In image-guided surgery, imaging systems such as (stereo-)endoscopes allow intra-operative 3D reconstructions in form of point clouds. However, endoscopes provide only a narrow field of view, resulting in a confined point cloud. In this paper, we present an analysis and registration pipeline for confined point clouds acquired by stereo endoscopes into a fused representation. For a coarse registration, TEASER is applied, while a refinement is conducted utilizing point-to-plane ICP. The pipeline is tested on two datasets: acquired point clouds of a head phantom using a EinsteinVision® 3.0 endoscope and point clouds from the Stereo Correspondence And Reconstruction of Endoscopic Data challenge. The results for both datasets show that 3D reconstructions of anatomical structures by utilizing stereo-endoscopes and point cloud registration is a promising contactless and radiation-free method. However, non-rigid deformations are not yet incorporated and evaluation of the method compared to reference data is challenging.

We present a stereo-multispectral microscope equipped with an additional illumination unit allowing further narrow-band illumination in the spectral range of 400n.m up to 800nm. The combination of the normal microscope illumination with the multispectral light unit allows different illumination modalities to be realized, which enables intraoperative spectral tissue analysis with direct visualization. Two illumination methods were tested in two cholesteatoma surgeries. In addition, two cholesteatom samples were illuminated and analyzed ex vivo. Cholesteatoma showed :fluorescent characteristics in our ex vivo analysis. This behavior could be used intraoperatively using a combination of white light and strong near-UV to blue illumination to highlight cholesteatoma tissue in the microscopic image. Thus, the visual differentiability of different tissue types can be improved and the clinical decision-making process can be accelerated.

Existing challenges in surgical education (See one, do one, teach one) as well as the Covid-19 pandemic make it necessary to develop new ways for surgical training. This is also crucial for the dissemination of new technological developments. As today’s live transmissions of surgeries to remote locations always come with high information loss, e.g. stereoscopic depth perception, and limited communication channels. This work describes the implementation of a scalable remote solution for surgical training, called TeleSTAR (Telepresence for Surgical Assistance and Training using Augmented Reality), using immersive, interactive and augmented reality elements with a bi-lateral audio pipeline to foster direct communication. The system uses a full digital surgical microscope with a modular software-based AR interface, which consists of an interactive annotation mode to mark anatomical landmarks using an integrated touch panel as well as an intraoperative image-based stereo-spectral algorithm unit to measure anatomical details and highlight tissue characteristics.We broadcasted three cochlea implant surgeries in the context of otorhinolaryngology. The intervention scaled to five different remote locations in Germany and the Netherlands with lowlatency. In total, more than 150 persons could be reached and included an evaluation of a participant’s questionnaire indicating that annotated AR-based 3D live transmissions add an extra level of surgical transparency and improve the learning outcome.

Tracking of surgical instruments is an essential step towards the modernization of the surgical workflow by a comprehensive surgical landscape guidance system (COMPASS). Real-time tracking of a laparoscopic camera used in minimally-invasive surgery is required for applications in surgical workflow documentation, machine learning, image-localization, and intra-operative visualization. In our approach, an inertial measurement unit (IMU) assists the tool tracking in situations when no line-of-sight is available for infrared (IR) based tracking of the laparoscopic camera. The novelty of this approach lies in the localization method adjusted for the laparoscopic visceral surgery, particularly when the line-of-sight is lost. It is based on IMU tracking and the positioning of the trocar entry point. The trocar entry point is the remote center of motion (RCM), reducing degrees of freedom. We developed a method to tackle localization and a real-time tool for position and orientation estimation. The main error sources are given and evaluated in a test scenario. It reveals that for small changes in penetration length (e.g., pivoting), the IMU’s accuracy determines the error.

We present a stereo-multispectral endoscopic prototype using a filter-wheel to guide the removal of cholesteatoma tissue in the middle ear. An image-based method is used that combines multispectral tissue classification for the detection of tissue to be removed and 3Dreconstruction to determine its metric dimensions. The multispectral illumination used for tissue classification ranges from λ = 400 nm to λ = 500 nm with step-size of 20 nm, which results in six different narrow-band illumination modes. For classical RGB imaging and metric calculations, a broadband illumination mode is applied before and after the narrow-band illumination. The spectral information is augmented into the broadband mode using an overlay technique. The combination of multispectral imaging with stereoscopic 3D-reconstruction results in new valuable visualization of intraoperative data. This allows to generate a 3D-model of the patients anatomy highlighting the identified malicious tissue and compare the anatomical dimensions with pre-operative CT data.