Search

Video streams are utilised to guide minimally-invasive surgery and diagnosis in a wide range of procedures, and many computer-assisted techniques have been developed to automatically analyse them. These approaches can provide additional information to the surgeon such as lesion detection, instrument navigation, or anatomy 3D shape modelling. However, the necessary image features to recognise these patterns are not always reliably detected due to the presence of irregular light patterns such as specular highlight reflections. In this paper, we aim at removing specular highlights from endoscopic videos using machine learning. We propose using a temporal generative adversarial network (GAN) to inpaint the hidden anatomy under specularities, inferring its appearance spatially and from neighbouring frames, where they are not present in the same location. This is achieved using in-vivo data from gastric endoscopy (Hyper Kvasir) in a fully unsupervised manner that relies on the automatic detection of specular highlights. System evaluations show significant improvements to other methods through direct comparison and ablation studies that depict the importance of the network's temporal and transfer learning components. The generalisability of our system to different surgical setups and procedures was also evaluated qualitatively on in-vivo data of gastric endoscopy and ex-vivo porcine data (SERV-CT, SCARED). We also assess the effect of our method in comparison to other methods on computer vision tasks that underpin 3D reconstruction and camera motion estimation, namely stereo disparity, optical flow, and sparse point feature matching. These are evaluated quantitatively and qualitatively and results show a positive effect of our specular inpainting method on these tasks in a novel comprehensive analysis. Our code and dataset are made available at https://github.com/endomapper/Endo-STTN., Competing Interests: Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Rema Daher reports financial support was provided by Engineering and Physical Sciences Research Council. Francisco Vasconcelos, Danail Stoyanov reports financial support was provided by Engineering and Physical Sciences Research Council. Francisco Vasconcelos, Danail Stoyanov reports financial support was provided by Wellcome Trust. Rema Daher reports financial support was provided by Wellcome Trust. Rema Daher reports financial support was provided by Horizon Europe. Rema Daher reports financial support was provided by Amazon Web Services, Inc. Danail Stoyanov reports a relationship with Odin Vision Ltd that includes: equity or stocks. Danail Stoyanov reports a relationship with Panda Surgical Ltd that includes: equity or stocks. Danail Stoyanov reports a relationship with Medtronic plc that includes: employment., (Copyright © 2023. Published by Elsevier B.V.)