Robust region encoding and layer attribute protection for the segmentation of retina with multifarious abnormalities.

Purpose: To robustly segment retinal layers that are affected by complex variety of retinal diseases for optical coherence tomography angiography (OCTA) en face projection generation. Methods: In this paper, we propose a robust retinal layer segmentation model to reduce the impact of multifarious abnormalities on model performance. OCTA vascular distribution that is regarded as the supplements of spectral domain optical coherence tomography (SD‐OCT) structural information is introduced to improve the robustness of layer region encoding. To further reduce the sensitivity of region encoding to retinal abnormalities, we propose a multitask layer‐wise refinement (MLR) module that can refine the initial layer region segmentation results layer‐by‐layer. Finally, we design a region‐to‐surface transformation (RtST) module without additional training parameters to convert the encoding layer regions to their corresponding layer surfaces. This transformation from layer regions to layer surfaces can remove the inaccurate segmentation regions, and the layer surfaces are easier to be used to protect the retinal layer natures than layer regions. Results: Experimental data includes 273 eyes, where 95 eyes are normal and 178 eyes contain complex retinal diseases, including age‐related macular degeneration (AMD), diabetic retinopathy (DR), central serous chorioretinopathy (CSC), choroidal neovascularization (CNV), and so forth. The dice similarity coefficient (DSC: %) of superficial, deep and outer retina achieves 98.92, 97.48, and 98.87 on normal eyes and 98.35, 95.33, and 98.17 on abnormal eyes. Compared with other commonly used layer segmentation models, our model achieves the state‐of‐the‐art layer segmentation performance. Conclusions: The final results prove that our proposed model obtains outstanding performance and has enough ability to resist retinal abnormalities. Besides, OCTA modality is helpful for retinal layer segmentation. [ABSTRACT FROM AUTHOR]