CSGO: A Deep Learning Pipeline for Whole-Cell Segmentation in Hematoxylin and Eosin Stained Tissues

Accurate whole-cell segmentation is essential in various biomedical applications, particularly in studying the tumor microenvironment (TME). Despite advancements in machine learning for nuclei segmentation in hematoxylin and eosin (H&E) stained images, there remains a need for effective whole-cell segmentation methods. This study aims to develop a deep learning-based pipeline to automatically segment cells in H&E-stained tissues, thereby advancing the capabilities of pathological image analysis. The Cell Segmentation with Globally Optimized boundaries (CSGO) framework integrates nuclei and membrane segmentation algorithms, followed by post-processing using an energy-based watershed method. Specifically, we employed the You Only Look Once (Yolo) object detection algorithm for nuclei segmentation and U-Net for membrane segmentation. The membrane detection model was trained on a dataset of 7 hepatocellular carcinomas and 11 normal liver tissue patches. The cell segmentation performance was extensively evaluated on five external datasets, including liver, lung, and oral disease cases. CSGO demonstrated superior performance over the state-of-the-art method Cellpose, achieving higher F1 scores ranging from 0.37 to 0.53 at an intersection over union (IoU) threshold of 0.5 in four of the five external datasets, compared to that of Cellpose from 0.21 to 0.36. These results underscore the robustness and accuracy of our approach in various tissue types. A web-based application is available at https://ai.swmed.edu/projects/csgo, providing a user-friendly platform for researchers to apply our method to their own datasets. Our method exhibits remarkable versatility in whole-cell segmentation across diverse cancer subtypes, serving as an accurate and reliable tool to facilitate TME studies. The advancements presented in this study have the potential to significantly enhance the precision and efficiency of pathological image analysis, contributing to better understanding and treatment of cancer.