Search

Pathology images of histopathology can be acquired from camera-mounted microscopes or whole-slide scanners. Utilizing similarity calculations to match patients based on these images holds significant potential in research and clinical contexts. Recent advancements in search technologies allow for implicit quantification of tissue morphology across diverse primary sites, facilitating comparisons, and enabling inferences about diagnosis, and potentially prognosis, and predictions for new patients when compared against a curated database of diagnosed and treated cases. In this article, we comprehensively review the latest developments in image search technologies for histopathology, offering a concise overview tailored for computational pathology researchers seeking effective, fast, and efficient image search methods in their work.

Objective: To assess the performance of the current foundation models in histopathology. Patients and Methods: The assessment involves a comprehensive evaluation of some foundation models, such as the CLIP derivatives, namely PLIP and BiomedCLIP, which were fine-tuned on data scraped from the internet. The comparison is performed against simpler and nonfoundational histology models that are trained on well-curated data, eg, the cancer genome atlas. All models are evaluated on 8 datasets, 4 of which are internal histology datasets collected and curated at Mayo Clinic, and 4 well-known public datasets: PANDA, BRACS, CAMELYON16, and DigestPath. Evaluation metrics include accuracy and macro-averaged F1 score, using a majority vote among top-k (eg, MV@5) at the whole slide image/patch levels. Moreover, all models are evaluated in classification settings. This detailed analysis allows for a deep understanding of each model’s performance across various datasets. Results: In various evaluation tasks, domain-specific (and nonfoundational) models like DinoSSLPath and KimiaNet outperform general-purpose foundation models. The DinoSSLPath excels in whole slide image-level retrieval for internal colorectal cancer and liver datasets with MV@5 macro-averaged F1 scores of 63% and 74%, respectively. The KimiaNet leads in breast and skin cancer tasks with respective Top-1 and MV@5 scores of 56% and 70%, respectively and scores 75% on the public CAMELYON16 dataset. Similar trends are observed in patch-level metrics, highlighting the advantage of using specialized datasets like the cancer genome atlas for histopathological analysis. Conclusion: To enable effective vision-language foundation models in biomedicine, high-quality, multi-modal medical datasets are essential. These datasets serve as the substrate for training models capable of translating research into clinical practice. Of importance, the alignment (correspondence) between textual and visual data—often diagnostic—is critical and requires validation by domain experts. Thus, advancing foundation models in this field necessitates collaborative efforts in data curation and validation.

Abstract Appearing traces of bias in deep networks is a serious reliability issue which can play a significant role in ethics and generalization related concerns. Recent studies report that the deep features extracted from the histopathology images of The Cancer Genome Atlas (TCGA), the largest publicly available archive, are surprisingly able to accurately classify the whole slide images (WSIs) based on their acquisition site while these features are extracted to primarily discriminate cancer types. This is clear evidence that the utilized Deep Neural Networks (DNNs) unexpectedly detect the specific patterns of the source site, i.e, the hospital of origin, rather than histomorphologic patterns, a biased behavior resulting in degraded trust and generalization. This observation motivated us to propose a method to alleviate the destructive impact of hospital bias through a novel feature selection process. To this effect, we have proposed an evolutionary strategy to select a small set of optimal features to not only accurately represent the histological patterns of tissue samples but also to eliminate the features contributing to internal bias toward the institution. The defined objective function for an optimal subset selection of features is to minimize the accuracy of the model to classify the source institutions which is basically defined as a bias indicator. By the conducted experiments, the selected features extracted by the state-of-the-art network trained on TCGA images (i.e., the KimiaNet), considerably decreased the institutional bias, while improving the quality of features to discriminate the cancer types. In addition, the selected features could significantly improve the results of external validation compared to the entire set of features which has been negatively affected by bias. The proposed scheme is a model-independent approach which can be employed when it is possible to define a bias indicator as a participating objective in a feature selection process; even with unknown bias sources.

Deep models for cell detection have demonstrated utility in bone marrow cytology, showing impressive results in terms of accuracy and computational efficiency. However, these models have yet to be implemented in the clinical diagnostic workflow. Additionally, the metrics used to evaluate cell detection models are not necessarily aligned with clinical goals and targets. In order to address these issues, we introduce novel, automatically generated visual summaries of bone marrow aspirate specimens called cell projection plots (CPPs). Encompassing relevant biological patterns such as neutrophil maturation, CPPs provide a compact summary of bone marrow aspirate cytology. To gauge clinical relevance, CPPs were inspected by 3 hematopathologists, who decided whether corresponding diagnostic synopses matched with generated CPPs. Pathologists were able to match CPPs to the correct synopsis with a matching degree of 85%. Our finding suggests CPPs can represent clinically relevant information from bone marrow aspirate specimens and may be used to efficiently summarize bone marrow cytology to pathologists. CPPs could be a step toward human-centered implementation of artificial intelligence (AI) in hematopathology, and a basis for a diagnostic-support tool for digital pathology workflows.

Image analysis in digital pathology has proven to be one of the most challenging fields in medical imaging for AI-driven classification and search tasks. Due to their gigapixel dimensions, whole slide images (WSIs) are difficult to represent for computational pathology. Self-supervised learning (SSL) has recently demonstrated excellent performance in learning effective representations on pretext objectives, which may improve the generalizations of downstream tasks. Previous self-supervised representation methods rely on patch selection and classification such that the effect of SSL on end-to-end WSI representation is not investigated. In contrast to existing augmentation-based SSL methods, this paper proposes a novel self-supervised learning scheme based on the available primary site information. We also design a fully supervised contrastive learning setup to increase the robustness of the representations for WSI classification and search for both pretext and downstream tasks. We trained and evaluated the model on more than 6000 WSIs from The Cancer Genome Atlas (TCGA) repository provided by the National Cancer Institute. The proposed architecture achieved excellent results on most primary sites and cancer subtypes. We also achieved the best result on validation on a lung cancer classification task.