Search

Artificial intelligence (AI) has extensive applications in a wide range of disciplines including healthcare and clinical practice. Advances in high-resolution whole-slide brightfield microscopy allow for the digitization of histologically stained tissue sections, producing gigapixel-scale whole-slide images (WSI). The significant improvement in computing and revolution of deep neural network (DNN)-based AI technologies over the last decade allow us to integrate massively parallelized computational power, cutting-edge AI algorithms, and big data storage, management, and processing. Applied to WSIs, AI has created opportunities for improved disease diagnostics and prognostics with the ultimate goal of enhancing precision medicine and resulting patient care. The National Institutes of Health (NIH) has recognized the importance of developing standardized principles for data management and discovery for the advancement of science and proposed the Findable, Accessible, Interoperable, Reusable, (FAIR) Data Principles 1 with the goal of building a modernized biomedical data resource ecosystem to establish collaborative research communities. In line with this mission and to democratize AI-based image analysis in digital pathology, we propose ComPRePS: an end-to-end automated Comp utational Re nal P athology S uite which combines massive scalability, on-demand cloud computing, and an easy-to-use web-based user interface for data upload, storage, management, slide-level visualization, and domain expert interaction. Moreover, our platform is equipped with both in-house and collaborator developed sophisticated AI algorithms in the back-end server for image analysis to identify clinically relevant micro-anatomic functional tissue units (FTU) and to extract image features.

Segmentation of histology tissue whole side images is an important step for tissue analysis. Given enough annotated training data, modern neural networks are capable of accurate reproducible segmentation; however, the annotation of training datasets is time consuming. Techniques such as human-in-the-loop annotation attempt to reduce this annotation burden, but still require vast initial annotation. Semi-supervised learning-a technique which leverages both labeled and unlabeled data to learn features-has shown promise for easing the burden of annotation. Towards this goal, we employ a recently published semi-supervised method, datasetGAN, for the segmentation of glomeruli from renal biopsy images. We compare the performance of models trained using datasetGAN and traditional annotation and show that datasetGAN significantly reduces the amount of annotation required to develop a highly performing segmentation model. We also explore the usefulness of datasetGAN for transfer learning and find that this method greatly enhances the performance when a limited number of whole slide images are used for training.

Diabetic nephropathy (DN) in the context of type 2 diabetes is the leading cause of end-stage renal disease (ESRD) in the United States. DN is graded based on glomerular morphology and has a spatially heterogeneous presentation in kidney biopsies that complicates pathologists' predictions of disease progression. Artificial intelligence and deep learning methods for pathology have shown promise for quantitative pathological evaluation and clinical trajectory estimation; but, they often fail to capture large-scale spatial anatomy and relationships found in whole slide images (WSIs). In this study, we present a transformer-based, multi-stage ESRD prediction framework built upon nonlinear dimensionality reduction, relative Euclidean pixel distance embeddings between every pair of observable glomeruli, and a corresponding spatial self-attention mechanism for a robust contextual representation. We developed a deep transformer network for encoding WSI and predicting future ESRD using a dataset of 56 kidney biopsy WSIs from DN patients at Seoul National University Hospital. Using a leave-one-out cross-validation scheme, our modified transformer framework outperformed RNNs, XGBoost, and logistic regression baseline models, and resulted in an area under the receiver operating characteristic curve (AUC) of 0.97 (95% CI: 0.90-1.00) for predicting two-year ESRD, compared with an AUC of 0.86 (95% CI: 0.66-0.99) without our relative distance embedding, and an AUC of 0.76 (95% CI: 0.59-0.92) without a denoising autoencoder module. While the variability and generalizability induced by smaller sample sizes are challenging, our distance-based embedding approach and overfitting mitigation techniques yielded results that suggest opportunities for future spatially aware WSI research using limited pathology datasets.

Accurate quantification of renal fibrosis has profound importance in the assessment of chronic kidney disease (CKD). Visual analysis of a biopsy stained with trichrome under the microscope by a pathologist is the gold standard for evaluation of fibrosis. Trichrome helps to highlight collagen and ultimately interstitial fibrosis. However, trichrome stains are not always reproducible, can underestimate collagen content and are not sensitive to subtle fibrotic patterns. Using the Dual-mode emission and transmission (DUET) microscopy approach, it is possible to capture both brightfield and fluorescence images from the same area of a tissue stained with hematoxylin and eosin (H&E) enabling reproducible extraction of collagen with high sensitivity and specificity. Manual extraction of spectrally overlapping collagen signals from tubular epithelial cells and red blood cells is still an intensive task. We employed a UNet++ architecture for pixel-level segmentation and quantification of collagen using 760 whole slide image (WSI) patches from six cases of varying stages of fibrosis. Our trained model (Deep-DUET) used the supervised extracted collagen mask as ground truth and was able to predict the extent of collagen signal with a MSE of 0.05 in a holdout testing set while achieving an average AUC of 0.94 for predicting regions of collagen deposits. Expanding this work to the level of the WSI can greatly improve the ability of pathologists and machine learning (ML) tools to quantify the extent of renal fibrosis reproducibly and reliably.

Reference histomorphometric data of healthy human kidneys are lacking due to laborious quantitation requirements. We leveraged deep learning to investigate the relationship of histomorphometry with patient age, sex, and serum creatinine in a multinational set of reference kidney tissue sections. A panoptic segmentation neural network was developed and used to segment viable and sclerotic glomeruli, cortical and medullary interstitia, tubules, and arteries/arterioles in digitized images of 79 periodic acid-Schiff (PAS)-stained human nephrectomy sections showing minimal pathologic changes. Simple morphometrics (e.g., area, radius, density) were measured from the segmented classes. Regression analysis was used to determine the relationship of histomorphometric parameters with age, sex, and serum creatinine. The model achieved high segmentation performance for all test compartments. We found that the size and density of nephrons, arteries/arterioles, and the baseline level of interstitium vary significantly among healthy humans, with potentially large differences between subjects from different geographic locations. Nephron size in any region of the kidney was significantly dependent on patient creatinine. Slight differences in renal vasculature and interstitium were observed between sexes. Finally, glomerulosclerosis percentage increased and cortical density of arteries/arterioles decreased as a function of age. We show that precise measurements of kidney histomorphometric parameters can be automated. Even in reference kidney tissue sections with minimal pathologic changes, several histomorphometric parameters demonstrated significant correlation to patient demographics and serum creatinine. These robust tools support the feasibility of deep learning to increase efficiency and rigor in histomorphometric analysis and pave the way for future large-scale studies.

Histological image data and molecular profiles provide context into renal condition. Often, a biopsy is drawn to diagnose or monitor a suspected kidney problem. However, molecular profiles can go beyond a pathologist's ability to see and diagnose. Using AI, we computationally incorporated urinary proteomic profiles with microstructural morphology from renal biopsy to investigate new and existing molecular links to image phenotypes. We studied whole slide images of periodic acid-Schiff stained renal biopsies from 56 DN patients matched with 2,038 proteins measured from each patient's urine. Using Seurat, we identified differentially expressed proteins in patients that developed end-stage renal disease within 2 years of biopsy. Glomeruli, globally sclerotic glomeruli, and tubules were segmented from WSI using our previously published HAIL pipeline. For each glomerulus, 315 handcrafted digital image features were measured, and for tubules, 207 features. We trained fully connected networks to predict urinary protein measurements that were differentially expressed between patients who did/ did not progress to ESRD within 2 years of biopsy. The input to this network was either glomerular or tubular histomorphological features in biopsy. Trained network weights were used as a proxy to rank which morphological features correlated most highly with specific urinary proteins. We identified significant image feature-protein pairs by ranking network weights by magnitude. We also looked at which features on average were most significant in predicting proteins. For both glomeruli and tubules, RGB color values and variance in PAS + areas (specifically basement membrane for tubules) were, on average, more predictive of molecular profiles than other features. There is a strong connection between molecular profile and image phenotype, which can be elucidated through computational methods. These discovered links can provide insight to disease pathways, and discover new factors contributing to incidence and progression.

The incorporation of automated computational tools has a great amount of potential to positively influence the field of pathology. However, pathologists and regulatory agencies are reluctant to trust the output of complex models such as Convolutional Neural Networks (CNNs) due to their usual implementation as black-box tools. Increasing the interpretability of quantitative analyses is a critical line of research in order to increase the adoption of modern Machine Learning (ML) pipelines in clinical environments. Towards that goal, we present HistoLens , a Graphical User Interface (GUI) designed to facilitate quantitative assessments of datasets of annotated histological compartments. Additionally, we introduce the use of hand-engineered feature visualizations to highlight regions within each structure that contribute to particular feature values. These feature visualizations can then be paired with feature hierarchy determinations in order to view which regions within an image are significant to a particular sub-group within the dataset. As a use case, we analyzed a dataset of old and young mouse kidney sections with glomeruli annotated. We highlight some of the functional components within HistoLens that allow non-computational experts to efficiently navigate a new dataset as well as allowing for easier transition to downstream computational analyses.

Podocyte injury plays a crucial role in the progression of diabetic kidney disease (DKD). Injured podocytes demonstrate variations in nuclear shape and chromatin distribution. These morphometric changes have not yet been quantified in podocytes. Furthermore, the molecular mechanisms underlying these variations are poorly understood. Recent advances in omics have shed new lights into the biological mechanisms behind podocyte injury. However, there currently exists no study analyzing the biological mechanisms underlying podocyte morphometric variations during DKD. First, to study the importance of nuclear morphometrics, we performed morphometric quantification of podocyte nuclei from whole slide images of renal tissue sections obtained from murine models of DKD. Our results indicated that podocyte nuclear textural features demonstrate statistically significant difference in diabetic podocytes when compared to control. Additionally, the morphometric features demonstrated the existence of multiple subpopulations of podocytes suggesting a potential cause for their varying response to injury. Second, to study the underlying pathophysiology, we employed single cell RNA sequencing data from the murine models. Our results again indicated five subpopulations of podocytes in control and diabetic mouse models, validating the morphometrics-based results. Additionally, gene set enrichment analysis revealed epithelial to mesenchymal transition and apoptotic pathways in a subgroup of podocytes exclusive to diabetic mice, suggesting the molecular mechanism behind injury. Lastly, our results highlighted two distinct lineages of podocytes in control and diabetic cases suggesting a phenotypical change in podocytes during DKD. These results suggest that textural variations in podocyte nuclei may be key to understanding the pathophysiology behind podocyte injury.

One of the strongest prognostic predictors of chronic kidney disease is interstitial fibrosis and tubular atrophy (IFTA). The ultimate goal of IFTA calculation is an estimation of the functional nephritic area. However, the clinical gold standard of estimation by pathologist is imprecise, primarily due to the overwhelming number of tubules sampled in a standard kidney biopsy. Artificial intelligence algorithms could provide significant benefit in this aspect as their high-throughput could identify and quantitatively measure thousands of tubules in mere minutes. Towards this goal, we use a custom panoptic convolutional network similar to Panoptic-DeepLab to detect tubules from 87 WSIs of biopsies from native diabetic kidneys and transplant kidneys. We measure 206 features on each tubule, including commonly understood features like tubular basement membrane thickness and tubular diameter. Finally, we have developed a tool which allows a user to select a range of tubule morphometric features to be highlighted in corresponding WSIs. The tool can also highlight tubules in WSI leveraging multiple morphometric features through selection of regions-of-interest in a uniform manifold approximation and projection plot.

It is commonly known that diverse datasets of WSIs are beneficial when training convolutional neural networks, however sharing medical data between institutions is often hindered by regulatory concerns. We have developed a cloud-based tool for federated WSI segmentation, allowing collaboration between institutions without the need to directly share data. To show the feasibility of federated learning on pathology data in the real world, We demonstrate this tool by segmenting IFTA from three institutions and show that keeping the three datasets separate does not hinder segmentation performance. This pipeline is deployed in the cloud for easy access for data viewing and annotation by each site's respective constituents.

Histologic examination of interstitial fibrosis and tubular atrophy (IFTA) is critical to determine the extent of irreversible kidney injury in renal disease. The current clinical standard involves pathologist's visual assessment of IFTA, which is prone to inter-observer variability. To address this diagnostic variability, we designed two case studies (CSs), including seven pathologists, using HistomicsTK- a distributed system developed by Kitware Inc. (Clifton Park, NY). Twenty-five whole slide images (WSIs) were classified into a training set of 21 and a validation set of four. The training set was composed of seven unique subsets, each provided to an individual pathologist along with four common WSIs from the validation set. In CS 1, all pathologists individually annotated IFTA in their respective slides. These annotations were then used to train a deep learning algorithm to computationally segment IFTA. In CS 2, manual and computational annotations from CS 1 were first reviewed by the annotators to improve concordance of IFTA annotation. Both the manual and computational annotation processes were then repeated as in CS1. The inter-observer concordance in the validation set was measured by Krippendorff's alpha (KA). The KA for the seven pathologists in CS1 was 0.62 with CI [0.57, 0.67], and after reviewing each other's annotations in CS2, 0.66 with CI [0.60, 0.72]. The respective CS1 and CS2 KA were 0.58 with CI [0.52, 0.64] and 0.63 with CI [0.56, 0.69] when including the deep learner as an eighth annotator. These results suggest that our designed annotation framework refines agreement of spatial annotation of IFTA and demonstrates a human-AI approach to significantly improve the development of computational models.

In diabetic kidney disease (DKD), podocyte depletion, and the subsequent migration of parietal epithelial cells (PECs) to the tuft, is a precursor to progressive glomerular damage, but the limitations of brightfield microscopy currently preclude direct pathological quantitation of these cells. Here we present an automated approach to podocyte and PEC detection developed using kidney sections from mouse model emulating DKD, stained first for Wilms' Tumor 1 (WT1) (podocyte and PEC marker) by immunofluorescence, then post-stained with periodic acid-Schiff (PAS). A generative adversarial network (GAN)-based pipeline was used to translate these PAS-stained sections into WT1-labeled IF images, enabling in silico label-free podocyte and PEC identification in brightfield images. Our method detected WT1-positive cells with high sensitivity/specificity (0.87/0.92). Additionally, our algorithm performed with a higher Cohen's kappa (0.85) than the average manual identification by three renal pathologists (0.78). We propose that this pipeline will enable accurate detection of WT1-positive cells in research applications.

With the rapid advancement in multiplex tissue staining, computer hardware, and machine learning, computationally-based tools are becoming indispensable for the evaluation of digital histopathology. Historically, standard histochemical staining methods such as hematoxylin and eosin, periodic acid-Schiff, and trichrome have been the gold standard for microscopic tissue evaluation by pathologists, and therefore brightfield microscopy images derived from such stains are primarily used for developing computational pathology tools. However, these histochemical stains are nonspecific in terms of highlighting structures and cell types. In contrast, immunohistochemical stains use antibodies to specifically detect and quantify proteins, which can be used to specifically highlight structures and cell types of interest. Traditionally, such immunofluorescence-based methods are only able to simultaneously stain a limited number of target proteins/antigens, typically up to three channels. Fluorescence-based multiplex immunohistochemistry (mIHC) is a new technology that enables simultaneous localization and quantification of numerous proteins/antigens, allowing for the possibility to detect a wide range of histologic structures and cell types within tissue. However, this method is limited by cost, specialized equipment, technical expertise, and time. In this study, we implemented a deep learning-based pipeline to synthetically generate in silico mIHC images from brightfield images of tissue slides-stained with routinely used histochemical stains, in particular PAS. Our tool was trained using fluorescence-based mIHC images as the ground-truth. The proposed pipeline offers high contrast detection of structures in brightfield imaged tissue sections stained with standard histochemical stains. We demonstrate the performance of our pipeline by computationally detecting multiple compartments in kidney biopsies, including cell nuclei, collagen/fibrosis, distal tubules, proximal tubules, endothelial cells, and leukocytes, from PAS-stained tissue sections. Our work can be extended for other histologic structures and tissue types and can be used as a basis for future automated annotation of histologic structures and cell types without the added cost of actually generating mIHC slides.

Convolutional neural networks, the state of the art for image segmentation, have been successfully applied to histology images by many computational researchers. However, the translatability of this technology to clinicians and biological researchers is limited due to the complex and undeveloped user interface of the code, as well as the extensive computer setup required. We have developed a plugin for segmentation of whole slide images (WSIs) with an easy to use graphical user interface. This plugin runs a state-of-the-art convolutional neural network for segmentation of WSIs in the cloud. Our plugin is built on the open source tool HistomicsTK by Kitware Inc. (Clifton Park, NY), which provides remote data management and viewing abilities for WSI datasets. The ability to access this tool over the internet will facilitate widespread use by computational non-experts. Users can easily upload slides to a server where our plugin is installed and perform the segmentation analysis remotely. This plugin is open source and once trained, has the ability to be applied to the segmentation of any pathological structure. For a proof of concept, we have trained it to segment glomeruli from renal tissue images, demonstrating it on holdout tissue slides.

The primary purpose of the kidney, specifically the glomerulus, is filtration. Filtration is accomplished through the glomerular filtration barrier, which consists of the fenestrated endothelium, glomerular basement membrane, and specialized epithelial cells called podocytes. In pathologic states, such as Diabetes Mellitus (DM) and diabetic kidney disease (DKD), variable glomerular conditions result in podocyte injury and depletion, followed by progressive glomerular injury and DKD progression. In this work we quantified glomerulus and podocyte structural changes in histopathology image data derived from a murine model of DM. Using a variety of image processing techniques, we studied changes in podocyte morphology and intra-glomerular distribution across healthy, mild DM, and DM glomeruli. Our feature analysis provided feature trends which we believe are reflective of DKD pathology; while glomerular area peaked in mild DM, average podocyte number and distance from the urinary pole continued to decrease and increase, respectively, throughout DM. Ultimately, this study aims to augment the set of quantifiable image biomarkers used for evaluation of DKD progression in digital pathology, as well as underscore the importance of engineering biologically-inspired image features.

In the age of modern medicine and artificial intelligence, image analysis and machine learning have revolutionized diagnostic pathology, facilitating the development of computer aided diagnostics (CADs) which circumvent prevalent diagnostic challenges. Although CADs will expedite and improve the precision of clinical workflow, their prognostic potential, when paired with clinical outcome data, remains indeterminate. In high impact renal diseases, such as diabetic nephropathy and lupus nephritis (LN), progression often occurs rapidly and without immediate detection, due to the subtlety of structural changes in transient disease states. In such states, exploration of quantifiable image biomarkers, such as Neutrophil Extracellular Traps (NETs), may reveal alternative progression measures which correlate with clinical data. NETs have been implicated in LN as immunogenic cellular structures, whose occurrence and dysregulation results in excessive tissue damage and lesion manifestation. We propose that renal biopsy NET distribution will function as a discriminate, predictive biomarker in LN, and will supplement existing classification schemes. We have developed a computational pipeline for segmenting NET-like structures in LN biopsies. NET-like structures segmented from our biopsies warrant further study as they appear pathologically distinct, and resemble non-lytic, vital NETs. Examination of corresponding H&E regions predominantly placed NET-like structures in glomeruli, including globally and segmentally sclerosed glomeruli, and tubule lumina. Our work continues to explore NET-like structures in LN biopsies by: 1.) revising detection and analytical methods based on evolving NETs definitions, and 2.) cataloguing NET morphology in order to implement supervised classification of NET-like structures in histopathology images.

Generative modeling using GANs has gained traction in machine learning literature, as training does not require labeled datasets. This is perfect for applications in biological datasets, where large labeled datasets are often difficult and expensive to acquire. However, generative models offer no easy way to encode real images into feature-sets, something that is desirable for network explainability and may yield potentially informative image features. For this reason, we test a VAE-GAN architecture for label-free modeling of glomerular structural features. We show that this network can generate realistic looking synthetic images, and be used to interpolate between images. To prove the biological relevance of the network encodings, we classify small-labeled sets of encoded glomeruli by biopsy Tervaert class and for the presence of sclerosis, obtaining a Cohen's kappa values of 0.87 and 0.78 respectfully.

Generative adversarial networks (GANs) have received immense attention in the field of machine learning for their potential to learn high-dimensional and real data distribution. These methods do not rely on any assumptions about the data distribution of the input sample and can generate real-like samples from latent vector space based on unsupervised learning. In the medical field, particularly, in digital pathology expert annotation and availability of a large set of training data is costly and the study of manifestations of various diseases is based on visual examination of stained slides. In clinical practice, various staining information is required to improve the pathological diagnosis process. But when the sampled tissue to be examined is limited, the final diagnosis made by the pathologist is based on limited stain styles. These limitations can be overcome by studying the usability and reliability of generative models in the field of digital pathology. To understand the usability of the generative models, we synthesize in an unsupervised manner, high resolution renal microanatomical structures like renal glomerulus in thin tissue histology images using state-of-art architectures like Deep Convolutional Generative Adversarial Network (DCGAN) and Enhanced Super-Resolution Generative Adversarial Network (ESRGAN). Successful generation of such structures will lead to obtaining a large set of labeled data for further developing supervised algorithms for disease classification and understanding progression. Our study suggests while GAN is able to attain formalin fixed and paraffin embedded tissue image quality, GAN requires further prior knowledge as input to model intrinsic micro-anatomical details, such as capillary wall, urinary pole, nuclei placement, suggesting developing semi-supervised architectures by using these above details as prior information. Also, the generative models can be used to create an artificial effect of staining without physically tampering the histopathological slide. To demonstrate this, we use a CycleGAN network to transform Hematoxylin and eosin (H&E) stain to Periodic acid-Schiff (PAS) stain and Jones methenamine silver (JMS) stain to PAS stain. In this way GAN can be employed to translate different renal pathology stain styles when the relevant staining information is not available in the clinical settings.

Diabetic Nephropathy (DN) progression is stratified into several stages with different levels of proteinuria, albuminuria, and physical characteristics as observed by pathologists. These physical changes are primarily visible within a patient's glomeruli which function as filtration units for blood returning for oxygenation. As DN stage increases, it is possible to observe the thickening of the glomerular basement membrane, expansion of the mesangium, and development of nodular sclerosis. Classification of different stages of DN by pathologists is based on semi-qualitative assessments of these characteristics on an individual glomerulus basis. Being able to probabilistically infer stage membership of individual glomeruli based on a combination of easily observable and hidden image features would be an invaluable tool for furthering our understanding of the drivers of DN progression. Markov Particle filters, included in the bnlearn package in R, were used to query a Bayesian Network (BN) constructed using the structural Hill-Climbing algorithm on a set of glomerular features. These features included both traditional characteristics such as glomerular area and number of mesangial nuclei as well as more abstract features derived from Minimum Spanning Trees (MST) to quantify spatial distribution of mesangial nuclei. Our results using images from multiple institutions suggest that these abstract features exercise a variable influence on DN stage membership over the course of disease progression. Further research incorporating clinical data will give nephrologists a "white box" visual of quantitative factors present in DN patients.

In diabetic nephropathy (DN), hyperglycemia drives a progressive thickening of glomerular filtration surfaces, increased cell proliferation as well as mesangial expansion and a constriction of capillary lumens. This leads to progressive structural changes inside the Glomeruli. In this work, we make a study of structural glomerular changes in DN from a graph-theoretic standpoint, using features extracted from Minimal Spanning Trees (MSTs) constructed over intercellular distances in order to classify the "packing signatures" of different DN stages. We further investigate the significance of the competing effects of Volume change measured here in 2Dimensional Pixel span area (Area) on one hand and increased cell proliferation on the other in determining the packing patterns. Towards that we formulate the problem as Dynamic Bayesian Network (DBN). From our preliminary results we do postulate that volume expansion caused by internal pressure as capillary lumens constriction has perhaps has a greater effect in the early stages.