Search

Molecular classification has transformed the management of brain tumors by enabling more accurate prognostication and personalized treatment. However, timely molecular diagnostic testing for patients with brain tumors is limited, complicating surgical and adjuvant treatment and obstructing clinical trial enrollment. In this study, we developed DeepGlioma, a rapid ($< 90$ seconds), artificial-intelligence-based diagnostic screening system to streamline the molecular diagnosis of diffuse gliomas. DeepGlioma is trained using a multimodal dataset that includes stimulated Raman histology (SRH); a rapid, label-free, non-consumptive, optical imaging method; and large-scale, public genomic data. In a prospective, multicenter, international testing cohort of patients with diffuse glioma ($n=153$) who underwent real-time SRH imaging, we demonstrate that DeepGlioma can predict the molecular alterations used by the World Health Organization to define the adult-type diffuse glioma taxonomy (IDH mutation, 1p19q co-deletion and ATRX mutation), achieving a mean molecular classification accuracy of $93.3\pm 1.6\%$. Our results represent how artificial intelligence and optical histology can be used to provide a rapid and scalable adjunct to wet lab methods for the molecular screening of patients with diffuse glioma., Paper published in Nature Medicine

The detection and tracking of metastatic cancer over the lifetime of a patient remains a major challenge in clinical trials and real-world care. 1–3 Recent advances in deep learning combined with massive, real-world datasets may enable the development of tools that can address this challenge. We present our work with the NYUMets Project to develop NYUMets-Brain and a novel longitudinal deep neural network (DNN), segmentation-through-time (STT). NYUMets-Brain is the world's largest, longitudinal, real-world dataset of cancer consisting of the imaging, clinical follow-up, and medical management of 1,429 patients with an average of six MRI studies obtained over 17 months per patient. The dataset includes over 2,367 expert segmentations of metastatic brain tumors, and 81,562 medical prescriptions. Using this dataset we developed Segmentation Through Time (STT), a deep neural network (DNN) which explicitly utilizes the longitudinal structure of the data and obtained state of the art results at tumor segmentation and detection of small (3) metastases. We also demonstrate that longitudinal measurements to assess the monthly rate of change of brain metastases over time are strongly predictive of overall survival (HR 1.27, 95%CI 1.18–1.38). We are releasing the entire dataset, codebase, trained model weights, and an interface for dataset access for other cancer researchers to build upon these results and to serve as a public benchmark. Massive real-world datasets and public benchmarks such as NYUMets-Brain may enable the tracking and detection of metastatic brain cancer, and be broadly applicable to advancing the development of AI models in other types of metastatic cancer as well.

Fluorescence guidance is widely utilized to improve the precision of cancer surgery. 5-aminolevulinic acid, the most widely used fluorophore in glioma surgery, is thought to cause selective accumulation of fluorescent protoporphyrin IX (PpIX) in tumor cells. 5-aminolevulinic acid is highly specific for densely tumor-infiltrated tissue but less effective for visualizing the tumor periphery. To improve clinical detection of PpIX, we developed a microscope to perform paired stimulated Raman histology and two-photon excitation fluorescence microscopy (TPEF) and validated it in 175 fresh tumor specimens from 75 high-grade glioma patients across three institutions. Here, we demonstrate that intracellular PpIX accumulation occurs most prominently in histiocytic, rather than neoplastic, appearing cells. Spatially resolved metabolomics, transcriptomics and RNA sequencing revealed that PPIX is most avidly concentrated in tumor associated macrophages. There was no correlation between the degree of tissue cellularity and PpIX concentration across all imaged specimens (R=-0.21). Our findings encourage reconsideration of the existing theory of 5-ALA-induced glioma cell fluorescence and demonstrate how 5-ALA and TPEF imaging can provide a window into the immune microenvironment of human gliomas.

Over 150,000 Americans are diagnosed with colorectal cancer (CRC) every year, and annually over 50,000 individuals will die from CRC, necessitating improvements in screening, prognostication, disease management, and therapeutic options. CRC tumors are removed en bloc with surrounding vasculature and lymphatics. Examination of regional lymph nodes at the time of surgical resection is essential for prognostication. Developing alternative approaches to indirectly assess recurrence risk would have utility in cases where lymph node yield is incomplete or inadequate. Spatially dependent, immune cell-specific (e.g., Tumor Infiltrating Lymphocytes– TILs), proteomic, and transcriptomic expression patterns inside and around the tumor - the tumor immune microenvironment (TIME) - can predict nodal/distant metastasis and probe the coordinated immune response from the primary tumor site. The comprehensive characterization of TILs and other immune infiltrates is possible using highly multiplexed spatial omics technologies, such as the GeoMX Digital Spatial Profiler (DSP). In this study, machine learning and differential co-expression analyses helped identify biomarkers from DSP-assayed protein expression patterns inside, at the invasive margin, and away from the tumor, associated with extracellular matrix remodeling (e.g., GZMB, fibronectin), immune suppression (e.g., FOXP3), exhaustion and cytotoxicity (e.g., CD8), PD-L1 expressing dendritic cells, neutrophil proliferation, amongst other concomitant alterations. Further investigation of these biomarkers may reveal independent risk factors of CRC metastasis that can be formulated into low-cost, widely available assays.

Molecular classification has transformed the management of brain tumors by enabling more accurate prognostication and personalized treatment. However, timely molecular diagnostic testing for brain tumor patients is limited [1–3], complicating surgical and adjuvant treatment and obstructing clinical trial enrollment [4]. Here, we developed DeepGlioma, a rapid (

The estimated incidence of pituitary adenomas in the general population is 10-30%, yet radiographic diagnosis remains a challenge. Diagnosis is complicated by the heterogeneity of radiographic features in both normal (e.g. complex anatomy, pregnancy) and pathologic states (e.g. primary endocrinopathy, hypophysitis). Clinical symptoms and laboratory testing are often equivocal, which can result in misdiagnosis or unnecessary specialist referrals. Computer vision models can aid in pituitary adenoma diagnosis; however, a major challenge to model development is the lack of dedicated pituitary imaging datasets. We hypothesized that deep volumetric segmentation models trained to extract the sellar and parasellar region from existing whole-brain MRI scans could be used to generate a novel dataset of pituitary imaging.Six open-source whole-brain MRI datasets, created for research purposes, were included for model development. Deep learning-based volumetric segmentation models were trained using 318 manually annotated MRI scans from a single open-source MRI dataset. Out-of-distribution volumetric segmentation performance was then tested on 418 MRIs from five held-out research datasets.On our annotated images, agreement between manual and model volumetric segmentations was high. Dice scores (a measure of overlap) ranged 0.76-0.82 for both in-distribution and out-of-distribution model testing. In total, 6,755 MRIs from six data sources were included in the final generated pituitary dataset.We present the first and largest dataset of pituitary imaging constructed using existing MRI data and deep volumetric segmentation models trained to identify sellar and parasellar anatomy. The model generalizes well across patient populations and MRI scanner types. We hope our pituitary dataset will be an integral part of future machine learning research on pituitary pathologies.

The development of accurate machine learning algorithms requires sufficient quantities of diverse data. This poses a challenge in health care because of the sensitive and siloed nature of biomedical information. Decentralized algorithms through federated learning (FL) avoid data aggregation by instead distributing algorithms to the data before centrally updating one global model.To establish a multicenter collaboration and assess the feasibility of using FL to train machine learning models for intracranial hemorrhage (ICH) detection without sharing data between sites.Five neurosurgery departments across the United States collaborated to establish a federated network and train a convolutional neural network to detect ICH on computed tomography scans. The global FL model was benchmarked against a standard, centrally trained model using a held-out data set and was compared against locally trained models using site data.A federated network of practicing neurosurgeon scientists was successfully initiated to train a model for predicting ICH. The FL model achieved an area under the ROC curve of 0.9487 (95% CI 0.9471-0.9503) when predicting all subtypes of ICH compared with a benchmark (non-FL) area under the ROC curve of 0.9753 (95% CI 0.9742-0.9764), although performance varied by subtype. The FL model consistently achieved top three performance when validated on any site's data, suggesting improved generalizability. A qualitative survey described the experience of participants in the federated network.This study demonstrates the feasibility of implementing a federated network for multi-institutional collaboration among clinicians and using FL to conduct machine learning research, thereby opening a new paradigm for neurosurgical collaboration.

Spatially resolved characterization of the transcriptome and proteome promises to provide further clarity on cancer pathogenesis and etiology, which may inform future clinical practice through classifier development for clinical outcomes. However, batch effects may potentially obscure the ability of machine learning methods to derive complex associations within spatial omics data. Profiling thirty-five stage three colon cancer patients using the GeoMX Digital Spatial Profiler, we found that mixed-effects machine learning (MEML) methods† may provide utility for overcoming significant batch effects to communicate key and complex disease associations from spatial information. These results point to further exploration and application of MEML methods within the spatial omics algorithm development life cycle for clinical deployment.

Spatially resolved characterization of the transcriptome and proteome promises to provide further clarity on cancer pathogenesis and etiology, which may inform future clinical practice through classifier development for clinical outcomes. However, batch effects may potentially obscure the ability of machine learning methods to derive complex associations within spatial omics data. Profiling thirty-five stage three colon cancer patients using the GeoMX Digital Spatial Profiler, we found that mixed-effects machine learning (MEML) methods† may provide utility for overcoming significant batch effects to communicate key and complex disease associations from spatial information. These results point to further exploration and application of MEML methods within the spatial omics algorithm development life cycle for clinical deployment.

INTRODUCTION Molecular classification has transformed the management of brain tumors by enabling more accurate prognostication and personalized treatment. Access to timely molecular diagnostic testing for brain tumor patients is limited, complicating surgical and adjuvant treatment and obstructing clinical trial enrollment. OBJECTIVE We aim to develop a rapid (< 90 seconds), AI-based diagnostic screening system that can provide molecular classification of diffuse gliomas and report its use in a prospective, multicenter, international clinical trial of diffuse glioma patients (n = 153). METHODS By combining stimulated Raman histology (SRH), a rapid, label-free, non-consumptive, optical imaging method, and deep learning-based image classification, we are able to predict the molecular genetic features used by the World Health Organization (WHO) to define the adult-type diffuse glioma taxonomy, including IDH-1/2, 1p19q-codeletion, and ATRX loss. We developed a multimodal deep neural network training strategy that uses both SRH images and large-scale, public diffuse glioma genomic data in order to achieve optimal molecular classification performance. RESULTS One institution was used for model training (University of Michigan) and four institutions (NYU, UCSF, Medical University of Vienna, and University Hospital Cologne) were included for prospective patient enrollment and model testing. Using our system, called DeepGlioma, we achieved an average molecular genetic classification accuracy of 93.2% and identified the correct diffuse glioma molecular subgroup with 91.5% accuracy. DeepGlioma outperformed conventional IDH1-R132H immunohistochemistry (94.2% versus 91.4% accuracy, respectively) as a first-line molecular diagnostic screening method for diffuse gliomas, detecting canonical and non-canonical IDH mutations with high accuracy. CONCLUSION Our results demonstrate how artificial intelligence and optical histology can be used to provide a rapid and scalable alternative to wet lab methods for the molecular diagnosis of brain tumor patients during surgery.

Fluorescence guidance is widely utilized to improve the precision of cancer surgery. 5-aminolevulinic acid, the most widely used fluorophore in glioma surgery, is thought to cause selective accumulation of fluorescent protoporphyrin IX (PpIX) in tumor cells by exploiting pathologic alterations in heme biosynthetic pathways. PpIX-induced fluorescence is highly specific for densely tumor-infiltrated tissue but less effective for visualizing the tumor periphery. To improve clinical detection of PpIX, we developed a microscope to perform paired stimulated Raman histology and two-photon excitation fluorescence microscopy (TPEF) and validated it in 175 fresh, unprocessed core tumor specimens from 75 high-grade glioma patients and three central nervous system lymphoma patients across three institutions. Surprisingly, intracellular PpIX accumulation was observed primarily in cells with histiocytic, rather than neoplastic morphology, and the number of cells concentrating PpIX within the cytoplasm was associated with the abundance of CD163 positive cells (p< 0.02). There was no correlation between the degree of tumor cellularity and the concentration of PpIX across all imaged specimens (R=-0.21). Our findings encourage reconsideration of the existing theory of 5-ALA-induced tumor cell fluorescence in gliomas and demonstrate how 5-ALA and TPEF imaging can provide a window into the immune microenvironment of human gliomas.

Applying curriculum learning requires both a range of difficulty in data and a method for determining the difficulty of examples. In many tasks, however, satisfying these requirements can be a formidable challenge. In this paper, we contend that histopathology image classification is a compelling use case for curriculum learning. Based on the nature of histopathology images, a range of difficulty inherently exists among examples, and, since medical datasets are often labeled by multiple annotators, annotator agreement can be used as a natural proxy for the difficulty of a given example. Hence, we propose a simple curriculum learning method that trains on progressively-harder images as determined by annotator agreement. We evaluate our hypothesis on the challenging and clinically-important task of colorectal polyp classification. Whereas vanilla training achieves an AUC of 83.7% for this task, a model trained with our proposed curriculum learning approach achieves an AUC of 88.2%, an improvement of 4.5%. Our work aims to inspire researchers to think more creatively and rigorously when choosing contexts for applying curriculum learning.

This diagnostic study evaluates the accuracy and assessment time of an artificial intelligence (AI)–augmented digital system compared with standard microscopic assessment for interpretation of slides with colorectal polyp samples., Key Points Question Can an artificial intelligence (AI)–augmented digital system improve classification accuracy of colorectal polyps by pathologists compared with standard microscopic assessment? Findings In this diagnostic study including 15 pathologists using microscopic examination and an AI-augmented digital system to interpret 100 slides with colorectal polyp samples, use of the AI-augmented digital system significantly improved pathologists' classification accuracy from 73.9% to 80.8% compared with standard microscopic assessment. Meaning These findings suggest that use of an AI-augmented digital system may be associated with improved treatment of patients with colorectal cancer and with improved follow-up surveillance planning to prevent subsequent cancer development., Importance Colorectal polyps are common, and their histopathologic classification is used in the planning of follow-up surveillance. Substantial variation has been observed in pathologists’ classification of colorectal polyps, and improved assessment by pathologists may be associated with reduced subsequent underuse and overuse of colonoscopy. Objective To compare standard microscopic assessment with an artificial intelligence (AI)–augmented digital system that annotates regions of interest within digitized polyp tissue and predicts polyp type using a deep learning model to assist pathologists in colorectal polyp classification. Design, Setting, and Participants In this diagnostic study conducted at a tertiary academic medical center and a community hospital in New Hampshire, 100 slides with colorectal polyp samples were read by 15 pathologists using a microscope and an AI-augmented digital system, with a washout period of at least 12 weeks between use of each modality. The study was conducted from February 10 to July 10, 2020. Main Outcomes and Measures Accuracy and time of evaluation were used to compare pathologists’ performance when a microscope was used with their performance when the AI-augmented digital system was used. Outcomes were compared using paired t tests and mixed-effects models. Results In assessments of 100 slides with colorectal polyp specimens, use of the AI-augmented digital system significantly improved pathologists' classification accuracy compared with microscopic assessment from 73.9% (95% CI, 71.7%-76.2%) to 80.8% (95% CI, 78.8%-82.8%) (P