Search

Introduction: Immune dysregulation plays a major role in cancer progression. The quantification of lymphocytic spatial inflammation may enable spatial system biology, improve understanding of therapeutic resistance, and contribute to prognostic imaging biomarkers., Methods: In this paper, we propose a knowledge-guided deep learning framework to measure the lymphocytic spatial architecture on human H&E tissue, where the fidelity of training labels is maximized through single-cell resolution image registration of H&E to IHC. We demonstrate that such an approach enables pixel-perfect ground-truth labeling of lymphocytes on H&E as measured by IHC. We then experimentally validate our technique in a genetically engineered, immune-compromised Rag2 mouse model, where Rag2 knockout mice lacking mature lymphocytes are used as a negative experimental control. Such experimental validation moves beyond the classical statistical testing of deep learning models and demonstrates feasibility of more rigorous validation strategies that integrate computational science and basic science., Results: Using our developed approach, we automatically annotated more than 111,000 human nuclei (45,611 CD3/CD20 positive lymphocytes) on H&E images to develop our model, which achieved an AUC of 0.78 and 0.71 on internal hold-out testing data and external testing on an independent dataset, respectively. As a measure of the global spatial architecture of the lymphocytic microenvironment, the average structural similarity between predicted lymphocytic density maps and ground truth lymphocytic density maps was 0.86 ± 0.06 on testing data. On experimental mouse model validation, we measured a lymphocytic density of 96.5 ± %1% in a Rag2 +/- control mouse, compared to an average of 16.2 ± %5% in Rag2 -/- immune knockout mice (p<0.0001, ANOVA-test)., Discussion: These results demonstrate that CD3/CD20 positive lymphocytes can be accurately detected and characterized on H&E by deep learning and generalized across species. Collectively, these data suggest that our understanding of complex biological systems may benefit from computationally-derived spatial analysis, as well as integration of computational science and basic science., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2024 Li, Heirman, Rickard, Sotolongo, Castillo, Adanlawo, Everitt, Hodgin, Watts, Janowczyk, Mowery, Barisoni and Lafata.)

Purpose: To develop a novel deep ensemble learning model for accurate prediction of brain metastasis (BM) local control outcomes after stereotactic radiosurgery (SRS)., Methods and Materials: A total of 114 brain metastases (BMs) from 82 patients were evaluated, including 26 BMs that developed biopsy-confirmed local failure post-SRS. The SRS spatial dose distribution (D map ) of each BM was registered to the planning contrast-enhanced T1 (T1-CE) magnetic resonance imaging (MRI). Axial slices of the D map , T1-CE, and planning target volume (PTV) segmentation (PTV seg ) intersecting the BM center were extracted within a fixed field of view determined by the 60% isodose volume in D map . A spherical projection was implemented to transform planar image content onto a spherical surface using multiple projection centers, and the resultant T1-CE/D map /PTV seg projections were stacked as a 3-channel variable. Four Visual Geometry Group (VGG-19) deep encoders were used in an ensemble design, with each submodel using a different spherical projection formula as input for BM outcome prediction. In each submodel, clinical features after positional encoding were fused with VGG-19 deep features to generate logit results. The ensemble's outcome was synthesized from the 4 submodel results via logistic regression. In total, 10 model versions with random validation sample assignments were trained to study model robustness. Performance was compared with (1) a single VGG-19 encoder, (2) an ensemble with a T1-CE MRI as the sole image input after projections, and (3) an ensemble with the same image input design without clinical feature inclusion., Results: The ensemble model achieved an excellent area under the receiver operating characteristic curve (AUC ROC : 0.89 ± 0.02) with high sensitivity (0.82 ± 0.05), specificity (0.84 ± 0.11), and accuracy (0.84 ± 0.08) results. This outperformed the MRI-only VGG-19 encoder (sensitivity: 0.35 ± 0.01, AUC ROC : 0.64 ± 0.08), the MRI-only deep ensemble (sensitivity: 0.60 ± 0.09, AUC ROC : 0.68 ± 0.06), and the 3-channel ensemble without clinical feature fusion (sensitivity: 0.78 ± 0.08, AUC ROC : 0.84 ± 0.03)., Conclusions: Facilitated by the spherical image projection method, a deep ensemble model incorporating D map and clinical variables demonstrated excellent performance in predicting BM post-SRS local failure. Our novel approach could improve other radiation therapy outcome models and warrants further evaluation., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Purpose: To develop a deep learning model design that integrates radiomics analysis for enhanced performance of COVID-19 and non-COVID-19 pneumonia detection using chest x-ray images., Methods: As a novel radiomics approach, a 2D sliding kernel was implemented to map the impulse response of radiomic features throughout the entire chest x-ray image; thus, each feature is rendered as a 2D map in the same dimension as the x-ray image. Based on each of the three investigated deep neural network architectures, including VGG-16, VGG-19, and DenseNet-121, a pilot model was trained using x-ray images only. Subsequently, two radiomic feature maps (RFMs) were selected based on cross-correlation analysis in reference to the pilot model saliency map results. The radiomics-boosted model was then trained based on the same deep neural network architecture using x-ray images plus the selected RFMs as input. The proposed radiomics-boosted design was developed using 812 chest x-ray images with 262/288/262 COVID-19/non-COVID-19 pneumonia/healthy cases, and 649/163 cases were assigned as training-validation/independent test sets. For each model, 50 runs were trained with random assignments of training/validation cases following the 7:1 ratio in the training-validation set. Sensitivity, specificity, accuracy, and ROC curves together with area-under-the-curve (AUC) from all three deep neural network architectures were evaluated., Results: After radiomics-boosted implementation, all three investigated deep neural network architectures demonstrated improved sensitivity, specificity, accuracy, and ROC AUC results in COVID-19 and healthy individual classifications. VGG-16 showed the largest improvement in COVID-19 classification ROC (AUC from 0.963 to 0.993), and DenseNet-121 showed the largest improvement in healthy individual classification ROC (AUC from 0.962 to 0.989). The reduced variations suggested improved robustness of the model to data partition. For the challenging non-COVID-19 pneumonia classification task, radiomics-boosted implementation of VGG-16 (AUC from 0.918 to 0.969) and VGG-19 (AUC from 0.964 to 0.970) improved ROC results, while DenseNet-121 showed a slight yet insignificant ROC performance reduction (AUC from 0.963 to 0.949). The achieved highest accuracy of COVID-19/non-COVID-19 pneumonia/healthy individual classifications were 0.973 (VGG-19)/0.936 (VGG-19)/ 0.933 (VGG-16), respectively., Conclusions: The inclusion of radiomic analysis in deep learning model design improved the performance and robustness of COVID-19/non-COVID-19 pneumonia/healthy individual classification, which holds great potential for clinical applications in the COVID-19 pandemic., (© 2022 American Association of Physicists in Medicine.)

Background: Patients' report of bowel movement consistency is unreliable. We demonstrate the feasibility of long‐term automated stool image data collection using a novel Smart Toilet and evaluate a deterministic computer‐vision analytic approach to assess stool form according to the Bristol Stool Form Scale (BSFS). Methods: Our smart toilet integrates a conventional toilet bowl with an engineered portal to image feces in a predetermined region of the plumbing post‐flush. The smart toilet was installed in a workplace bathroom and used by six healthy volunteers. Images were annotated by three experts. A computer vision method based on deep learning segmentation and mathematically defined hand‐crafted features was developed to quantify morphological attributes of stool from images. Key Results: 474 bowel movements images were recorded in total from six subjects over a mean period of 10 months. 3% of images were rated abnormal with stool consistency BSFS 2 and 4% were BSFS 6. Our image analysis algorithm leverages interpretable morphological features and achieves classification of abnormal stool form with 94% accuracy, 81% sensitivity and 95% specificity. Conclusions: Our study supports the feasibility and accuracy of long‐term, non‐invasive automated stool form monitoring with the novel smart toilet system which can eliminate the patient burden of tracking bowel forms. [ABSTRACT FROM AUTHOR]

Objective: To develop a Multi-Feature-Combined (MFC) model for proof-ofconcept in predicting local failure (LR) in NSCLC patients after surgery or SBRT using pre-treatment CT images. This MFC model combines handcrafted radiomic features, deep radiomic features, and patient demographic information in an integrated machine learning workflow. Methods: The MFC model comprised three key steps. (1) Extraction of 92 handcrafted radiomic features from the GTV segmented on pre-treatment CT images. (2) Extraction of 512 deep radiomic features from pre-trained U-Net encoder. (3) The extracted handcrafted radiomic features, deep radiomic features, along with 4 patient demographic information (i.e., gender, age, tumor volume, and Charlson comorbidity index), were concatenated as a multi-dimensional input to the classifiers for LR prediction. Two NSCLC patient cohorts from our institution were investigated: (1) the surgery cohort includes 83 patients with segmentectomy or wedge resection (7 LR), and (2) the SBRT cohort includes 84 patients with lung SBRT (9 LR). The MFC model was developed and evaluated independently for both cohorts, and was subsequently compared against the prediction models based on only handcrafted radiomic features (R models), patient demographic information (PI models), and deep learning modeling (DL models). ROC with AUC was adopted to evaluate model performance with leave-one-out cross-validation (LOOCV) and 100-fold Monte Carlo random validation (MCRV). The t-test was performed to identify the statistically significant differences. Results: In LOOCV, the AUC range (surgery/SBRT) of the MFC model was 0.858-0.895/0.868-0.913, which was higher than the three other models: 0.356-0.480/0.322-0.650 for PI models, 0.559-0.618/0.639-0.682 for R models, and 0.809/0.843 for DL models. In 100-fold MCRV, the MFC model again showed the highest AUC results (surgery/SBRT): 0.742-0.825/0.888-0.920, which were significantly higher than PI models: 0.464-0.564/0.538-0.628, R models: 0.557-0.652/0.551-0.732, and DL models: 0.702/0.791. Conclusion: We successfully developed an MFC model that combines feature information from multiple sources for proof-of-concept prediction of LR in patients with surgical and SBRT early-stage NSCLC. Initial results suggested that incorporating pre-treatment patient information from multiple sources improves the ability to predict the risk of local failure. [ABSTRACT FROM AUTHOR]

Background: Due to intrinsic differences in data formatting, data structure, and underlying semantic information, the integration of imaging data with clinical data can be non‐trivial. Optimal integration requires robust data fusion, that is, the process of integrating multiple data sources to produce more useful information than captured by individual data sources. Here, we introduce the concept of fusion quality for deep learning problems involving imaging and clinical data. We first provide a general theoretical framework and numerical validation of our technique. To demonstrate real‐world applicability, we then apply our technique to optimize the fusion of CT imaging and hepatic blood markers to estimate portal venous hypertension, which is linked to prognosis in patients with cirrhosis of the liver. Purpose: To develop a measurement method of optimal data fusion quality deep learning problems utilizing both imaging data and clinical data. Methods: Our approach is based on modeling the fully connected layer (FCL) of a convolutional neural network (CNN) as a potential function, whose distribution takes the form of the classical Gibbs measure. The features of the FCL are then modeled as random variables governed by state functions, which are interpreted as the different data sources to be fused. The probability density of each source, relative to the probability density of the FCL, represents a quantitative measure of source‐bias. To minimize this source‐bias and optimize CNN performance, we implement a vector‐growing encoding scheme called positional encoding, where low‐dimensional clinical data are transcribed into a rich feature space that complements high‐dimensional imaging features. We first provide a numerical validation of our approach based on simulated Gaussian processes. We then applied our approach to patient data, where we optimized the fusion of CT images with blood markers to predict portal venous hypertension in patients with cirrhosis of the liver. This patient study was based on a modified ResNet‐152 model that incorporates both images and blood markers as input. These two data sources were processed in parallel, fused into a single FCL, and optimized based on our fusion quality framework. Results: Numerical validation of our approach confirmed that the probability density function of a fused feature space converges to a source‐specific probability density function when source data are improperly fused. Our numerical results demonstrate that this phenomenon can be quantified as a measure of fusion quality. On patient data, the fused model consisting of both imaging data and positionally encoded blood markers at the theoretically optimal fusion quality metric achieved an AUC of 0.74 and an accuracy of 0.71. This model was statistically better than the imaging‐only model (AUC = 0.60; accuracy = 0.62), the blood marker‐only model (AUC = 0.58; accuracy = 0.60), and a variety of purposely sub‐optimized fusion models (AUC = 0.61–0.70; accuracy = 0.58–0.69). Conclusions: We introduced the concept of data fusion quality for multi‐source deep learning problems involving both imaging and clinical data. We provided a theoretical framework, numerical validation, and real‐world application in abdominal radiology. Our data suggests that CT imaging and hepatic blood markers provide complementary diagnostic information when appropriately fused. [ABSTRACT FROM AUTHOR]