Gradient modulated contrastive distillation of low-rank multi-modal knowledge for disease diagnosis.

The fusion of multi-modal data, e.g., medical images and genomic profiles, can provide complementary information and further benefit disease diagnosis. However, multi-modal disease diagnosis confronts two challenges: (1) how to produce discriminative multi-modal representations by exploiting complementary information while avoiding noisy features from different modalities. (2) how to obtain an accurate diagnosis when only a single modality is available in real clinical scenarios. To tackle these two issues, we present a two-stage disease diagnostic framework. In the first multi-modal learning stage, we propose a novel Momentum-enriched Multi-Modal Low-Rank (M 3 LR) constraint to explore the high-order correlations and complementary information among different modalities, thus yielding more accurate multi-modal diagnosis. In the second stage, the privileged knowledge of the multi-modal teacher is transferred to the unimodal student via our proposed Discrepancy Supervised Contrastive Distillation (DSCD) and Gradient-guided Knowledge Modulation (GKM) modules, which benefit the unimodal-based diagnosis. We have validated our approach on two tasks: (i) glioma grading based on pathology slides and genomic data, and (ii) skin lesion classification based on dermoscopy and clinical images. Experimental results on both tasks demonstrate that our proposed method consistently outperforms existing approaches in both multi-modal and unimodal diagnoses. • We present a two-stage disease diagnostic framework that first trains a multi-modal network and then distills the multi-modal knowledge to train a unimodal network. • In the multi-modal training stage, we devise a novel Momentum-enriched Multi-Modal Low-Rank ( M 3 L R) constraint to capture the consensus and high-order complementary information among different modalities and momentum encoders. • In the unimodal training stage, we propose a novel Discrepancy Supervised Contrastive Distillation (DSCD) paradigm and a Gradient-guided Knowledge Modulation (GKM) scheme to transfer the multi-modal knowledge toward a more accurate unimodal diagnosis. • The DSCD module distills rich structural knowledge by pulling multiple positive pairs (from the same class) with teacher–student discrepancy while pushing apart negative pairs from different classes. The GKM constructs a comprehensive knowledge bank and adaptively modulates the contributions of multiple knowledge according to their agreement in the gradient space. • We perform extensive experiments on the glioma grading and skin lesion classification tasks to validate the effectiveness of the proposed framework and the experimental results demonstrate the superiority of the proposed method over existing approaches in both multi-modal and unimodal diagnoses. [ABSTRACT FROM AUTHOR]