Search

Most of the previous machine learning-based models for multimodal medical diagnosis, primarily designed for unimodal images, usually do not fully leverage the potential of multimodal medical images, leading to limited classification accuracy. These conventional methods typically focus only on the intermodality common information, neglecting the intramodality specific information and assuming that the common information is more effective in disease diagnosis. Moreover, they do not adequately address the impact of fuzzy information between different medical imaging modalities on diagnostic results. To this end, we propose a modality fuzzy information separation network for disease classification, which extracts both common and specific information from fuzzy information to construct a comprehensive representation of multimodal medical images. Specifically, we extract modality invariant features as common information by explicitly modeling and maximizing loss constraints on mutual information. For specific information extraction, a constraint on feature space independence between specific and common information is imposed on each modality. Above-mentioned two steps, we concatenate common information and specific information to construct a comprehensive multimodal representation for separating fuzzy information. Finally, we purposely design a decoder network to reconstruct medical images from unimodal specific information and common information to demonstrate the effectiveness of the modality fuzzy information separation network. We conducted a validation of the proposed method's performance in classifying cardiomegaly, pneumothorax, edema, and skin disease. The experimental results substantiate the effectiveness of our proposed approach.