Search

Accurate medical classification requires a large number of multi-modal data, and in many cases, different feature types. Previous studies have shown promising results when using multi-modal data, outperforming single-modality models when classifying diseases such as Alzheimer’s Disease (AD). However, those models are usually not flexible enough to handle missing modalities. Currently, the most common workaround is discarding samples with missing modalities which leads to considerable data under-utilisation. Adding to the fact that labelled medical images are already scarce, the performance of data-driven methods like deep learning can be severely hampered. Therefore, a multi-modal method that can handle missing data in various clinical settings is highly desirable. In this paper, we present Multi-Modal Mixing Transformer (3MT), a disease classification transformer that not only leverages multi-modal data but also handles missing data scenarios. In this work, we test 3MT for AD and Cognitively normal (CN) classification and mild cognitive impairment (MCI) conversion prediction to progressive MCI (pMCI) or stable MCI (sMCI) using clinical and neuroimaging data. The model uses a novel Cascaded Modality Transformers architecture with cross-attention to incorporate multi-modal information for more informed predictions. We propose a novel modality dropout mechanism to ensure an unprecedented level of modality independence and robustness to handle missing data scenarios. The result is a versatile network that enables the mixing of arbitrary numbers of modalities with different feature types and also ensures full data utilization in missing data scenarios. The model is trained and evaluated on the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset with the state-of-the-art performance and further evaluated with The Australian Imaging Biomarker & Lifestyle Flagship Study of Ageing (AIBL) dataset with missing data.

Quantitative susceptibility mapping (QSM) is an MRI post-processing technique that produces spatially resolved magnetic susceptibility maps from phase data. However, the traditional QSM reconstruction pipeline involves multiple non-trivial steps, including phase unwrapping, background field removal, and dipole inversion. These intermediate steps not only increase the reconstruction time but accumulates errors. This study aims to overcome existing limitations by developing a Laplacian-of-Trigonometric-functions (LoT) enhanced deep neural network for near-instant quantitative field and susceptibility mapping (i.e., iQFM and iQSM) from raw MRI phase data. The proposed iQFM and iQSM methods were compared with established reconstruction pipelines on simulated and in vivo datasets. In addition, experiments on patients with intracranial hemorrhage and multiple sclerosis were also performed to test the generalization of the proposed neural networks. The proposed iQFM and iQSM methods in healthy subjects yielded comparable results to those involving the intermediate steps while dramatically improving reconstruction accuracies on intracranial hemorrhages with large susceptibilities. High susceptibility contrast between multiple sclerosis lesions and healthy tissue was also achieved using the proposed methods. Comparative studies indicated that the most significant contributor to iQFM and iQSM over conventional multi-step methods was the elimination of traditional Laplacian unwrapping. The reconstruction time on the order of minutes for traditional approaches was shortened to around 0.1 s using the trained iQFM and iQSM neural networks.