Knowledge distillation in transformers with tripartite attention: Multiclass brain tumor detection in highly augmented MRIs

The advent of attention-based architectures in medical imaging has ushered in an era of precision diagnostics, particularly in the detection and classification of brain tumors. This study introduced an innovative knowledge distillation framework employing a tripartite attention mechanism within transformer encoder models, specifically tailored for the identification of multiple brain tumor classes through magnetic resonance imaging (MRI). The proposed methodology synergistically harnesses the capabilities of large, highly parameterized teacher models to train more compact, efficient student models suitable for deployment in resource-constrained environments such as the internet of medical things and smart healthcare devices. Utilizing a diverse array of MRI sequences—including T1, contrast-enhanced T1, and T2—this study accounts for the nuanced variations across brain tumor classes derived from three extensive datasets. The tripartite attention mechanism addresses the limitation of traditional attention models by innovatively integrating temperature-softening neighborhood attention, global attention, and cross-attention layers. This sophisticated approach allows for a richer and more nuanced feature representation, capturing both local and global contextual information and intricate tumor features within MRI scans. This is supplemented by a unique augmentation pipeline and shifted patch tokenization technique, which enrich the model's input representation, especially for underrepresented classes. Through meticulous experimentation and ablation studies, the study demonstrates that the proposed model not only retains the robustness of its larger counterparts but also delivers enhanced performance metrics. When juxtaposed with benchmarking models—including traditional deep CNNs and various transformer-based architectures—the proposed model consistently showcases superior results. Its effectiveness is reflected in its lower teacher and student losses, commendable Brier scores, and noteworthy top-1 and top-5 accuracies, as well as AUC metrics across all datasets. This paper not only validates the efficacy of knowledge distillation in complex medical image analysis tasks but also provides a promising pathway for the integration of cutting-edge AI techniques in real-world clinical applications, potentially revolutionizing the early detection and treatment of brain tumors.