Abstract
The integration of multi-modal Magnetic Resonance Imaging (MRI) and clinical data holds great promise for enhancing the diagnosis of neurological disorders (NDs) in real-world clinical settings. Deep Learning (DL) has recently emerged as a powerful tool for extracting meaningful patterns from medical data to aid in diagnosis. However, existing DL approaches struggle to effectively leverage multi-modal MRI and clinical data, leading to suboptimal performance. To address this challenge, we utilize a unique, proprietary multi-modal clinical dataset curated for ND research. Based on this dataset, we propose a novel transformer-based Mixture-of-Experts (MoE) framework for ND classification, leveraging multiple MRI modalities-anatomical (aMRI), Diffusion Tensor Imaging (DTI), and functional (fMRI)-alongside clinical assessments. Our framework employs transformer encoders to capture spatial relationships within volumetric MRI data while utilizing modality-specific experts for targeted feature extraction. A gating mechanism with adaptive fusion dynamically integrates expert outputs, ensuring optimal predictive performance. Comprehensive experiments and comparisons with multiple baselines demonstrate that our multi-modal approach significantly enhances diagnostic accuracy, particularly in distinguishing overlapping disease states. Our framework achieves a validation accuracy of 82.47\%, outperforming baseline methods by over 10\%, highlighting its potential to improve ND diagnosis by applying multi-modal learning to real-world clinical data.
Abstract (translated)
多模式磁共振成像（MRI）与临床数据的结合在真实世界临床环境中提高神经疾病诊断能力方面具有巨大潜力。深度学习（DL）最近作为从医学数据中提取有意义模式以辅助诊断的强大工具而崭露头角。然而，现有的深度学习方法难以有效利用多种模态的MRI和临床数据，导致性能不佳。为了解决这一挑战，我们采用了一套独特且专有的多模态临床数据集，该数据集经过精心整理用于神经疾病研究。基于此数据集，我们提出了一种新型的、基于变压器的专家混合（MoE）框架，用于神经疾病的分类。我们的方法结合了多种MRI模式——解剖学（aMRI）、扩散张量成像（DTI）和功能性（fMRI），以及临床评估。该框架使用变压器编码器来捕捉体积MRI数据中的空间关系，并利用特定模态的专家进行目标特征提取。一种具有自适应融合功能的门控机制动态整合了各个专家输出，确保预测性能最优。全面的实验及与多个基线方法的比较表明，我们的多模式方法显著提高了诊断准确性，特别是在区分重叠疾病状态方面有突出表现。我们框架在验证集上的准确率为82.47%，优于所有基准方法超过10%，这凸显了其通过应用于真实世界临床数据的多模态学习来改善神经疾病诊断的巨大潜力。
URL
https://arxiv.org/abs/2506.14970
