Abstract
Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources. However, vanilla fusion methods are limited by (1) inability to account for heterogeneous interactions between modalities and (2) lack of interpretability in uncovering the multimodal interactions inherent in the data. To this end, we propose I2MoE (Interpretable Multimodal Interaction-aware Mixture of Experts), an end-to-end MoE framework designed to enhance modality fusion by explicitly modeling diverse multimodal interactions, as well as providing interpretation on a local and global level. First, I2MoE utilizes different interaction experts with weakly supervised interaction losses to learn multimodal interactions in a data-driven way. Second, I2MoE deploys a reweighting model that assigns importance scores for the output of each interaction expert, which offers sample-level and dataset-level interpretation. Extensive evaluation of medical and general multimodal datasets shows that I2MoE is flexible enough to be combined with different fusion techniques, consistently improves task performance, and provides interpretation across various real-world scenarios. Code is available at this https URL.
Abstract (translated)
模态融合是多模态学习的核心，它能够整合来自不同数据源的信息。然而，传统的融合方法在（1）无法处理模态之间的异构交互以及（2）缺乏解释性来揭示数据中固有的多模态交互方面存在局限性。为此，我们提出了一种新的框架I2MoE（可解释的多模态交互感知专家混合模型），这是一种端到端的专家混合框架，旨在通过显式建模多种多模态交互以及在局部和全局层面提供解释来增强模态融合。 首先，I2MoE 使用带有弱监督交互损失的不同交互专家来以数据驱动的方式学习多模态交互。其次，I2MoE 部署了一个重新加权模型，该模型为每个交互专家的输出分配重要性分数，这提供了样本级别和数据集级别的解释。 对医疗和通用多模态数据集进行广泛的评估显示，I2MoE 具有足够的灵活性与不同的融合技术结合使用，并且在各种现实场景中持续改进任务性能并提供解释。代码可在以下链接获取：[此 URL](https://this https URL)（请将此处的“此 https URL”替换为实际提供的具体链接）。
URL
https://arxiv.org/abs/2505.19190
