Abstract
Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities across tasks, yet they often exhibit difficulty in distinguishing task-relevant from irrelevant signals, particularly in tasks like Visual Question Answering (VQA), which can lead to susceptibility to misleading or spurious inputs. We refer to this broader limitation as the Cross-Modality Competency Problem: the model's inability to fairly evaluate all modalities. This vulnerability becomes more evident in modality-specific tasks such as image classification or pure text question answering, where models are expected to rely solely on one modality. In such tasks, spurious information from irrelevant modalities often leads to significant performance degradation. We refer to this failure as Modality Interference, which serves as a concrete and measurable instance of the cross-modality competency problem. We further design a perturbation-based causal diagnostic experiment to verify and quantify this problem. To mitigate modality interference, we propose a novel framework to fine-tune MLLMs, including perturbation-based data augmentations with both heuristic perturbations and adversarial perturbations via Projected Gradient Descent (PGD), and a consistency regularization strategy applied to model outputs with original and perturbed inputs. Experiments on multiple benchmark datasets (image-heavy, text-heavy, and VQA tasks) and multiple model families with different scales demonstrate significant improvements in robustness and cross-modality competency, indicating our method's effectiveness in boosting unimodal reasoning ability while enhancing performance on multimodal tasks.
Abstract (translated)
多模态大型语言模型（MLLMs）在各种任务中展示了令人印象深刻的性能，但它们经常难以区分与任务相关的信号和不相关的信号，特别是在如视觉问答（VQA）这样的任务中。这会导致这些模型容易受到误导性或错误输入的影响。我们将这种更广泛的问题称为跨模态胜任力问题：即模型无法公平评估所有模式的能力。在特定于某一模式的任务（如图像分类或纯文本问答）中，这一脆弱性变得更为明显，在这类任务中，模型应仅依赖一种模式。在这种情况下，来自不相关模式的错误信息通常会导致性能显著下降。我们将这种失败称为模态干扰，这是跨模态胜任力问题的一个具体而可量化的实例。 为了验证和量化这个问题，我们设计了一个基于扰动的因果诊断实验。为减轻模态干扰，我们提出了一种新颖的微调MLLM框架，包括基于数据增强的启发式和对抗性扰动（通过投影梯度下降实现），以及应用于模型输出的一致性正则化策略，这些输出是基于原始输入和扰动后输入生成的。 在多个基准数据集上进行实验（图像密集型、文本密集型及VQA任务）以及不同规模的多种模型家族显示了显著的鲁棒性和跨模态胜任力改进。这表明我们的方法不仅能够增强单一模式推理能力，还能提高多模式任务的表现。
URL
https://arxiv.org/abs/2505.19616
