Abstract
Medical vision-language models show strong potential for joint reasoning over medical images and clinical text, but their performance often degrades under domain shift caused by variations in imaging devices, acquisition protocols, and reporting styles. Existing multi-modal pre-training methods largely overlook robustness, treating it as a downstream adaptation problem. In this work, we propose Robust Multi-Modal Masked Reconstruction (Robust-MMR), a self-supervised pre-training framework that explicitly incorporates robustness objectives into masked vision-language learning. Robust-MMR integrates asymmetric perturbation-aware masking, domain-consistency regularization, and modality-resilience constraints to encourage domain-invariant representations. We evaluate Robust-MMR on multiple medical vision-language benchmarks, including medical visual question answering (VQA-RAD, SLAKE, VQA-2019), cross-domain image-text classification (MELINDA), and robust image-caption retrieval (ROCO). Robust-MMR achieves 78.9% cross-domain accuracy on VQA-RAD, outperforming the strongest baseline by 3.8 percentage points, and reaches 74.6% and 77.0% accuracy on SLAKE and VQA-2019, respectively. Under perturbed evaluation, Robust-MMR improves VQA-RAD accuracy from 69.1% to 75.6%. For image-text classification, cross-domain MELINDA accuracy increases from 70.3% to 75.2%, while retrieval experiments show a reduction in mean rank degradation from over 16 to 4.1 under perturbation. Qualitative results further demonstrate improved clinical reasoning for disease detection and structural abnormality assessment. These findings show that explicitly modeling robustness during pre-training leads to more reliable and transferable medical vision-language representations for real-world deployment.
Abstract (translated)
医学视觉-语言模型在对医疗图像和临床文本进行联合推理方面展示了强大的潜力，但其性能往往因成像设备、采集协议和报告风格的差异而产生的领域变化（domain shift）而下降。现有的多模态预训练方法很大程度上忽视了鲁棒性问题，并将其视为下游适应任务。在此项研究中，我们提出了Robust Multi-Modal Masked Reconstruction (Robust-MMR)，这是一种自监督预训练框架，它明确地将鲁棒性的目标纳入到了遮蔽视觉-语言学习之中。Robust-MMR集成了不对称扰动感知的屏蔽、领域一致性正则化和模态韧性约束以鼓励领域的不变表示。我们对多个医学视觉-语言基准进行了评估，包括医疗视觉问答（VQA-RAD, SLAKE, VQA-2019）、跨域图像文本分类（MELINDA）以及鲁棒的图像标题检索（ROCO）。在VQA-RAD上，Robust-MMR实现了78.9%的跨领域准确率，比最强基线高出了3.8个百分点，并分别达到了SLAKE和VQA-2019上的74.6%和77.0%准确率。在扰动评估下，Robust-MMR将VQA-RAD的准确性从69.1%提高到了75.6%。对于图像文本分类，在跨领域的MELINDA上精度从70.3%提升到75.2%，而在检索实验中显示了均位序退化率在扰动下的减少，从超过16降至4.1。定性结果进一步展示了改进的临床推理能力，用于疾病检测和结构异常评估。这些发现表明，在预训练期间显式地建模鲁棒性可以为真实世界的部署生成更可靠且可转移的医学视觉-语言表示。
URL
https://arxiv.org/abs/2602.17689
