Abstract
Adversarial attacks have been fairly explored for computer vision and vision-language models. However, the avenue of adversarial attack for the vision language segmentation models (VLSMs) is still under-explored, especially for medical image analysis. Thus, we have investigated the robustness of VLSMs against adversarial attacks for 2D medical images with different modalities with radiology, photography, and endoscopy. The main idea of this project was to assess the robustness of the fine-tuned VLSMs specially in the medical domain setting to address the high risk scenario. First, we have fine-tuned pre-trained VLSMs for medical image segmentation with adapters. Then, we have employed adversarial attacks -- projected gradient descent (PGD) and fast gradient sign method (FGSM) -- on that fine-tuned model to determine its robustness against adversaries. We have reported models' performance decline to analyze the adversaries' impact. The results exhibit significant drops in the DSC and IoU scores after the introduction of these adversaries. Furthermore, we also explored universal perturbation but were not able to find for the medical images. \footnote{this https URL}
Abstract (translated)
对抗攻击在计算机视觉和视觉-语言模型领域已得到了相当程度的研究。然而，针对视觉语言分割模型（VLSMs）的对抗攻击研究仍然较少，特别是在医学图像分析方面。因此，我们调查了不同模态2D医学图像上经过微调的VLSMs面对对抗攻击时的鲁棒性，尤其是在放射学、摄影和内窥镜检查领域。该项目的主要目的是评估在特定医疗场景下针对高风险情况下的微调VLSM的稳健性。 首先，我们在医学图像分割任务中使用适配器对预训练的VLSMs进行了微调。然后，我们对该模型应用了对抗攻击——包括投影梯度下降（PGD）和快速梯度符号方法（FGSM），以评估其在面对对手时的鲁棒性。我们报告了模型性能的下降情况来分析这些对抗样本的影响。实验结果显示，在引入这些对抗样本后，Dice相似系数（DSC）和交并比（IoU）得分有显著下降。 此外，我们也探索了一种通用扰动方法但未能在医学图像上找到有效应用。\footnote{此链接指向的是原始研究或相关资源的URL，但由于没有提供具体的URL地址，这里仅以说明性文本形式表达。} 这段翻译详细描述了项目的目标、方法和结果，并指出对抗攻击对VLSMs在医疗图像分析中的影响显著，但通用扰动法的应用尚未成功实现。
URL
https://arxiv.org/abs/2505.02971
