Abstract
Segment anything model (SAM) has presented impressive objectness identification capability with the idea of prompt learning and a new collected large-scale dataset. Given a prompt (e.g., points, bounding boxes, or masks) and an input image, SAM is able to generate valid segment masks for all objects indicated by the prompts, presenting high generalization across diverse scenarios and being a general method for zero-shot transfer to downstream vision tasks. Nevertheless, it remains unclear whether SAM may introduce errors in certain threatening scenarios. Clarifying this is of significant importance for applications that require robustness, such as autonomous vehicles. In this paper, we aim to study the testing-time robustness of SAM under adversarial scenarios and common corruptions. To this end, we first build a testing-time robustness evaluation benchmark for SAM by integrating existing public datasets. Second, we extend representative adversarial attacks against SAM and study the influence of different prompts on robustness. Third, we study the robustness of SAM under diverse corruption types by evaluating SAM on corrupted datasets with different prompts. With experiments conducted on SA-1B and KITTI datasets, we find that SAM exhibits remarkable robustness against various corruptions, except for blur-related corruption. Furthermore, SAM remains susceptible to adversarial attacks, particularly when subjected to PGD and BIM attacks. We think such a comprehensive study could highlight the importance of the robustness issues of SAM and trigger a series of new tasks for SAM as well as downstream vision tasks.
Abstract (translated)
Segment anything模型(Sam)以prompt learning和收集大型数据集的新想法，展示了令人印象深刻的对象识别能力。给定prompt(例如点、边界框或掩膜)和输入图像，Sam能够生成所有由prompt指示的对象的有效分块Mask，在各种情况下表现出高泛化能力，是直接转移到后续视觉任务通用的方法。然而，仍然不清楚Sam在某些威胁情况下可能会引入错误。澄清这一点对于需要鲁棒性的应用程序，例如自动驾驶车辆等具有重要意义。在本文中，我们旨在研究Sam在对抗场景和常见腐败情况下的测试时鲁棒性。为此，我们首先建立了Sam的测试时鲁棒性评估基准，通过整合现有公共数据集。其次，我们扩展了代表性的对抗攻击对Sam进行研究，并探讨不同prompt对鲁棒性的影响了。通过在SAB和KITTI数据集上进行实验，我们发现Sam表现出对多种腐败的 remarkable 鲁棒性，除了与模糊相关的腐败。此外，Sam仍然容易受到对抗攻击，特别是在受到PGD和BIM攻击的情况下。我们认为这种全面研究可以强调Sam的鲁棒性问题的重要性，并触发一系列新的任务，为Sam以及后续视觉任务。
URL
https://arxiv.org/abs/2305.16220
