Abstract
Visual Prompting (VP), an efficient method for transfer learning, has shown its potential in vision tasks. However, previous works focus exclusively on VP from standard source models, it is still unknown how it performs under the scenario of a robust source model: Can the robustness of the source model be successfully inherited? Does VP also encounter the same trade-off between robustness and generalization ability as the source model during this process? If such a trade-off exists, is there a strategy specifically tailored to VP to mitigate this limitation? In this paper, we thoroughly explore these three questions for the first time and provide affirmative answers to them. To mitigate the trade-off faced by VP, we propose a strategy called Prompt Boundary Loosening (PBL). As a lightweight, plug-and-play strategy naturally compatible with VP, PBL effectively ensures the successful inheritance of robustness when the source model is a robust model, while significantly enhancing VP's generalization ability across various downstream datasets. Extensive experiments across various datasets show that our findings are universal and demonstrate the significant benefits of the proposed strategy.
Abstract (translated)
视觉提示法（VP）作为一种高效的迁移学习方法，在视觉任务中展示了其潜力。然而，之前的工作仅关注从标准源模型进行的VP，对于使用鲁棒性更强的源模型时VP的表现尚不清楚：源模型的鲁棒性能否被成功继承？在这一过程中，VP是否会像源模型一样面临鲁棒性和泛化能力之间的权衡？如果这种权衡确实存在，是否有一种专门针对VP的方法来缓解此限制？ 本文首次全面探讨了这三个问题，并给出了肯定的回答。为了缓解VP面临的权衡，我们提出了一种名为“提示边界放松”（PBL）的策略。作为一种轻量级、即插即用且与VP自然兼容的策略，PBL在源模型为鲁棒性模型时有效确保了其鲁棒性的成功继承，并显著提高了VP在各种下游数据集上的泛化能力。跨多个数据集进行的广泛实验表明，我们的发现具有普遍适用性，并展示了所提出策略的重大益处。
URL
https://arxiv.org/abs/2506.06823
