Abstract
Recent progress has shown great potential of visual prompt tuning (VPT) when adapting pre-trained vision transformers to various downstream tasks. However, most existing solutions independently optimize prompts at each layer, thereby neglecting the usage of task-relevant information encoded in prompt tokens across layers. Additionally, existing prompt structures are prone to interference from task-irrelevant noise in input images, which can do harm to the sharing of task-relevant information. In this paper, we propose a novel VPT approach, \textbf{iVPT}. It innovatively incorporates a cross-layer dynamic connection (CDC) for input prompt tokens from adjacent layers, enabling effective sharing of task-relevant information. Furthermore, we design a dynamic aggregation (DA) module that facilitates selective sharing of information between layers. The combination of CDC and DA enhances the flexibility of the attention process within the VPT framework. Building upon these foundations, iVPT introduces an attentive reinforcement (AR) mechanism, by automatically identifying salient image tokens, which are further enhanced by prompt tokens in an additive manner. Extensive experiments on 24 image classification and semantic segmentation benchmarks clearly demonstrate the advantage of the proposed iVPT, compared to the state-of-the-art counterparts.
Abstract (translated)
近年来，将预训练的视觉 transformer 适应各种下游任务显示出巨大的潜力。然而，大多数现有解决方案在每层独立优化提示，从而忽视了提示词在层间编码的任务相关信息。此外，现有的提示结构容易受到输入图像中与任务无关的噪声的干扰，这会损害任务相关信息的共享。在本文中，我们提出了一个新颖的 VPT 方法：iVPT。它创新地引入了跨层动态连接（CDC）来共享相邻层输入提示的 task-relevant information，实现有效共享任务相关信息。此外，我们还设计了一个动态聚合（DA）模块，促进层间信息的选择性共享。CDC 和 DA 的结合增强了 VPT 框架内注意过程的灵活性。在此基础上，iVPT 引入了注意性的强化（AR）机制，通过自适应地识别显眼的图像词元，进一步通过提示词进行增强。在 24 个图像分类和语义分割基准上的实验表明，与最先进的对照相比，iVPT 具有显著的优势。
URL
https://arxiv.org/abs/2404.05207
