Abstract
Pre-trained large vision-language models (VLMs) like CLIP demonstrate impressive generalization ability. Existing prompt-based and adapter-based works have made significant progress in fine-tuning VLMs but still face the challenges of maintaining strong generalization abilities, particularly towards unseen new classes. This limitation partly arises from these methods treating all tokens of the image and text encoder equally, which can lead to overfitting on less informative features (e.g., background noise, template words) and degrade the general representations that are crucial for novel concept recognition. To address this issue, we propose Dynamic Rank Adaptation (DRA), a novel adapter variant method, designed specifically to enhance new class generalization. DRA dynamically allocates adaptation ranks based on the importance of features during training to preserve general knowledge. DRA first employs token importance grouping, using sequence attention to evaluate and group tokens by their importance. Then, we adopt rank adaptation according to the importance of each token group dynamically by assigning higher feature ranks to the more important tokens. Also, we design a new channel response mechanism to prioritize the preservation and adaptation of feature channels identified as the most informative for each instance. In addition, a L1 regularization term is introduced to stabilize the training. Extensive experiments demonstrate the effectiveness and superiority of our proposed DRA over existing works, especially on enhancing the performance of new classes on various benchmarks, including base-new classes, cross-datasets evaluation and domain generalization. The source code will be published after the paper is received.
Abstract (translated)
预训练的大规模视觉-语言模型（VLM）如CLIP展现了令人印象深刻的泛化能力。现有的基于提示和适配器的工作在微调VLM方面取得了显著进展，但仍面临保持强泛化能力的挑战，特别是在未见过的新类别上。这一限制部分源于这些方法对图像和文本编码器中的所有标记一视同仁，这可能导致过度拟合于信息量较少的特征（例如背景噪音、模板词汇），从而损害对于新概念识别至关重要的通用表示。 为了解决这个问题，我们提出了动态秩适配（DRA）——一种新型的适配器变体方法，旨在特别增强对新类别的泛化能力。DRA在训练期间根据特征的重要性动态分配适应秩，以保留通用知识。首先，DRA采用标记重要性分组的方法，利用序列注意力来评估并按重要性将标记进行分组。然后，我们根据每个标记组的重要程度动态地采用秩适配，给更重要的令牌分配更高的特征等级。此外，我们设计了一种新的通道响应机制，优先保留和适应每例中识别为最具有信息量的特征通道。另外，引入L1正则化项以稳定训练过程。 广泛的实验展示了我们的DRA方法在现有工作中的有效性和优越性，特别是在增强各种基准上新类别的性能方面，包括基础-新类别、跨数据集评估和领域泛化。论文接收后将公开源代码。
URL
https://arxiv.org/abs/2507.05668
