Abstract
Contrastive vision-language models excel in zero-shot image recognition but face challenges in few-shot scenarios due to computationally intensive offline fine-tuning using prompt learning, which risks overfitting. To overcome these limitations, we propose Attn-Adapter, a novel online few-shot learning framework that enhances CLIP's adaptability via a dual attention mechanism. Our design incorporates dataset-specific information through two components: the Memory Attn-Adapter, which refines category embeddings using support examples, and the Local-Global Attn-Adapter, which enriches image embeddings by integrating local and global features. This architecture enables dynamic adaptation from a few labeled samples without retraining the base model. Attn-Adapter outperforms state-of-the-art methods in cross-category and cross-dataset generalization, maintaining efficient inference and scaling across CLIP backbones.
Abstract (translated)
对比视觉语言模型在零样本图像识别方面表现出色，但在少量样本场景中却面临挑战。由于使用提示学习进行离线微调计算成本高昂，并且存在过拟合的风险，这些问题限制了它们的应用范围。为了克服这些局限性，我们提出了一种新颖的在线少量样本学习框架——Attn-Adapter。该框架通过双注意力机制增强了CLIP（对比语言-图像预训练模型）的适应能力。 我们的设计包括两个组件，旨在利用特定数据集的信息：记忆Attn-Adapter使用支持示例来优化类别嵌入；全局-局部Attn-Adapter则通过融合局部和全局特征来丰富图像嵌入。这种架构使得从少量标注样本进行动态适应成为可能，而无需重新训练基础模型。 实验结果显示，Attn-Adapter在跨类别的泛化能力和跨数据集的迁移学习上超越了当前最先进的方法，并且能够在保持高效推理的同时，在不同的CLIP骨干网络中实现规模扩展。
URL
https://arxiv.org/abs/2509.03895
