Abstract
Pre-trained Vision-Language Models (VLMs), \textit{e.g.} CLIP, have become essential tools in multimodal transfer learning. However, fine-tuning VLMs in few-shot scenarios poses significant challenges in balancing task-specific adaptation and generalization in the obtained model. Meanwhile, current researches have predominantly focused on prompt-based adaptation methods, leaving adapter-based approaches underexplored and revealing notable performance gaps. To address these challenges, we introduce a novel Reconstruction-based Multimodal Adapter (RMAdapter), which leverages a dual-branch architecture. Unlike conventional single-branch adapters, RMAdapter consists of: (1) an adaptation branch that injects task-specific knowledge through parameter-efficient fine-tuning, and (2) a reconstruction branch that preserves general knowledge by reconstructing latent space features back into the original feature space. This design facilitates a dynamic balance between general and task-specific knowledge. Importantly, although RMAdapter introduces an additional reconstruction branch, it is carefully optimized to remain lightweight. By computing reconstruction loss locally at each layer and sharing projection modules, the overall computational overhead is kept minimal. A consistency constraint is also incorporated to better regulate the trade-off between discriminability and generalization. We comprehensively evaluate the effectiveness of RMAdapter on three representative tasks: generalization to new categories, generalization to new target datasets, and domain generalization. Without relying on data augmentation or duplicate prompt designs, our RMAdapter consistently outperforms state-of-the-art approaches across all evaluation metrics.
Abstract (translated)
预训练的视觉-语言模型（VLMs），例如CLIP，已成为多模态迁移学习中的重要工具。然而，在少量样本场景中微调VLM面临着在特定任务适应和泛化能力之间保持平衡的巨大挑战。目前的研究主要集中在基于提示的方法上，而适配器（adapter）方法却未得到充分探索，并且表现出显著的性能差距。为了解决这些挑战，我们引入了一种新颖的基于重构的多模态适配器（RMAdapter），它利用了双分支架构。不同于传统的单分支适配器，RMAdapter由以下两部分组成：(1) 一个通过参数高效的微调注入特定任务知识的适应分支；(2) 一个通过将潜在空间特征重构回原始特征空间来保留通用知识的重构分支。这种设计有助于在通用性和特定任务知识之间动态平衡。值得注意的是，尽管RMAdapter引入了额外的重构分支，但其经过精心优化以保持轻量化。通过在每一层计算本地重构损失并共享投影模块，整体计算开销被最小化。此外还融入了一致性约束来更好地调节判别能力与泛化的折衷。 我们对三个代表性任务进行了全面评估：新类别的泛化、新的目标数据集的泛化以及领域泛化。在不依赖于数据增强或重复提示设计的情况下，我们的RMAdapter在所有评价指标上始终优于当前最先进的方法。
URL
https://arxiv.org/abs/2512.06811
