Abstract
Vision-language models (VLMs) pre-trained on natural image and language data, such as CLIP, have exhibited significant potential in few-shot image recognition tasks, leading to development of various efficient transfer learning methods. These methods exploit inherent pre-learned knowledge in VLMs and have achieved strong performance on standard image datasets. However, their effectiveness is often limited when confronted with cross-domain tasks where imaging domains differ from natural images. To address this limitation, we propose Consistency-guided Multi-view Collaborative Optimization (CoMuCo), a novel fine-tuning strategy for VLMs. This strategy employs two functionally complementary expert modules to extract multi-view features, while incorporating prior knowledge-based consistency constraints and information geometry-based consensus mechanisms to enhance the robustness of feature learning. Additionally, a new cross-domain few-shot benchmark is established to help comprehensively evaluate methods on imaging domains distinct from natural images. Extensive empirical evaluations on both existing and newly proposed benchmarks suggest CoMuCo consistently outperforms current methods in few-shot tasks. The code and benchmark will be released.
Abstract (translated)
预训练于自然图像和语言数据上的视觉-语言模型（VLMs，如CLIP），在少量样本的图像识别任务中展现了巨大的潜力，并促进了各种高效迁移学习方法的发展。这些方法利用了VLMs内部预先学习的知识，在标准图像数据集上取得了强劲的表现。然而，当面对跨域任务时，即成像领域与自然图像不同，它们的有效性往往受到限制。 为了解决这一局限性，我们提出了一种新的微调策略：一致性引导的多视图协同优化（CoMuCo）。该策略采用两个功能互补的专家模块来提取多视图特征，并通过引入基于先验知识的一致性约束以及基于信息几何的共识机制，增强了特征学习的鲁棒性。此外，我们还建立了一个新的跨域少量样本基准测试集，以便全面评估那些与自然图像领域不同的成像领域的方法。 在现有和新提出的基准上进行广泛的经验评测表明，CoMuCo在少量样本任务中始终优于当前的方法。代码和基准将在不久的将来发布。
URL
https://arxiv.org/abs/2508.12861
