Abstract
While deep learning, including Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), has significantly advanced classification performance, its typical reliance on extensive annotated datasets presents a major obstacle in many practical scenarios where such data is scarce. Vision-language models (VLMs) and transfer learning with pre-trained visual models appear as promising techniques to deal with this problem. This paper proposes a novel zero-shot image classification framework that combines a VLM and a pre-trained visual model within a self-learning cycle. Requiring only the set of class names and no labeled training data, our method utilizes a confidence-based pseudo-labeling strategy to train a lightweight classifier directly on the test data, enabling dynamic adaptation. The VLM identifies high-confidence samples, and the pre-trained visual model enhances their visual representations. These enhanced features then iteratively train the classifier, allowing the system to capture complementary semantic and visual cues without supervision. Notably, our approach avoids VLM fine-tuning and the use of large language models, relying on the visual-only model to reduce the dependence on semantic representation. Experimental evaluations on ten diverse datasets demonstrate that our approach outperforms the baseline zero-shot method.
Abstract (translated)
虽然深度学习（包括卷积神经网络(CNNs)和视觉变换器(ViTs)）显著提升了分类性能，但其通常依赖于大规模的标注数据集，在许多实际场景中存在重大障碍，因为在这些场景中很难获得足够的标注数据。视觉-语言模型(VLMs)和基于预训练视觉模型的迁移学习被视为解决这一问题的有前景的技术。本文提出了一种新的零样本图像分类框架，该框架结合了VLM和一个预训练的视觉模型，并在一个自我学习循环内工作。我们的方法仅需类别名称集合而不需要标注的训练数据，并采用基于置信度的伪标签策略，在测试数据上直接训练轻量级分类器，从而实现动态适应性。VLM识别高置信度样本，而预训练的视觉模型则增强这些样本的视觉表示。这些改进后的特征将迭代地用于训练分类器，使系统能够无监督地捕捉互补的语义和视觉线索。值得注意的是，我们的方法避免了对VLM进行微调以及使用大型语言模型，并依赖于仅基于视觉的模型来减少对语义表达的依赖。 在十个不同数据集上的实验评估表明，与基线零样本方法相比，我们的方法表现更优。
URL
https://arxiv.org/abs/2509.18938
