Abstract
The recent growth of large foundation models that can easily generate pseudo-labels for huge quantity of unlabeled data makes unsupervised Zero-Shot Cross-Domain Image Retrieval (UZS-CDIR) less relevant. In this paper, we therefore turn our attention to weakly supervised ZS-CDIR (WSZS-CDIR) with noisy pseudo labels generated by large foundation models such as CLIP. To this end, we propose CLAIR to refine the noisy pseudo-labels with a confidence score from the similarity between the CLIP text and image features. Furthermore, we design inter-instance and inter-cluster contrastive losses to encode images into a class-aware latent space, and an inter-domain contrastive loss to alleviate domain discrepancies. We also learn a novel cross-domain mapping function in closed-form, using only CLIP text embeddings to project image features from one domain to another, thereby further aligning the image features for retrieval. Finally, we enhance the zero-shot generalization ability of our CLAIR to handle novel categories by introducing an extra set of learnable prompts. Extensive experiments are carried out using TUBerlin, Sketchy, Quickdraw, and DomainNet zero-shot datasets, where our CLAIR consistently shows superior performance compared to existing state-of-the-art methods.
Abstract (translated)
近期，能够轻易为大量未标记数据生成伪标签的大规模基础模型（如CLIP）的发展使得无监督的零样本跨域图像检索（UZS-CDIR）变得不那么相关了。因此，在这篇论文中，我们转向研究由大型基础模型（如CLIP）生成的带有噪声伪标签的弱监督零样本跨域图像检索（WSZS-CDIR）。为此，我们提出了CLAIR方法，该方法通过计算CLIP文本特征与图像特征之间的相似性来赋予每个伪标签一个置信度分数，并以此精炼掉噪声伪标签。此外，我们设计了实例间的和簇间的对比损失，将图像编码到类感知的潜在空间中，并引入跨域对比损失以减轻领域差异。同时，我们还学习了一种新颖的闭式形式跨域映射函数，仅使用CLIP文本嵌入来将一个领域的图像特征投影到另一个领域，从而进一步对齐用于检索的图像特征。最后，为了增强CLAIR处理新类别的零样本泛化能力，我们在系统中引入了额外的一组可学习的提示词。 通过TUBerlin、Sketchy、Quickdraw和DomainNet零样本数据集进行了广泛的实验，结果显示我们的CLAIR方法在性能上始终优于现有的最先进方法。
URL
https://arxiv.org/abs/2508.12290
