Abstract
Among the ever-evolving development of vision-language models, contrastive language-image pretraining (CLIP) has set new benchmarks in many downstream tasks such as zero-shot classifications by leveraging self-supervised contrastive learning on large amounts of text-image pairs. However, its dependency on rigid one-to-one mappings overlooks the complex and often multifaceted relationships between and within texts and images. To this end, we introduce RankCLIP, a novel pretraining method that extends beyond the rigid one-to-one matching framework of CLIP and its variants. By leveraging both in-modal and cross-modal ranking consistency, RankCLIP improves the alignment process, enabling it to capture the nuanced many-to-many relationships between and within each modality. Through comprehensive experiments, we demonstrate the enhanced capability of RankCLIP to effectively improve performance across various downstream tasks, notably achieving significant gains in zero-shot classifications over state-of-the-art methods, underscoring the potential of RankCLIP in further advancing vision-language pretraining.
Abstract (translated)
在视觉语言模型不断演变的背景下，对比语言-图像预训练（CLIP）已经在许多下游任务中达到了新的基准，例如利用大规模文本-图像对的自监督对比学习来进行零散shot分类。然而，它对固定一对一映射的依赖性忽视了文本和图像之间以及文本内部复杂多面之间的关系。为此，我们引入了RankCLIP，一种超越了CLIP及其变体的 rigid one-to-one matching 框架的新预训练方法。通过利用 both in-modal 和 cross-modal ranking consistency, RankCLIP 改善了alignment 过程，使其能够捕捉每个模态之间微妙的 many-to-many 关系。通过全面的实验，我们证明了 RankCLIP 在各种下游任务中的增强能力，特别是在零散shot分类方面取得了显著的进步，突出了 RankCLIP 在进一步推动视觉语言预训练方面的潜在可能性。
URL
https://arxiv.org/abs/2404.09387
