Abstract
Open-vocabulary vision-language models (VLMs) like CLIP, trained using contrastive loss, have emerged as a promising new paradigm for text-to-image retrieval. However, do VLMs understand compound nouns (CNs) (e.g., lab coat) as well as they understand nouns (e.g., lab)? We curate Compun, a novel benchmark with 400 unique and commonly used CNs, to evaluate the effectiveness of VLMs in interpreting CNs. The Compun benchmark challenges a VLM for text-to-image retrieval where, given a text prompt with a CN, the task is to select the correct image that shows the CN among a pair of distractor images that show the constituent nouns that make up the CN. Next, we perform an in-depth analysis to highlight CLIPs' limited understanding of certain types of CNs. Finally, we present an alternative framework that moves beyond hand-written templates for text prompts widely used by CLIP-like models. We employ a Large Language Model to generate multiple diverse captions that include the CN as an object in the scene described by the caption. Our proposed method improves CN understanding of CLIP by 8.25% on Compun. Code and benchmark are available at: this https URL
Abstract (translated)
开放词汇视觉语言模型（VLMs）如CLIP，通过对比损失训练，已经成为文本到图像检索的有前景的新范式。然而，VLMs是否能够理解复合名词（CN）（例如实验室外套）以及它们是否能够理解名词（例如实验室）还有待观察。我们创建了Compun基准，一个包含400个独特且常用CN的新基准，以评估VLMs在解释CN方面的有效性。Compun基准挑战了一个VLM在文本到图像检索的任务，其中，给定一个文本提示，任务是选择一张正确的图像，该图像在一对显示构成CN的干扰图像中显示。接下来，我们进行了深入分析，以突出CLIP对某些类型的CN理解有限。最后，我们提出了一个超越了广泛使用的CLIP类似模型的手写模板的新框架。我们使用一个大型语言模型生成多个具有场景中CN作为对象的多样性 caption。我们的方法在Compun基准上提高了CLIP对CN的理解8.25%。代码和基准您可以在此处查看：https://this URL
URL
https://arxiv.org/abs/2404.00419
