Abstract
Given a query consisting of a reference image and a relative caption, Composed Image Retrieval (CIR) aims to retrieve target images visually similar to the reference one while incorporating the changes specified in the relative caption. The reliance of supervised methods on labor-intensive manually labeled datasets hinders their broad applicability. In this work, we introduce a new task, Zero-Shot CIR (ZS-CIR), that addresses CIR without the need for a labeled training dataset. We propose an approach named iSEARLE (improved zero-Shot composEd imAge Retrieval with textuaL invErsion) that involves mapping the visual information of the reference image into a pseudo-word token in CLIP token embedding space and combining it with the relative caption. To foster research on ZS-CIR, we present an open-domain benchmarking dataset named CIRCO (Composed Image Retrieval on Common Objects in context), the first CIR dataset where each query is labeled with multiple ground truths and a semantic categorization. The experimental results illustrate that iSEARLE obtains state-of-the-art performance on three different CIR datasets -- FashionIQ, CIRR, and the proposed CIRCO -- and two additional evaluation settings, namely domain conversion and object composition. The dataset, the code, and the model are publicly available at this https URL.
Abstract (translated)
给定一个由参考图像和相对描述组成的查询，组合图像检索（CIR）旨在通过包含在相对描述中指定的变化来检索与参考图像视觉上相似的目标图像，同时实现这一点。依赖于有监督方法对劳动密集型手动标注数据集的依赖会限制其广泛的适用性。在这项工作中，我们引入了一个新的任务，名为零 shot组合图像检索（ZS-CIR），它不需要有标签的训练数据集来解决组合图像检索（CIR）。我们提出了一个名为iSEARLE（改进零 shot组合图像age检索与文本ual invErsion）的方法，该方法涉及将参考图像的视觉信息映射到CLIP标记嵌入空间中的伪词标记，并将其与相对描述相结合。为了促进对零 shot组合图像检索的研究，我们提出了一个名为CIRCO（在上下文中共同对象图像检索）的开源领域基准数据集，它是第一个每个查询都带有多个地面真实值和语义分类的CIR数据集。实验结果表明，iSEARLE在三个不同的CIR数据集--FashionIQ，CIRR和所提出的CIRCO--上都取得了最先进的性能，同时还取得了另外两个评估设置，即领域转换和对象组合。数据集、代码和模型都可以在https://这个链接上获得。
URL
https://arxiv.org/abs/2405.02951
