Abstract
Composed Image Retrieval (CIR) is a task that retrieves images similar to a query, based on a provided textual modification. Current techniques rely on supervised learning for CIR models using labeled triplets of the reference image, text, target image. These specific triplets are not as commonly available as simple image-text pairs, limiting the widespread use of CIR and its scalability. On the other hand, zero-shot CIR can be relatively easily trained with image-caption pairs without considering the image-to-image relation, but this approach tends to yield lower accuracy. We propose a new semi-supervised CIR approach where we search for a reference and its related target images in auxiliary data and learn our large language model-based Visual Delta Generator (VDG) to generate text describing the visual difference (i.e., visual delta) between the two. VDG, equipped with fluent language knowledge and being model agnostic, can generate pseudo triplets to boost the performance of CIR models. Our approach significantly improves the existing supervised learning approaches and achieves state-of-the-art results on the CIR benchmarks.
Abstract (translated)
组合图像检索（CIR）是一个根据给定文本修改查询图像的任务。目前的方法依赖于有标签的三元组来训练CIR模型，这些三元组并不像简单的图像-文本对那么常见，从而限制了CIR的广泛应用和其可扩展性。另一方面，零散式CIR可以通过图像描述性对无监督训练进行相对容易的实现，但不考虑图像之间的关系，这种方法往往导致较低的准确性。我们提出了一种新的半监督CIR方法，其中我们在辅助数据中寻找参考图像及其相关目标图像，并使用基于大型语言模型（VDG）生成描述两个视觉差异（即视觉差）的文本。VDG，拥有流畅的语义知识，且对模型无依赖，可以生成伪三元组来提高CIR模型的性能。我们的方法显著提高了现有监督学习方法，并在CIR基准测试中实现了最先进的性能。
URL
https://arxiv.org/abs/2404.15516
