Abstract
Composed Image Retrieval (CIR) is a complex task that retrieves images using a query, which is configured with an image and a caption that describes desired modifications to that image. Supervised CIR approaches have shown strong performance, but their reliance on expensive manually-annotated datasets restricts their scalability and broader applicability. To address these issues, previous studies have proposed pseudo-word token-based Zero-Shot CIR (ZS-CIR) methods, which utilize a projection module to map images to word tokens. However, we conjecture that this approach has a downside: the projection module distorts the original image representation and confines the resulting composed embeddings to the text-side. In order to resolve this, we introduce a novel ZS-CIR method that uses Spherical Linear Interpolation (Slerp) to directly merge image and text representations by identifying an intermediate embedding of both. Furthermore, we introduce Text-Anchored-Tuning (TAT), a method that fine-tunes the image encoder while keeping the text encoder fixed. TAT closes the modality gap between images and text, making the Slerp process much more effective. Notably, the TAT method is not only efficient in terms of the scale of the training dataset and training time, but it also serves as an excellent initial checkpoint for training supervised CIR models, thereby highlighting its wider potential. The integration of the Slerp-based ZS-CIR with a TAT-tuned model enables our approach to deliver state-of-the-art retrieval performance across CIR benchmarks.
Abstract (translated)
组成图像检索（CIR）是一个复杂的任务，它使用查询来检索图像，该查询配置了一个图像和一个描述对图像所需修改的文本。监督的CIR方法已经展示了强大的性能，但它们依赖于昂贵的手动标注数据集，从而限制了它们的可扩展性和更广泛的适用性。为解决这些问题，以前的研究提出了基于伪词词向量的零 shots CIR（ZS-CIR）方法，该方法利用投影模块将图像映射到词向量。然而，我们推测这种方法的一个缺点是：投影模块扭曲了原始图像表示，并将所得组合嵌入限制在文本侧。为了解决这个问题，我们引入了一种新颖的ZS-CIR方法，该方法使用球面线性插值（Slerp）直接将图像和文本表示合并。此外，我们还引入了文本锚定调整（TAT）方法，该方法在保持文本编码器固定的情况下，对图像编码器进行微调。TAT缩小了图像和文本之间的模式差距，使得Slerp过程更加有效。值得注意的是，TAT方法不仅在训练数据规模和训练时间方面具有效率，而且还可以作为训练监督CIR模型的良好初始检查点，从而突出其更广泛的潜力。将Slerp-based ZS-CIR与TAT调整的模型相结合，使得我们的方法在CIR基准测试中实现了最先进的检索性能。
URL
https://arxiv.org/abs/2405.00571
