Abstract
Composed Image Retrieval (CIR) task aims to retrieve target images based on reference images and modification texts. Current CIR methods primarily rely on fine-tuning vision-language pre-trained models. However, we find that these approaches commonly suffer from severe overfitting, posing challenges for CIR with limited triplet data. To better understand this issue, we present a systematic study of overfitting in VLP-based CIR, revealing a significant and previously overlooked generalization gap across different models and datasets. Motivated by these findings, we introduce WRF4CIR, a Weight-Regularized Fine-tuning network for CIR. Specifically, during the fine-tuning process, we apply adversarial perturbations to the model weights for regularization, where these perturbations are generated in the opposite direction of gradient descent. Intuitively, WRF4CIR increases the difficulty of fitting the training data, which helps mitigate overfitting in CIR under limited triplet supervision. Extensive experiments on benchmark datasets demonstrate that WRF4CIR significantly narrows the generalization gap and achieves substantial improvements over existing methods.
Abstract (translated)
组合图像检索（CIR）任务旨在基于参考图像和修改文本检索目标图像。当前CIR方法主要依赖对视觉-语言预训练模型进行微调。然而，我们发现这些方法普遍存在严重的过拟合问题，给有限三元组数据下的CIR带来挑战。为深入理解此问题，我们对基于VLP的CIR中的过拟合进行了系统性研究，揭示了一个显著且此前被忽视的跨模型与数据集的泛化差距。受此发现启发，我们提出了WRF4CIR，一种用于CIR的权重正则化微调网络。具体而言，在微调过程中，我们通过对模型权重施加对抗性扰动进行正则化，这些扰动在梯度下降的反方向生成。直观上，WRF4CIR增加了拟合训练数据的难度，从而有助于缓解有限三元组监督下CIR的过拟合问题。在基准数据集上的大量实验表明，WRF4CIR显著缩小了泛化差距，并相较于现有方法取得了实质性提升。
URL
https://arxiv.org/abs/2604.05583
