Abstract
Exploiting fine-grained correspondence and visual-semantic alignments has shown great potential in image-text matching. Generally, recent approaches first employ a cross-modal attention unit to capture latent region-word interactions, and then integrate all the alignments to obtain the final similarity. However, most of them adopt one-time forward association or aggregation strategies with complex architectures or additional information, while ignoring the regulation ability of network feedback. In this paper, we develop two simple but quite effective regulators which efficiently encode the message output to automatically contextualize and aggregate cross-modal representations. Specifically, we propose (i) a Recurrent Correspondence Regulator (RCR) which facilitates the cross-modal attention unit progressively with adaptive attention factors to capture more flexible correspondence, and (ii) a Recurrent Aggregation Regulator (RAR) which adjusts the aggregation weights repeatedly to increasingly emphasize important alignments and dilute unimportant ones. Besides, it is interesting that RCR and RAR are plug-and-play: both of them can be incorporated into many frameworks based on cross-modal interaction to obtain significant benefits, and their cooperation achieves further improvements. Extensive experiments on MSCOCO and Flickr30K datasets validate that they can bring an impressive and consistent R@1 gain on multiple models, confirming the general effectiveness and generalization ability of the proposed methods. Code and pre-trained models are available at: this https URL.
Abstract (translated)
在图像-文本匹配中，利用精细的对应关系和视觉语义对齐展现了巨大的潜力。一般而言，最近的方法先使用跨媒体注意力单元捕捉潜在区域-单词相互作用，然后将它们整合到一起以获得最终相似性。然而，大多数方法采用复杂的架构或附加信息，同时忽略了网络反馈的调节能力。在本文中，我们开发了两个非常简单但非常有效的监管者，它们高效编码了消息输出，自动 contextualize 和聚合跨媒体表示。具体而言，我们提出了 (i) 循环对应监管器(RCR)，它促进跨媒体注意力单元逐步适应注意力因素，以捕捉更灵活的对应关系，以及 (ii) 循环聚合监管器(RAR)，它反复调整聚合权重，越来越强调重要的对齐，并稀释不重要的对齐。此外，有趣的是，RCR 和 RAR是可插拔的：它们都可以被整合到基于跨媒体交互的许多框架中，以获得重大好处，并且它们的合作可以实现进一步的改进。在 MSCOCO 和 Flickr30K 数据集上的广泛实验证实了它们可以在多个模型上产生令人印象深刻且一致的 R@1 增益，确认了所提出方法的一般效率和泛化能力。代码和预训练模型可在 this https URL 中找到。
URL
https://arxiv.org/abs/2303.13371
