Abstract
A recent trend in multimodal retrieval is related to postprocessing test set results via the dual-softmax loss (DSL). While this approach can bring significant improvements, it usually presumes that an entire matrix of test samples is available as DSL input. This work introduces a new postprocessing approach based on Sinkhorn transformations that outperforms DSL. Further, we propose a new postprocessing setting that does not require access to multiple test queries. We show that our approach can significantly improve the results of state of the art models such as CLIP4Clip, BLIP, X-CLIP, and DRL, thus achieving a new state-of-the-art on several standard text-video retrieval datasets both with access to the entire test set and in the single-query setting.
Abstract (translated)
近期在多模态检索的一个趋势是使用双软max损失（DSL）通过双软max损失（DSL）对多模态检索测试集结果进行后处理。虽然这种方法可以带来显著的改进，但它通常假定整个测试样本矩阵作为DSL输入。本文介绍了一种基于Sinkhorn变换的新后处理方法，该方法优于DSL。此外，我们提出了一个不需要访问多个测试查询的新后处理设置。我们证明了我们的方法可以显著提高诸如CLIP4Clip、BLIP、X-CLIP和DRL等先进模型的性能，从而在访问整个测试集和在单查询设置上实现新的最先进水平。
URL
https://arxiv.org/abs/2311.08143
