Abstract
The rapid expansion of multimedia content has made accurately retrieving relevant videos from large collections increasingly challenging. Recent advancements in text-video retrieval have focused on cross-modal interactions, large-scale foundation model training, and probabilistic modeling, yet often neglect the crucial user perspective, leading to discrepancies between user queries and the content retrieved. To address this, we introduce MERLIN (Multimodal Embedding Refinement via LLM-based Iterative Navigation), a novel, training-free pipeline that leverages Large Language Models (LLMs) for iterative feedback learning. MERLIN refines query embeddings from a user perspective, enhancing alignment between queries and video content through a dynamic question answering process. Experimental results on datasets like MSR-VTT, MSVD, and ActivityNet demonstrate that MERLIN substantially improves Recall@1, outperforming existing systems and confirming the benefits of integrating LLMs into multimodal retrieval systems for more responsive and context-aware multimedia retrieval.
Abstract (translated)
多媒体内容的迅速扩展使得准确从大型集合中检索相关视频变得越来越具有挑战性。最近在文本视频检索方面的进步主要集中在跨模态交互、大规模基础模型训练和概率建模，然而往往忽视了用户视角，导致用户查询和检索内容之间的差异。为解决这个问题，我们引入了MERLIN（通过LLM的迭代反馈学习优化多模态嵌入），一种新颖的、无需训练的训练通道，它利用大型语言模型（LLMs）进行迭代反馈学习。MERLIN从用户角度优化查询嵌入，通过动态问答过程增强查询和视频内容之间的对齐。在像MSR-VTT、MSVD和ActivityNet这样的数据集上的实验结果表明，MERLIN极大地提高了Recall@1，超过了现有系统，证实了将LLM集成到多模态检索系统中可以实现更响应、更上下文关注意味多媒体检索的优点。
URL
https://arxiv.org/abs/2407.12508
