Abstract
Recent approaches have shown impressive proficiency in extracting and leveraging parametric knowledge from Large-Language Models (LLMs) and Vision-Language Models (VLMs). In this work, we consider how we can improve the identification and retrieval of videos related to complex real-world events by automatically extracting latent parametric knowledge about those events. We present Q2E: a Query-to-Event decomposition method for zero-shot multilingual text-to-video retrieval, adaptable across datasets, domains, LLMs, or VLMs. Our approach demonstrates that we can enhance the understanding of otherwise overly simplified human queries by decomposing the query using the knowledge embedded in LLMs and VLMs. We additionally show how to apply our approach to both visual and speech-based inputs. To combine this varied multimodal knowledge, we adopt entropy-based fusion scoring for zero-shot fusion. Through evaluations on two diverse datasets and multiple retrieval metrics, we demonstrate that Q2E outperforms several state-of-the-art baselines. Our evaluation also shows that integrating audio information can significantly improve text-to-video retrieval. We have released code and data for future research.
Abstract (translated)
近期的研究展示了从大型语言模型（LLMs）和视觉-语言模型（VLMs）中提取参数知识并加以利用的出色能力。在本工作中，我们探讨了如何通过自动抽取关于复杂现实事件的潜在参数化知识来改进视频识别与检索的方法。为此，我们提出了Q2E：一种适用于零样本跨语种文本到视频检索的查询拆解方法（Query-to-Event），此方法具有数据集、领域、LLMs或VLMs间的适应性。我们的方法表明可以通过使用嵌入在LLMs和VLMs中的知识来分解查询，从而增强对原本过于简化的用户查询的理解能力。此外，我们还展示了如何将这种方法应用于视觉与语音输入。 为了整合这种多样化的多模态信息，我们采用了基于熵的融合评分进行零样本融合。通过两个不同的数据集及多个检索指标的评估，证明了Q2E优于几个最新的基准方法。我们的评估结果也表明，结合音频信息可以显著提升文本到视频检索的效果。我们已发布了代码和数据以供未来研究之用。
URL
https://arxiv.org/abs/2506.10202
