Abstract
A more robust and holistic language-video representation is the key to pushing video understanding forward. Despite the improvement in training strategies, the quality of the language-video dataset is less attention to. The current plain and simple text descriptions and the visual-only focus for the language-video tasks result in a limited capacity in real-world natural language video retrieval tasks where queries are much more complex. This paper introduces a method to automatically enhance video-language datasets, making them more modality and context-aware for more sophisticated representation learning needs, hence helping all downstream tasks. Our multifaceted video captioning method captures entities, actions, speech transcripts, aesthetics, and emotional cues, providing detailed and correlating information from the text side to the video side for training. We also develop an agent-like strategy using language models to generate high-quality, factual textual descriptions, reducing human intervention and enabling scalability. The method's effectiveness in improving language-video representation is evaluated through text-video retrieval using the MSR-VTT dataset and several multi-modal retrieval models.
Abstract (translated)
更健壮和完整的语言-视频表示是推动视频理解前进的关键。尽管在训练策略的提高方面取得了改善，但语言-视频数据集的质量并没有得到足够的关注。当前的简单和直接的文本描述和仅关注语言-视频任务的视觉化，导致在现实世界的自然语言视频检索任务中，查询非常复杂，查询能力有限。本文介绍了一种自动增强视频-语言数据集的方法，使其更具模态和上下文意识，以满足更复杂表示学习的需求，从而帮助所有下游任务。我们的多方面视频摘要方法捕捉实体、动作、语音转录、美学和情感线索，提供文本侧到视频侧的详细和相关的信息，以进行训练。我们还使用语言模型开发了一个类似代理的策略，以生成高质量的事实性文本描述，减少人类干预并实现可扩展性。通过评估通过MSR-VTT数据集进行的文本-视频检索，以及几种多模态检索模型，来评估方法改善语言-视频表示的有效性。
URL
https://arxiv.org/abs/2406.13809
