Abstract
We study the capability of Video-Language (VidL) models in understanding compositions between objects, attributes, actions and their relations. Composition understanding becomes particularly challenging for video data since the compositional relations rapidly change over time in videos. We first build a benchmark named AARO to evaluate composition understanding related to actions on top of spatial concepts. The benchmark is constructed by generating negative texts with incorrect action descriptions for a given video and the model is expected to pair a positive text with its corresponding video. Furthermore, we propose a training method called NAVERO which utilizes video-text data augmented with negative texts to enhance composition understanding. We also develop a negative-augmented visual-language matching loss which is used explicitly to benefit from the generated negative text. We compare NAVERO with other state-of-the-art methods in terms of compositional understanding as well as video-text retrieval performance. NAVERO achieves significant improvement over other methods for both video-language and image-language composition understanding, while maintaining strong performance on traditional text-video retrieval tasks.
Abstract (translated)
我们研究了Video-Language（VidL）模型在理解物体、属性、动作及其关系的组合方面的能力。对于视频数据，组合理解变得特别具有挑战性，因为组合关系在视频中会快速变化。首先，我们构建了一个名为AARO的基准来评估与动作相关的组合理解。基准是通过为给定视频生成错误的动作描述的负向文本来构建的，模型被期望将积极文本与其相应视频进行匹配。此外，我们提出了一个利用视频文本数据增强负向文本的训练方法NAVERO，以增强组合理解。我们还开发了一个负向文本增强的视觉语言匹配损失，该损失用于明确利用生成的负向文本。我们比较NAVERO与其他最先进的 methods在组合理解和视频文本检索性能方面的表现。NAVERO在视频语言和图像语言的组合理解方面取得了显著的改进，同时保持传统文本-视频检索任务的强大性能。
URL
https://arxiv.org/abs/2408.09511
