Abstract
Understanding the content of events occurring in the video and their inherent temporal logic is crucial for video-text retrieval. However, web-crawled pre-training datasets often lack sufficient event information, and the widely adopted video-level cross-modal contrastive learning also struggles to capture detailed and complex video-text event alignment. To address these challenges, we make improvements from both data and model perspectives. In terms of pre-training data, we focus on supplementing the missing specific event content and event temporal transitions with the proposed event augmentation strategies. Based on the event-augmented data, we construct a novel Event-Aware Video-Text Retrieval model, ie, EA-VTR, which achieves powerful video-text retrieval ability through superior video event awareness. EA-VTR can efficiently encode frame-level and video-level visual representations simultaneously, enabling detailed event content and complex event temporal cross-modal alignment, ultimately enhancing the comprehensive understanding of video events. Our method not only significantly outperforms existing approaches on multiple datasets for Text-to-Video Retrieval and Video Action Recognition tasks, but also demonstrates superior event content perceive ability on Multi-event Video-Text Retrieval and Video Moment Retrieval tasks, as well as outstanding event temporal logic understanding ability on Test of Time task.
Abstract (translated)
理解视频中发生的事件及其固有时间逻辑对于视频文本检索至关重要。然而，从网络爬取的预训练数据集中往往缺乏足够的事件信息，而广泛采用的视频级跨模态对比学习也难以捕捉详细和复杂视频文本事件对齐。为了应对这些挑战，我们在数据和模型方面都进行了改进。在预训练数据方面，我们专注于通过所提出的事件增强策略补充缺失的具体事件内容和事件时间转换。基于事件增强的数据，我们构建了一种新颖的事件感知视频文本检索模型，即EA-VTR，通过卓越的视觉事件意识实现强大的视频文本检索能力。EA-VTR可以同时编码帧级和视频级视觉表示，实现详细的事件内容和复杂的事件时间跨模态对齐，最终提高视频事件的全面理解。我们的方法不仅在多个数据集上的Text-to-Video Retrieval和Video Action Recognition任务上显著优于现有方法，而且在多事件视频文本检索和视频时刻检索任务上表现出卓越的事件内容感知能力，以及出色的时间逻辑理解能力。
URL
https://arxiv.org/abs/2407.07478
