Abstract
The ability of perceiving fine-grained spatial and temporal information is crucial for video-language retrieval. However, the existing video retrieval benchmarks, such as MSRVTT and MSVD, fail to efficiently evaluate the fine-grained retrieval ability of video-language models (VLMs) due to a lack of detailed annotations. To address this problem, we present FIBER, a FIne-grained BEnchmark for text to video Retrieval, containing 1,000 videos sourced from the FineAction dataset. Uniquely, our FIBER benchmark provides detailed human-annotated spatial annotations and temporal annotations for each video, making it possible to independently evaluate the spatial and temporal bias of VLMs on video retrieval task. Besides, we employ a text embedding method to unlock the capability of fine-grained video-language understanding of Multimodal Large Language Models (MLLMs). Surprisingly, the experiment results show that our Video Large Language Encoder (VLLE) performs comparably to CLIP-based models on traditional benchmarks and has a stronger capability of fine-grained representation with lower spatial-temporal bias. Project page: this https URL.
Abstract (translated)
感知细微的时空信息对于视频与语言检索至关重要。然而，现有的视频检索基准测试（如MSRVTT和MSVD）由于缺乏详细的注释而无法有效评估视频-语言模型（VLMs）的细粒度检索能力。为了解决这个问题，我们提出了FIBER，即用于文本到视频检索的细粒度基准测试，该测试包含来自FineAction数据集的1,000个视频。特别的是，我们的FIBER基准提供了每个视频详细的、人工注释的空间标注和时间标注，从而能够独立评估VLMs在视频检索任务中的空间偏差和时间偏差。此外，我们采用了一种文本嵌入方法来解锁多模态大型语言模型（MLLMs）的细粒度视频-语言理解能力。令人惊讶的是，实验结果显示我们的视频大型语言编码器（VLLE）在传统基准测试上的表现与CLIP基线模型相当，并且具有更强的细粒度表示能力和较低的空间时间偏差。 项目页面：[此链接](this https URL)。
URL
https://arxiv.org/abs/2501.00513
