Abstract
Automatic identification of events and recurrent behavior analysis are critical for video surveillance. However, most existing content-based video retrieval benchmarks focus on scene-level similarity and do not evaluate the action discrimination required in surveillance. To address this gap, we introduce SOVABench (Surveillance Opposite Vehicle Actions Benchmark), a real-world retrieval benchmark built from surveillance footage and centered on vehicle-related actions. SOVABench defines two evaluation protocols (inter-pair and intra-pair) to assess cross-action discrimination and temporal direction understanding. Although action distinctions are generally intuitive for human observers, our experiments show that they remain challenging for state-of-the-art vision and multimodal models. Leveraging the visual reasoning and instruction-following capabilities of Multimodal Large Language Models (MLLMs), we present a training-free framework for producing interpretable embeddings from MLLM-generated descriptions for both images and videos. The framework achieves strong performance on SOVABench as well as on several spatial and counting benchmarks where contrastive Vision-Language Models often fail. The code, annotations, and instructions to construct the benchmark are publicly available.
Abstract (translated)
自动识别事件和重复行为分析对于视频监控至关重要。然而，大多数现有的基于内容的视频检索基准测试主要关注场景级别的相似性，并且不评估在监控中所需的动作辨别能力。为了弥补这一不足，我们引入了SOVABench（监控相对车辆动作基准），这是一个从监控录像构建的真实世界检索基准，专注于与车辆相关的行动。SOVABench定义了两种评估协议（跨对和内对）来评价跨动作的区分能力和时间方向的理解力。虽然动作之间的区别对于人类观察者来说通常是直观的，但我们的实验表明，这些任务仍然为最先进的视觉和多模态模型带来了挑战。 通过利用多模态大型语言模型（MLLMs）在视觉推理和指令遵循方面的能力，我们提出了一种无需训练框架，用于从MLLM生成的描述中生产出具有解释性的图像和视频嵌入。该框架在SOVABench上实现了强大的性能，并且还在多个空间和计数基准测试中表现出色，这些地方对比度视觉语言模型常常失败。 该项目的代码、标注以及构建基准的说明已公开提供。
URL
https://arxiv.org/abs/2601.04824
