Abstract
AI-driven video analytics has become increasingly pivotal across diverse domains. However, existing systems are often constrained to specific, predefined tasks, limiting their adaptability in open-ended analytical scenarios. The recent emergence of Video-Language Models (VLMs) as transformative technologies offers significant potential for enabling open-ended video understanding, reasoning, and analytics. Nevertheless, their limited context windows present challenges when processing ultra-long video content, which is prevalent in real-world applications. To address this, we introduce AVA, a VLM-powered system designed for open-ended, advanced video analytics. AVA incorporates two key innovations: (1) the near real-time construction of Event Knowledge Graphs (EKGs) for efficient indexing of long or continuous video streams, and (2) an agentic retrieval-generation mechanism that leverages EKGs to handle complex and diverse queries. Comprehensive evaluations on public benchmarks, LVBench and VideoMME-Long, demonstrate that AVA achieves state-of-the-art performance, attaining 62.3% and 64.1% accuracy, respectively, significantly surpassing existing VLM and video Retrieval-Augmented Generation (RAG) systems. Furthermore, to evaluate video analytics in ultra-long and open-world video scenarios, we introduce a new benchmark, AVA-100. This benchmark comprises 8 videos, each exceeding 10 hours in duration, along with 120 manually annotated, diverse, and complex question-answer pairs. On AVA-100, AVA achieves top-tier performance with an accuracy of 75.8%.
Abstract (translated)
AI驱动的视频分析在各个领域中变得越来越重要。然而，现有的系统通常局限于特定、预定义的任务，这限制了它们在开放性分析场景中的适应能力。最近出现的视频-语言模型（VLMs）作为变革技术，为实现开放性的视频理解、推理和分析提供了巨大潜力。然而，这些模型有限的上下文窗口在处理现实应用中常见的超长视频内容时遇到了挑战。为此，我们引入了AVA系统，这是一个基于VLM的高级视频分析解决方案，旨在应对开放性场景的需求。AVA包含了两个关键创新：（1）几乎实时构建事件知识图谱（EKGs），以高效地索引长时间或连续播放的视频流；（2）一种代理检索-生成机制，利用EKG处理复杂多样的查询。 在公共基准测试LVBench和VideoMME-Long上的全面评估显示，AVA达到了行业领先的性能水平，在这两个数据集上分别获得了62.3%和64.1%的准确率，远超现有的VLM系统以及视频检索增强生成（RAG）系统的成绩。此外，为了在超长及开放世界视频场景中评测视频分析技术，我们提出了一项新的基准测试——AVA-100。该基准包括8个视频片段，每个视频超过10小时，并配有总共120对经过人工标注的多样化且复杂的问答组合。 在AVA-100上，AVA以75.8%的准确率取得了顶级性能表现。
URL
https://arxiv.org/abs/2505.00254
