Abstract
The inherent complexity of video understanding makes it difficult to attribute whether performance gains stem from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, the essential criteria that constitute video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the current landscape of video understanding. In this work, we provide Video-Oasis, a sustainable diagnostic suite designed to systematically evaluate existing evaluations and distill spatio-temporal challenges for video understanding. Our analysis reveals two critical findings: (1) 54% of existing benchmark samples are solvable without visual input or temporal context, and (2) on the remaining samples, state-of-the-art models exhibit performance barely exceeding random guessing. To bridge this gap, we investigate which algorithmic design choices contribute to robust video understanding, providing practical guidelines for future research. We hope our work serves as a standard guideline for benchmark construction and the rigorous evaluation of architecture development. Code is available at this https URL.
Abstract (translated)
视频理解固有的复杂性使得难以归因性能提升究竟源于视觉感知、语言推理还是知识先验。尽管已涌现出许多评估高层推理的基准测试，但构成视频理解的核心标准却 largely 被忽视。我们并未推出另一个新基准，而是退后一步重新审视视频理解的现状。本研究提供 Video-Oasis——一个可持续的诊断套件，旨在系统评估现有评测并提炼视频理解的时空挑战。我们的分析揭示两个关键发现：(1) 现有基准样本中 54% 可在无视觉输入或时序上下文的情况下解决；(2) 在剩余样本上，最先进模型的性能仅略高于随机猜测。为弥合这一差距，我们探究哪些算法设计选择有助于实现鲁棒的视频理解，并为未来研究提供实用指南。我们希望本研究能为基准构建和架构开发的严格评估提供标准指南。代码已公开于此 https URL。
URL
https://arxiv.org/abs/2603.29616
