Abstract
The rapid advancement of multimodal large language models has demonstrated impressive capabilities, yet nearly all operate in an offline paradigm, hindering real-time interactivity. Addressing this gap, we introduce the Real-tIme Video intERaction Bench (RIVER Bench), designed for evaluating online video comprehension. RIVER Bench introduces a novel framework comprising Retrospective Memory, Live-Perception, and Proactive Anticipation tasks, closely mimicking interactive dialogues rather than responding to entire videos at once. We conducted detailed annotations using videos from diverse sources and varying lengths, and precisely defined the real-time interactive format. Evaluations across various model categories reveal that while offline models perform well in single question-answering tasks, they struggle with real-time processing. Addressing the limitations of existing models in online video interaction, especially their deficiencies in long-term memory and future perception, we proposed a general improvement method that enables models to interact with users more flexibly in real time. We believe this work will significantly advance the development of real-time interactive video understanding models and inspire future research in this emerging field. Datasets and code are publicly available at this https URL.
Abstract (translated)
多模态大型语言模型的快速进步已经展示了其令人印象深刻的性能，然而几乎所有这些模型都运行在离线模式中，这阻碍了其实时互动能力。为解决这一问题，我们引入了实时视频互动基准测试（RIVER Bench），旨在评估在线视频理解能力。RIVER Bench 引入了一个新颖的框架，包括回顾性记忆、现场感知和前瞻性预测任务，这些任务更加接近于模拟交互对话而不是一次性回应整个视频内容。我们使用来自各种来源且长度不同的视频进行了详细的标注，并精确定义了实时互动格式。 在不同模型类别的评估中发现，尽管离线模型在单一问答任务上表现良好，但在实时处理方面却遇到了困难。针对现有模型在线视频互动中的局限性，特别是其长期记忆和未来感知的不足，我们提出了一种通用改进方法，使模型能够更灵活地进行实时互动。我们认为这项工作将显著推动实时交互式视频理解模型的发展，并激发这一新兴领域的未来研究。 数据集和代码已公开发布在以下链接：[此 URL](https://this-URL)（请替换为实际的网址）。
URL
https://arxiv.org/abs/2603.03985
