Abstract
Real-time, continuous understanding of visual signals is essential for real-world interactive AI applications, and poses a fundamental system-level challenge. Existing research on streaming video understanding, however, typically focuses on isolated aspects such as question-answering accuracy under limited visual context or improvements in encoding efficiency, while largely overlooking practical deployability under realistic resource constraints. To bridge this gap, we introduce StreamingEval, a unified evaluation framework for assessing the streaming video understanding capabilities of Video-LLMs under realistic constraints. StreamingEval benchmarks both mainstream offline models and recent online video models under a standardized protocol, explicitly characterizing the trade-off between efficiency, storage and accuracy. Specifically, we adopt a fixed-capacity memory bank to normalize accessible historical visual context, and jointly evaluate visual encoding efficiency, text decoding latency, and task performance to quantify overall system deployability. Extensive experiments across multiple datasets reveal substantial gaps between current Video-LLMs and the requirements of realistic streaming applications, providing a systematic basis for future research in this direction. Codes will be released at this https URL.
Abstract (translated)
实时、连续地理解视觉信号对于现实世界的交互式AI应用至关重要，并构成了一个根本性的系统级挑战。然而，现有关于流式视频理解的研究通常局限于孤立层面，例如在有限视觉上下文下的问答准确性或编码效率的提升，而很大程度上忽视了现实资源约束下的实际可部署性。为弥合这一差距，我们提出了StreamingEval——一个在现实约束下评估视频大语言模型流式视频理解能力的统一评估框架。StreamingEval通过标准化协议对主流离线模型和近期在线视频模型进行基准测试，明确刻画了效率、存储与准确性之间的权衡关系。具体而言，我们采用固定容量的记忆库来规范化可访问的历史视觉上下文，并联合评估视觉编码效率、文本解码延迟及任务性能，以量化整体系统可部署性。在多个数据集上的大量实验表明，当前视频大语言模型与现实流式应用需求之间存在显著差距，为未来该方向研究提供了系统性基础。代码将在此https URL发布。
URL
https://arxiv.org/abs/2603.21493
