Abstract
Streaming video question answering (Streaming Video QA) poses distinct challenges for multimodal large language models (MLLMs), as video frames arrive sequentially and user queries can be issued at arbitrary time points. Existing solutions relying on fixed-size memory or naive compression often suffer from context loss or memory overflow, limiting their effectiveness in long-form, real-time scenarios. We present Vista, a novel framework for scene-aware streaming video QA that enables efficient and scalable reasoning over continuous video streams. The innovation of Vista can be summarized in three aspects: (1) scene-aware segmentation, where Vista dynamically clusters incoming frames into temporally and visually coherent scene units; (2) scene-aware compression, where each scene is compressed into a compact token representation and stored in GPU memory for efficient index-based retrieval, while full-resolution frames are offloaded to CPU memory; and (3) scene-aware recall, where relevant scenes are selectively recalled and reintegrated into the model input upon receiving a query, enabling both efficiency and completeness. Vista is model-agnostic and integrates seamlessly with a variety of vision-language backbones, enabling long-context reasoning without compromising latency or memory efficiency. Extensive experiments on StreamingBench demonstrate that Vista achieves state-of-the-art performance, establishing a strong baseline for real-world streaming video understanding.
Abstract (translated)
流媒体视频问答（Streaming Video QA）对多模态大型语言模型（Multimodal Large Language Models，MLLMs）提出了独特的挑战。由于视频帧是按顺序到达的，并且用户可以在任意时间点提出问题，因此现有的解决方案依赖于固定大小的记忆或简单压缩方法往往会导致上下文丢失或内存溢出，在长篇幅、实时场景中效果受限。我们介绍了一个名为Vista的新框架，该框架专门用于基于场景感知的流媒体视频问答，使对连续视频流的有效和可扩展推理成为可能。 Vista 的创新之处可以总结为三个方面： 1. **基于场景的分割**：Vista 动态地将传入的帧聚类成时间和视觉上连贯的场景单元。 2. **基于场景的压缩**：每个场景都被压缩成一个紧凑的令牌表示并存储在 GPU 内存中，以实现高效的索引检索。同时，全分辨率帧被卸载到 CPU 内存。 3. **基于场景的召回**：在接受到查询时，相关场景会被选择性地召回并重新整合到模型输入中，从而保证效率和完整性。 Vista 是一种与多种视觉语言骨干网无缝集成、无需牺牲延迟或内存效率就能实现长上下文推理的方法。在 StreamingBench 上进行的广泛实验表明，Vista 达到了最先进的性能，并为实际流媒体视频理解建立了强有力的基准。
URL
https://arxiv.org/abs/2602.08448
