Abstract
Streaming video understanding requires models to robustly encode, store, and retrieve information from a continuous video stream to support accurate video question answering (VQA). Existing state-of-the-art approaches rely on key-value caching to accumulate frame-level information over time, but use a limited number of tokens per frame, leading to the loss of fine-grained visual details. In this work, we propose scaling the token budget to enable more granular spatiotemporal understanding and reasoning. First, we find that current methods are ill-equipped to handle dense streams: their feature encoding causes query-frame similarity scores to increase over time, biasing retrieval toward later frames. To address this, we introduce an adaptive selection strategy that reduces token redundancy while preserving local spatiotemporal information. We further propose a training-free retrieval mixture-of-experts that leverages external models to better identify relevant frames. Our method, MemStream, achieves +8.0% on CG-Bench, +8.5% on LVBench, and +2.4% on VideoMME (Long) over ReKV with Qwen2.5-VL-7B.
Abstract (translated)
流媒体视频理解需要模型能够稳健地编码、存储和从连续的视频流中检索信息，以支持准确的视频问答（VQA）。现有的最先进的方法依赖于键值缓存来随着时间积累帧级信息，但每帧使用的令牌数量有限，导致丢失了细微的视觉细节。在本工作中，我们提出增加令牌预算，以便实现更细致的空间和时间理解与推理。首先，我们发现当前的方法处理密集流时存在不足：它们的特征编码会导致查询-帧相似度分数随时间增加，偏向于检索较晚的帧。为解决这一问题，我们引入了一种自适应选择策略，在减少令牌冗余的同时保留局部时空信息。此外，我们提出了一种无需训练的混合专家检索方法，利用外部模型更好地识别相关帧。我们的方法MemStream在CG-Bench上比ReKV with Qwen2.5-VL-7B提高了8.0%，在LVBench上提高了8.5%，在VideoMME（Long）上提高了2.4%。
URL
https://arxiv.org/abs/2602.18434
