Abstract
Multimodal Large Language Models have demonstrated remarkable capabilities in video understanding, yet face prohibitive computational costs and performance degradation from ''context rot'' due to massive visual token redundancy. Existing compression strategies typically rely on heuristics or fixed transformations that are often decoupled from the downstream task objectives, limiting their adaptability and effectiveness. To address this, we propose SCORE (Surprise-augmented token COmpression via REinforcement learning), a unified framework that learns an adaptive token compression policy. SCORE introduces a lightweight policy network conditioned on a surprise-augmented state representation that incorporates inter-frame residuals to explicitly capture temporal dynamics and motion saliency. We optimize this policy using a group-wise reinforcement learning scheme with a split-advantage estimator, stabilized by a two-stage curriculum transferring from static pseudo-videos to real dynamic videos. Extensive experiments on diverse video understanding benchmarks demonstrate that SCORE significantly outperforms state-of-the-art baselines. Notably, SCORE achieves a 16x prefill speedup while preserving 99.5% of original performance at a 10% retention ratio, offering a scalable solution for efficient long-form video understanding.
Abstract (translated)
多模态大语言模型在视频理解方面展现出卓越能力，但面临因海量视觉标记冗余导致的巨大计算成本及“上下文衰减”引起的性能下降。现有压缩策略通常依赖启发式方法或固定变换，且常与下游任务目标脱节，限制了其适应性和有效性。针对此问题，我们提出SCORE（基于增强惊讶度的强化学习标记压缩）统一框架，该框架可学习自适应标记压缩策略。SCORE引入轻量级策略网络，其条件基于融合帧间残差的增强惊讶度状态表征，以显式捕捉时间动态与运动显著性。我们采用分组强化学习方案结合分裂优势估计器优化该策略，并通过从静态伪视频到真实动态视频的两阶段课程迁移实现稳定。在多种视频理解基准测试上的广泛实验表明，SCORE显著优于最先进基线方法。值得注意的是，在保留10%标记比例时，SCORE实现16倍预填充加速同时保持99.5%原始性能，为高效长视频理解提供了可扩展解决方案。
URL
https://arxiv.org/abs/2603.26365
