Abstract
Multi-modal large language models (MLLMs) have advanced general-purpose video understanding but struggle with long, high-resolution videos -- they process every pixel equally in their vision transformers (ViTs) or LLMs despite significant spatiotemporal redundancy. We introduce AutoGaze, a lightweight module that removes redundant patches before processed by a ViT or an MLLM. Trained with next-token prediction and reinforcement learning, AutoGaze autoregressively selects a minimal set of multi-scale patches that can reconstruct the video within a user-specified error threshold, eliminating redundancy while preserving information. Empirically, AutoGaze reduces visual tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x, enabling scaling MLLMs to 1K-frame 4K-resolution videos and achieving superior results on video benchmarks (e.g., 67.0% on VideoMME). Furthermore, we introduce HLVid: the first high-resolution, long-form video QA benchmark with 5-minute 4K-resolution videos, where an MLLM scaled with AutoGaze improves over the baseline by 10.1% and outperforms the previous best MLLM by 4.5%. Project page: this https URL.
Abstract (translated)
多模态大型语言模型（MLLMs）在通用视频理解方面取得了显著进展，但在处理长、高分辨率视频时却面临挑战——它们的视觉变换器（ViTs）或大型语言模型对每个像素进行同等处理，尽管存在显著的空间和时间冗余。为了解决这一问题，我们引入了AutoGaze模块，它可以在视频被输入到ViT或MLLM之前去除冗余的图像块。通过下一令牌预测和强化学习训练，AutoGaze能够自回归地选择一组最小化尺度的补丁集，在用户指定的误差阈值内重建视频，从而在减少冗余的同时保留信息。 从实验结果来看，AutoGaze可以将视觉标记减少4到100倍，并使ViTs和MLLMs的速度提升最多19倍，使得MLLMs能够扩展至处理包含1K帧、4K分辨率的视频。此外，在诸如VideoMME等视频基准测试中取得了优于现有方法的表现（例如在VideoMME中的得分达到了67.0%）。 我们还推出了HLVid：首个高分辨率长格式视频问答基准，它包含了5分钟时长、4K分辨率的视频数据集。在这个新的基准上，使用AutoGaze扩展后的MLLM相对于基线模型提高了10.1%，并且相比之前的最佳MLLM性能提升了4.5%。 项目页面链接：[此处插入具体网址]
URL
https://arxiv.org/abs/2603.12254
