Abstract
Long-form video understanding has always been a challenging problem due to the significant redundancy in both temporal and spatial contents. This challenge is further exacerbated by the limited context length of Multimodal Large Language Models (MLLMs). To address this issue, many previous works have attempted to extract key video information, where the "key" is typically semantic-aware and heavily dependent on the CLIP model as prior. In this paper, we propose Flow4Agent, a novel framework that pioneeringly incorporates motion priors from optical flow to facilitate LLM-based long video understanding. Flow4Agent mitigates the redundancy in long videos at both temporal and spatial levels through two core modules: Temporal Granularity Optimization (TGO) adaptively refines framelevel hierarchies, which first leverages coarse flow priors to group similar visual contents and then applies semantic priors to filter out highly irrelevant scene information. Motion Token Pruning (MTP) further refines the intra-frame visual representations, pruning high-redundancy video tokens using fine-grained optical flow information. Extensive experiments demonstrate that our Flow4Agent outperforms existing methods across a wide range of video MLLM benchmarks, especially for hour-level video understanding tasks, achieving 64.7% on Video-MME, 71.4% on MLVU and 60.4% on LongVideoBench.
Abstract (translated)
长视频理解一直是一个具有挑战性的问题，原因是时间上和空间上的内容都存在大量冗余。这一难题由于多模态大型语言模型（MLLMs）的有限上下文长度而进一步加剧。为了应对这个问题，许多先前的研究试图提取关键视频信息，“关键”通常指的是语义感知的信息，并且高度依赖于CLIP模型作为先验知识。在本文中，我们提出了Flow4Agent，这是一个创新框架，它首次将来自光学流的运动先验引入到基于LLM（大型语言模型）的长视频理解中。通过两个核心模块——时间粒度优化（TGO）和运动标记修剪（MTP），Flow4Agent在时间和空间层面上减轻了长时间视频中的冗余。 具体来说： - **时间粒度优化（TGO）** 模块会根据粗略的光学流先验将相似视觉内容进行分组，然后利用语义先验过滤掉高度无关场景信息。 - **运动标记修剪（MTP）** 进一步精炼帧内视觉表示，通过细粒度的光学流信息删除高冗余视频标记。 实验结果表明，我们的Flow4Agent在广泛的视频MLLM基准测试中优于现有方法，特别是在小时级别的视频理解任务上表现出色。具体来说，在Video-MME、MLVU和LongVideoBench这些数据集上的表现分别为64.7%、71.4% 和 60.4%，显示了该框架在处理长时视频中的优越性。
URL
https://arxiv.org/abs/2510.05836
