Abstract
Efficiently understanding long-form videos remains a fundamental challenge for multimodal large language models (MLLMs). In this paper, we present MLLM-Sampler Joint Evolution (MSJoE), a novel framework that jointly evolves the MLLM and a lightweight key-frame sampler for efficient long-form video understanding. MSJoE builds upon a key assumption that only a small subset of key-frames is truly informative for answering each question to a video. Specifically, MSJoE first reasons out several queries, which describe diverse visual perspectives relevant to the question. Then, these queries interact with a frozen CLIP model to produce a query-frame similarity matrix. Finally, a lightweight sampler predicts key-frame sampling weights from this matrix, selecting a compact set of informative frames, which are then fed into the MLLM for answer generation. Both the MLLM and sampler are jointly optimized through reinforcement learning, enabling co-adaptation of query-reasoning, frame-sampling, and key-frame understanding. A new long-video QA dataset containing 2.8K videos with 7K question-answer pairs is collected to support the training process. Extensive experiments on VideoMME, LongVideoBench, LVBench, and MLVU show that MSJoE achieves 8.0\% accuracy gain upon the base MLLM, and 1.1\% higher accuracy than strongest baseline method.
Abstract (translated)
理解长视频的有效方法仍然是多模态大型语言模型（MLLM）面临的基本挑战。本文介绍了一种新的框架——MLLM-Sampler联合演化（MSJoE），该框架能够使MLLM和一个轻量级的关键帧采样器协同进化，从而实现对长视频的高效理解和分析。 MSJoE基于这样一个关键假设：对于每个问题而言，只有视频中的一小部分关键帧才是真正具有信息价值的。具体来说，MSJoE首先生成几个描述与问题相关且多样化的视觉视角的查询语句。接着，这些查询与一个冻结状态下的CLIP模型互动，以产生一个查询-帧相似度矩阵。然后，轻量级采样器从这个矩阵中预测出关键帧的采样权重，并选择一套精简的信息丰富帧集，这些被选中的帧随后输入到MLLM进行答案生成。 通过强化学习的方式，对MLLM和采样器同时进行优化，使得查询推理、帧采样以及关键帧理解能够共同适应。为此，我们收集了一个新的长视频问答数据集，其中包含2800个视频和7000组问题-答案配对以支持训练过程。 在VideoMME、LongVideoBench、LVBench 和 MLVU上进行的广泛实验表明，MSJoE相比基础MLLM提高了8.0%的准确率，并且比最强基线方法高出1.1%。
URL
https://arxiv.org/abs/2602.22932
