Abstract
With the rapid development of Large Language Models (LLMs), Video Multi-Modal Large Language Models (Video MLLMs) have achieved remarkable performance in video-language tasks such as video understanding and question answering. However, Video MLLMs face high computational costs, particularly in processing numerous video frames as input, which leads to significant attention computation overhead. A straightforward approach to reduce computational costs is to decrease the number of input video frames. However, simply selecting key frames at a fixed frame rate (FPS) often overlooks valuable information in non-key frames, resulting in notable performance degradation. To address this, we propose Text-guided Video Frame Reduction (TiFRe), a framework that reduces input frames while preserving essential video information. TiFRe uses a Text-guided Frame Sampling (TFS) strategy to select key frames based on user input, which is processed by an LLM to generate a CLIP-style prompt. Pre-trained CLIP encoders calculate the semantic similarity between the prompt and each frame, selecting the most relevant frames as key frames. To preserve video semantics, TiFRe employs a Frame Matching and Merging (FMM) mechanism, which integrates non-key frame information into the selected key frames, minimizing information loss. Experiments show that TiFRe effectively reduces computational costs while improving performance on video-language tasks.
Abstract (translated)
随着大型语言模型（LLMs）的快速发展，视频多模态大型语言模型（Video MLLMs）在视频理解和问答等任务中取得了显著的成绩。然而，Video MLLMs面临着高昂的计算成本问题，特别是在处理大量视频帧输入时会导致显著的关注力计算开销。减少计算成本的一种直接方法是降低输入视频帧的数量。但是，简单地以固定帧率（FPS）选择关键帧往往会忽略非关键帧中的重要信息，导致性能下降。 为了应对这一挑战，我们提出了文本引导的视频帧缩减框架（Text-guided Video Frame Reduction, TiFRe），该框架在减少输入帧数量的同时保留了重要的视频信息。TiFRe采用了一种基于用户输入的文本引导的帧采样策略（TFS）来选择关键帧，并由LLM生成类似CLIP风格的提示词。预训练好的CLIP编码器计算这些提示词与每个视频帧之间的语义相似度，从而选出最相关的帧作为关键帧。 为了保持视频的语义信息不丢失，TiFRe采用了一个帧匹配和合并（Frame Matching and Merging, FMM）机制，将非关键帧的信息整合到选定的关键帧中，从而最小化信息损失。实验结果表明，TiFRe不仅能有效降低计算成本，还能提升在视频语言任务中的性能表现。
URL
https://arxiv.org/abs/2602.08861
