Abstract
Video Large Language Models (VideoLLMs) have demonstrated impressive capabilities in video understanding, yet the massive number of input video tokens incurs a significant computational burden for deployment. Existing methods mainly prune video tokens at input level while neglecting the inherent information structure embedded in videos and large language models (LLMs). To address this, we propose HieraVid, a hierarchical pruning framework that progressively and dynamically reduces visual redundancy. Based on two observations that videos possess the segment-frame structure and LLMs internally propagate multi-modal information unidirectionally, we decompose pruning into three levels: 1) segment-level, where video tokens are first temporally segmented and spatially merged; 2) frame-level, where similar frames within the same segment are jointly pruned to preserve diversity; 3) layer-level, redundancy gradually shrinks as LLM layer increases w/o compromising performance. We conduct extensive experiments on four widely used video understanding benchmarks to comprehensively evaluate the effectiveness of HieraVid. Remarkably, with only 30% of tokens retained, HieraVid achieves new state-of-the-art performance, while maintaining over 98% and 99% of the performance of LLaVA-Video-7B and LLaVA-OneVision-7B, respectively.
Abstract (translated)
视频大语言模型（VideoLLMs）在视频理解方面展现出卓越能力，但海量的输入视频令牌给实际部署带来了显著的计算负担。现有方法主要在输入层面对视频令牌进行剪枝，却忽视了视频与大语言模型（LLMs）本身蕴含的信息结构。为此，我们提出HieraVid——一个分层剪枝框架，通过渐进式动态方式降低视觉冗余。基于视频具有段-帧结构、LLMs内部以单向方式传播多模态信息这两个观察，我们将剪枝分解为三个层次：1）段级，首先对视频令牌进行时间分段与空间合并；2）帧级，在同一段内联合剪除相似帧以保留多样性；3）层级，随着LLM层数增加，冗余度逐渐降低且不影响性能。我们在四个广泛使用的视频理解基准上开展大量实验，全面评估HieraVid的有效性。值得注意的是，HieraVid仅保留30%的令牌即达到新的最优性能，同时分别保持LLaVA-Video-7B与LLaVA-OneVision-7B超过98%与99%的性能表现。
URL
https://arxiv.org/abs/2604.01881
