Abstract
Vision Large Language Models (VLLMs) incur high computational costs due to their reliance on hundreds of visual tokens to represent images. While token pruning offers a promising solution for accelerating inference, this paper, however, identifies a key observation: in deeper layers (e.g., beyond the 20th), existing training-free pruning methods perform no better than random pruning. We hypothesize that this degradation is caused by "vanishing token information", where visual tokens progressively lose their salience with increasing network depth. To validate this hypothesis, we quantify a token's information content by measuring the change in the model output probabilities upon its removal. Using this proposed metric, our analysis of the information of visual tokens across layers reveals three key findings: (1) As layers deepen, the information of visual tokens gradually becomes uniform and eventually vanishes at an intermediate layer, which we term as "information horizon", beyond which the visual tokens become redundant; (2) The position of this horizon is not static; it extends deeper for visually intensive tasks, such as Optical Character Recognition (OCR), compared to more general tasks like Visual Question Answering (VQA); (3) This horizon is also strongly correlated with model capacity, as stronger VLLMs (e.g., Qwen2.5-VL) employ deeper visual tokens than weaker models (e.g., LLaVA-1.5). Based on our findings, we show that simple random pruning in deep layers efficiently balances performance and efficiency. Moreover, integrating random pruning consistently enhances existing methods. Using DivPrune with random pruning achieves state-of-the-art results, maintaining 96.9% of Qwen-2.5-VL-7B performance while pruning 50% of visual tokens. The code will be publicly available at this https URL.
Abstract (translated)
视觉大型语言模型（VLLMs）由于依赖数百个视觉令牌来表示图像，因此计算成本高昂。虽然令牌修剪为加速推理提供了一个有前景的解决方案，但本文却发现了关键观察：在深层网络中（例如超过第20层），现有的无训练修剪方法的表现并不比随机修剪更好。我们假设这种退化是由“消失的令牌信息”引起的，在这种情况下，随着网络深度的增加，视觉令牌逐渐失去其显著性。 为了验证这一假设，我们通过测量移除某个令牌后模型输出概率的变化来量化令牌的信息内容。使用这一提出的指标，对跨层的视觉令牌信息进行分析揭示了三个关键发现： 1. 随着层数加深，视觉令牌的信息变得越来越均匀，并在中间某一层完全消失，我们将这个点称为“信息地平线”，在此之后，视觉令牌变得多余。 2. 这个地平线的位置不是固定的；对于光学字符识别（OCR）等视觉密集型任务而言，它比诸如视觉问答（VQA）之类的通用任务延伸更深。 3. 此外，该地平线也与模型能力密切相关。更强的 VLLM 模型（例如 Qwen2.5-VL）使用更深层的视觉令牌，而较弱的模型则不如此。 根据我们的发现，我们在深层网络中展示了简单的随机修剪可以有效地平衡性能和效率。此外，整合随机修剪能够持续改进现有的方法。利用 DivPrune 结合随机修剪实现了最先进的成果，在剪枝50%视觉令牌的情况下，维持了 Qwen-2.5-VL-7B 模型96.9%的性能。 相关代码将公开发布在 https://this-url.com 的链接中。
URL
https://arxiv.org/abs/2512.07580
