Abstract
Large Vision Language Models show impressive performance across image and video understanding tasks, yet their computational cost grows rapidly with the number of visual tokens. Existing token pruning methods mitigate this issue through empirical approaches while overlooking the internal mechanism of attention. In this paper, we propose a novel training free token pruning framework grounded in the dual form perspective of attention. We reformulate attention as an implicit linear layer whose weight matrix is the sum of rank 1 outer products, each generated by a single token's key value pair. Token pruning thus reduces to selecting an optimal subset of these rank 1 updates that best approximates the original dual weight matrix. Extending this perspective to standard softmax attention in LVLMs, we derive a novel metric quantifying both a token's information magnitude and information duplication. To efficiently select the subset with the proposed metric, we introduce Progressive Chunked Maximal Marginal Relevance. Extensive experiments demonstrate that our method achieves a better trade off between performance and efficiency, while providing another perspective on existing pruning approaches.
Abstract (translated)
大型视觉语言模型在图像与视频理解任务中展现出卓越性能，但其计算成本随视觉token数量急剧增加。现有token剪枝方法多依赖经验性策略，却忽视了注意力的内在机制。本文提出一种基于注意力对偶形式视角的新型无训练token剪枝框架。我们将注意力重构为一个隐式线性层，其权重矩阵由多个秩1外积之和构成，每个外积由单个token的键值对生成。因此，token剪枝问题转化为选择最优子集，使这些秩1更新能最佳逼近原始对偶权重矩阵。将该视角扩展至LVLM的标准softmax注意力，我们提出一种同时量化token信息量及信息冗余的新指标。为高效筛选指标定义的子集，我们设计了渐进分块最大边际相关性算法。大量实验表明，该方法在性能与效率间取得更优平衡，并为现有剪枝方法提供了新的理论视角。
URL
https://arxiv.org/abs/2604.00757
