Abstract
Although current Video-LLMs achieve impressive performance in video understanding tasks, their autoregressive decoding efficiency remains constrained by the massive number of video tokens. Visual token pruning can partially ease this bottleneck, yet existing approaches still suffer from information loss and yield only modest acceleration in decoding. In this paper, we propose ParallelVLM, a training-free draft-then-verify speculative decoding framework that overcomes both mutual waiting and limited speedup-ratio problems between draft and target models in long-video settings. ParallelVLM features two parallelized stages that maximize hardware utilization and incorporate an Unbiased Verifier-Guided Pruning strategy to better align the draft and target models by eliminating the positional bias in attention-guided pruning. Extensive experiments demonstrate that ParallelVLM effectively expands the draft window by $1.6\sim1.8\times$ with high accepted lengths, and accelerates various video understanding benchmarks by 3.36$\times$ on LLaVA-Onevision-72B and 2.42$\times$ on Qwen2.5-VL-32B compared with vanilla autoregressive decoding.
Abstract (translated)
尽管当前视频大语言模型在视频理解任务中表现出色，但其自回归解码效率仍受海量视频标记的制约。视觉标记剪枝虽能部分缓解这一瓶颈，现有方法仍存在信息损失问题，且解码加速效果有限。本文提出ParallelVLM，一种无需训练的草稿-验证推测性解码框架，可解决长视频场景下草稿模型与目标模型间的相互等待及加速比受限问题。该框架采用两阶段并行设计以最大化硬件利用率，并引入无偏验证器引导剪枝策略，通过消除注意力引导剪枝中的位置偏差来更好对齐草稿与目标模型。大量实验表明，ParallelVLM在保持高接受长度的同时，将草稿窗口有效扩展1.6~1.8倍，并在LLaVA-Onevision-72B和Qwen2.5-VL-32B上分别实现3.36倍与2.42倍加速，显著优于原生自回归解码。
URL
https://arxiv.org/abs/2603.19610
