Abstract
Diffusion large language models (dLLMs) have shown advantages in text generation, particularly due to their inherent ability for parallel decoding. However, constrained by the quality--speed trade-off, existing inference solutions adopt conservative parallel strategies, leaving substantial efficiency potential underexplored. A core challenge is that parallel decoding assumes each position can be filled independently, but tokens are often semantically coupled. Thus, the correct choice at one position constrains valid choices at others. Without modeling these inter-token dependencies, parallel strategies produce deteriorated outputs. Motivated by this insight, we propose DAWN, a training-free, dependency-aware decoding method for fast dLLM inference. DAWN extracts token dependencies and leverages two key motivations: (1) positions dependent on unmasked certain positions become more reliable, (2) simultaneously unmasking strongly coupled uncertain positions induces errors. Given those findings, DAWN leverages a dependency graph to select more reliable unmasking positions at each iteration, achieving high parallelism with negligible loss in generation quality. Extensive experiments across multiple models and datasets demonstrate that DAWN speedups the inference by 1.80-8.06x over baselines while preserving the generation quality. Code is released at this https URL.
Abstract (translated)
扩散大型语言模型（dLLMs）在文本生成方面表现出色，尤其得益于其固有的并行解码能力。然而，受限于质量与速度的权衡问题，现有的推理解决方案采用了保守的并行策略，未能充分利用潜在的效率优势。一个核心挑战在于，并行解码假设每个位置都可以独立填充，但实际上令牌之间往往存在语义上的关联性。因此，在某一位置上做出正确的选择会限制其他位置的有效选择范围。若不考虑这些跨令牌依赖关系，简单的并行策略会产生质量较差的结果。 基于这一洞察，我们提出了DAWN——一种训练无关的、具备依存感知能力的解码方法，以实现快速且高效的dLLM推理过程。DAWN能够提取令牌间的依赖关系，并利用两个关键动机：（1）在未屏蔽特定位置时，那些与此相关的位置变得更加可靠；（2）同时对高度相关的不确定位置进行解锁操作会导致错误。 基于这些发现，DAWN通过使用一个依存图来选择在每次迭代中更加可靠的未屏蔽位置，从而实现了高并行性，并且几乎不会损失生成的质量。广泛的实验表明，在多个模型和数据集上，相比基线方法，DAWN能够将推理速度提升1.80到8.06倍，同时保持生成质量不变。 代码已发布于[此链接](https://this-url.com/)（请替换为实际的URL）。
URL
https://arxiv.org/abs/2602.06953
