Abstract
Discrete diffusion language models (dLLMs) accelerate text generation by unmasking multiple tokens in parallel. However, parallel decoding introduces a distributional mismatch: it approximates the joint conditional using a fully factorized product of per-token marginals, which degrades output quality when selected tokens are strongly dependent. We propose DEMASK (DEpendency-guided unMASKing), a lightweight dependency predictor that attaches to the final hidden states of a dLLM. In a single forward pass, it estimates pairwise conditional influences between masked positions. Using these predictions, a greedy selection algorithm identifies positions with bounded cumulative dependency for simultaneous unmasking. Under a sub-additivity assumption, we prove this bounds the total variation distance between our parallel sampling and the model's joint. Empirically, DEMASK achieves 1.7-2.2$\times$ speedup on Dream-7B while matching or improving accuracy compared to confidence-based and KL-based baselines.
Abstract (translated)
离散扩散语言模型（dLLMs）通过并行解掩多个词元来加速文本生成。然而，并行解码会引入分布失配问题：它使用完全因式分解的逐词元边缘乘积来近似联合条件分布，当所选词元高度相关时，这会降低输出质量。我们提出了DEMASK（基于依赖引导的解掩码），一种轻量级依赖预测器，它附加在dLLM的最终隐藏状态上。在单次前向传播中，它估计掩码位置之间的成对条件影响。利用这些预测，贪心选择算法可识别累积依赖有界的同步解掩位置。在次可加性假设下，我们证明了这限制了并行采样与模型联合分布之间的总变差距离。实验表明，DEMASK在Dream-7B上实现了1.7-2.2倍加速，同时在准确率上匹配或优于基于置信度和KL散度的基线方法。
URL
https://arxiv.org/abs/2604.02560
