Abstract
Diffusion Language Models (DLMs) have recently achieved strong results in text generation. However, their multi-step sampling leads to slow inference, limiting practical use. To address this, we extend Inverse Distillation, a technique originally developed to accelerate continuous diffusion models, to the discrete setting. Nonetheless, this extension introduces both theoretical and practical challenges. From a theoretical perspective, the inverse distillation objective lacks uniqueness guarantees, which may lead to suboptimal solutions. From a practical standpoint, backpropagation in the discrete space is non-trivial and often unstable. To overcome these challenges, we first provide a theoretical result demonstrating that our inverse formulation admits a unique solution, thereby ensuring valid optimization. We then introduce gradient-stable relaxations to support effective training. As a result, experiments on multiple DLMs show that our method, Inverse-distilled Diffusion Language Models (IDLM), reduces the number of inference steps by 4x-64x, while preserving the teacher model's entropy and generative perplexity.
Abstract (translated)
扩散语言模型（DLM）最近在文本生成方面取得了显著成果。然而，其多步骤采样导致推理速度慢，限制了其实用性。为了解决这个问题，我们将逆向蒸馏技术扩展到离散场景中使用。这种技术最初是为了加速连续扩散模型而开发的。但是，这一扩展带来了理论和实践上的挑战。从理论上讲，逆向蒸馏目标缺乏唯一性保证，可能导致次优解出现。在实践中，离散空间中的反向传播复杂且不稳定。 为了克服这些挑战，我们首先提供了一个理论结果，证明我们的逆向公式具有唯一的解决方案，从而确保了有效的优化过程。接着，我们引入了梯度稳定的松弛方法来支持有效训练。实验结果显示，在多个DLM上应用这种方法——逆蒸馏扩散语言模型（IDLM）之后，推理步骤减少了4到64倍，并且保持了教师模型的熵和生成困惑度。 这种改进使得在不牺牲性能的情况下加速文本生成过程成为可能，从而提高了模型的实际使用效率。
URL
https://arxiv.org/abs/2602.19066
