Abstract
The slow, sequential nature of autoregressive (AR) language models has driven the adoption of parallel decoding methods. However, these non-AR models often sacrifice generation quality as they struggle to model the complex joint distribution of token sequences. To narrow this performance gap, we introduce Gumbel Distillation, a novel distillation technique that enables parallel decoders to learn this distribution effectively. Our method leverages the Gumbel-Max trick to create a deterministic mapping from a latent Gumbel noise space to the output tokens of a high-performing AR teacher. As a model-agnostic technique, Gumbel Distillation seamlessly integrates with diverse parallel decoding architectures, including MDLM and BD3-LM. Experiments on LM1B and OpenWebText show that Gumbel Distillation substantially improves the generation quality of parallel language models, achieving a 30.0% improvement in MAUVE score and 10.5% in generative perplexity over MDLM trained on OpenWebText dataset. Code available at this https URL.
Abstract (translated)
自回归（AR）语言模型的缓慢、顺序特性推动了并行解码方法的采用。然而，这些非自回归模型往往在生成质量上做出妥协，因为它们难以对标记序列的复杂联合分布进行建模。为缩小这一性能差距，我们提出了冈贝尔蒸馏（Gumbel Distillation），一种新颖的蒸馏技术，使并行解码器能够有效学习该分布。我们的方法利用冈贝尔-最大值技巧（Gumbel-Max trick），建立了从潜在冈贝尔噪声空间到高性能自回归教师模型输出标记之间的确定性映射。作为一种与模型无关的技术，冈贝尔蒸馏可无缝集成到多种并行解码架构中，包括MDLM和BD3-LM。在LM1B和OpenWebText数据集上的实验表明，冈贝尔蒸馏显著提升了并行语言模型的生成质量，在OpenWebText数据集上相较于MDLM，MAUVE分数提高30.0%，生成困惑度降低10.5%。代码已公开于此链接。
URL
https://arxiv.org/abs/2603.22216
