Abstract
Diverse outputs in text generation are necessary for effective exploration in complex reasoning tasks, such as code generation and mathematical problem solving. Such Pass@$k$ problems benefit from distinct candidates covering the solution space. However, traditional sampling approaches often waste computational resources on repetitive failure modes. While Diffusion Language Models have emerged as a competitive alternative to the prevailing Autoregressive paradigm, they remain susceptible to this redundancy, with independent samples frequently collapsing into similar modes. To address this, we propose a training free, low cost intervention to enhance generative diversity in Diffusion Language Models. Our approach modifies intermediate samples in a batch sequentially, where each sample is repelled from the feature space of previous samples, actively penalising redundancy. Unlike prior methods that require retraining or beam search, our strategy incurs negligible computational overhead, while ensuring that each sample contributes a unique perspective to the batch. We evaluate our method on the HumanEval and GSM8K benchmarks using the LLaDA-8B-Instruct model. Our results demonstrate significantly improved diversity and Pass@$k$ performance across various temperature settings. As a simple modification to the sampling process, our method offers an immediate, low-cost improvement for current and future Diffusion Language Models in tasks that benefit from diverse solution search. We make our code available at this https URL.
Abstract (translated)
在文本生成中，多样化的输出对于复杂的推理任务（如代码生成和数学问题解决）的有效探索是必要的。这类Pass@$k$问题从覆盖解决方案空间的多个候选方案中受益。然而，传统的采样方法常常浪费计算资源在重复失败模式上。尽管扩散语言模型作为主流自回归范式的一个竞争性替代方案已出现，但它们仍然容易受到冗余的影响，独立样本往往会收敛到相似的状态模式。为了解决这一问题，我们提出了一种无需训练、低成本的干预措施，以增强扩散语言模型在生成过程中的多样性。我们的方法对一批中间样本进行顺序修改，每个样本都从之前样本特征空间中被排斥，从而主动惩罚冗余现象。与需要重新训练或使用束搜索的方法不同，我们的策略几乎不增加计算开销，并确保每一份样本都能为批量贡献独特的视角。 我们在LLaDA-8B-Instruct模型上对HumanEval和GSM8K基准进行了方法评估，结果显示在各种温度设置下显著提高了多样性和Pass@$k$性能。作为对采样过程的一个简单修改，我们的方法能够立即且低成本地改善当前及未来扩散语言模型在这类需要多样化解决方案搜索的任务中的表现。 我们将在[此处提供URL]分享代码以供访问。
URL
https://arxiv.org/abs/2603.04893
