Abstract
We present MoE-DiffuSeq, a mixture of experts based framework for enhancing diffusion models in long document generation. Existing diffusion based text generation models, such as DiffuSeq, suffer from high computational cost and memory overhead when applied to extended sequences. To address these challenges, MoE-DiffuSeq integrates sparse attention with a mixture of experts architecture, enabling efficient and scalable long sequence modeling. Our approach introduces a customized sparse attention mechanism designed to reduce computational complexity while preserving text quality and coherence. In addition, we incorporate a soft absorbing state within the diffusion process to accelerate sequence reconstruction and improve generation precision. Extensive experiments demonstrate that MoE-DiffuSeq significantly improves training efficiency and sampling speed compared to existing diffusion models. These advantages are particularly effective for long document scenarios, including scientific article generation, code repository modeling, and long form dialogue generation. Benchmark results further show that MoE-DiffuSeq improves efficiency, speed, accuracy, and expressiveness, advancing the practical applicability of diffusion models for high quality long form text generation.
Abstract (translated)
我们提出了一种基于专家混合框架的MoE-DiffuSeq，旨在通过增强扩散模型来改进长文档生成。现有的基于扩散的文本生成模型（如DiffuSeq）在应用于扩展序列时会面临高昂的计算成本和内存开销问题。为了解决这些问题，MoE-DiffuSeq将稀疏注意力机制与专家混合架构相结合，从而实现高效且可扩展的长序列建模。我们的方法引入了一种定制化的稀疏注意力机制，旨在减少计算复杂度的同时保持文本质量和连贯性。此外，我们在扩散过程中加入了软吸收状态，以加速序列重构并提高生成精度。 广泛的实验表明，与现有的基于扩散的模型相比，MoE-DiffuSeq在训练效率和采样速度方面显著提升。这些优势尤其适用于长文档场景，包括科学文章生成、代码库建模以及长篇对话生成等领域。基准测试结果显示，MoE-DiffuSeq不仅提高了效率和速度，还提升了准确性和表现力，从而推进了扩散模型在高质量长文本生成领域的实际应用性。
URL
https://arxiv.org/abs/2512.20604
