Abstract
Diffusion based approaches to long form text generation suffer from prohibitive computational cost and memory overhead as sequence length increases. We introduce SA-DiffuSeq, a diffusion framework that integrates sparse attention to fundamentally improve scalability for long document modeling. By selectively allocating attention within the diffusion process, SA-DiffuSeq significantly reduces computational complexity while maintaining semantic coherence and generation quality. A key component of our method is a soft absorbing state tailored to sparse attention dynamics, which stabilizes diffusion trajectories and accelerates sequence reconstruction. This design improves sampling efficiency and enhances precision in long range dependency modeling. Extensive experiments demonstrate that SA-DiffuSeq consistently surpasses state of the art diffusion baselines in both training efficiency and sampling speed, with especially strong gains on extended sequences. These properties make SA-DiffuSeq well suited for demanding long form applications such as scientific writing, large scale code generation, and multi turn long context dialogue. Overall, our results indicate that incorporating structured sparsity into diffusion models is a promising direction for efficient and expressive long text generation.
Abstract (translated)
基于扩散的方法在生成长篇文本时，随着序列长度的增加，会面临高昂的计算成本和内存开销问题。我们提出了一种名为SA-DiffuSeq的新框架，该框架通过整合稀疏注意力机制，从根本上提高了长文档建模的可扩展性。通过在扩散过程中选择性地分配注意力，SA-DiffuSeq显著降低了计算复杂度，并保持了语义连贯性和生成质量。 我们的方法的关键组件是一个专门针对稀疏注意动态特制的软吸收状态，这有助于稳定扩散轨迹并加速序列重构过程。这种设计提高了采样效率，并增强了对长距离依赖关系建模的精确性。 大量的实验表明，SA-DiffuSeq在训练效率和采样速度方面持续超越现有的基于扩散的方法，在处理扩展序列时尤其表现出色。这些特性使得SA-DiffuSeq非常适合需要生成长篇文本的应用场景，例如科学写作、大规模代码生成以及多轮长上下文对话等。 总体而言，我们的结果表明，在扩散模型中引入结构化稀疏性是实现高效且富有表现力的长文本生成的一个有前景的方向。
URL
https://arxiv.org/abs/2512.20724
