Abstract
Diffusion models (DMs) are powerful generative models capable of producing high-fidelity images but are constrained by high computational costs due to iterative multi-step inference. While Neural Architecture Search (NAS) can optimize DMs, existing methods are hindered by retraining requirements, exponential search complexity from step-wise optimization, and slow evaluation relying on massive image generation. To address these challenges, we propose Flexiffusion, a training-free NAS framework that jointly optimizes generation schedules and model architectures without modifying pre-trained parameters. Our key insight is to decompose the generation process into flexible segments of equal length, where each segment dynamically combines three step types: full (complete computation), partial (cache-reused computation), and null (skipped computation). This segment-wise search space reduces the candidate pool exponentially compared to step-wise NAS while preserving architectural diversity. Further, we introduce relative FID (rFID), a lightweight evaluation metric for NAS that measures divergence from a teacher model's outputs instead of ground truth, slashing evaluation time by over $90\%$. In practice, Flexiffusion achieves at least $2\times$ acceleration across LDMs, Stable Diffusion, and DDPMs on ImageNet and MS-COCO, with FID degradation under $5\%$, outperforming prior NAS and caching methods. Notably, it attains $5.1\times$ speedup on Stable Diffusion with near-identical CLIP scores. Our work pioneers a resource-efficient paradigm for searching high-speed DMs without sacrificing quality.
Abstract (translated)
扩散模型（DM）是一种强大的生成式模型，能够产生高质量的图像，但由于其迭代多步推理过程中的高计算成本而受到限制。虽然神经架构搜索（NAS）可以优化扩散模型，但现有方法面临着重新训练需求、基于步骤级优化的指数级搜索复杂性以及依赖大规模图像生成导致的评估速度慢等问题。为解决这些挑战，我们提出了Flexiffusion，这是一种无需训练即可进行NAS框架的方法，能够在不修改预训练参数的情况下同时优化生成调度和模型架构。 我们的关键洞察是将生成过程分解为长度相等但灵活的段落，在每个段落中动态结合三种步骤类型：完整（完整的计算）、部分（利用缓存重复的计算）和空缺（跳过的计算）。这种基于段落的搜索空间与步长NAS相比，候选池的数量呈指数级减少，同时保持架构多样性。此外，我们引入了相对FID（rFID），这是一种轻量级的NAS评估指标，通过衡量模型输出相对于教师模型输出而非真实值的差异来测量发散度，从而将评估时间减少了超过90%。 在实践中，Flexiffusion在LDMs、Stable Diffusion和DDPMs上对ImageNet和MS-COCO数据集实现了至少两倍的速度提升，并且FID退化不超过5%，优于先前的NAS和缓存方法。值得注意的是，在Stable Diffusion中，它达到了近似相等CLIP得分的五点一次加速效果。 我们的工作开创了一种资源高效的范式，用于在不牺牲质量的情况下搜索高速扩散模型。
URL
https://arxiv.org/abs/2506.02488
