Abstract
Diffusion models have attained remarkable breakthroughs in the real-world super-resolution (SR) task, albeit at slow inference and high demand on devices. To accelerate inference, recent works like GenDR adopt step distillation to minimize the step number to one. However, the memory boundary still restricts the maximum processing size, necessitating tile-by-tile restoration of high-resolution images. Through profiling the pipeline, we pinpoint that the variational auto-encoder (VAE) is the bottleneck of latency and memory. To completely solve the problem, we leverage pixel-(un)shuffle operations to eliminate the VAE, reversing the latent-based GenDR to pixel-space GenDR-Pix. However, upscale with x8 pixelshuffle may induce artifacts of repeated patterns. To alleviate the distortion, we propose a multi-stage adversarial distillation to progressively remove the encoder and decoder. Specifically, we utilize generative features from the previous stage models to guide adversarial discrimination. Moreover, we propose random padding to augment generative features and avoid discriminator collapse. We also introduce a masked Fourier space loss to penalize the outliers of amplitude. To improve inference performance, we empirically integrate a padding-based self-ensemble with classifier-free guidance to improve inference scaling. Experimental results show that GenDR-Pix performs 2.8x acceleration and 60% memory-saving compared to GenDR with negligible visual degradation, surpassing other one-step diffusion SR. Against all odds, GenDR-Pix can restore 4K image in only 1 second and 6GB.
Abstract (translated)
扩散模型在实际超分辨率（SR）任务中取得了显著突破，尽管推理速度较慢且对设备要求较高。为了加速推理过程，近期的研究如GenDR采用步长蒸馏技术将步骤数最小化至一步。然而，内存限制仍然制约着最大处理尺寸，需要分块恢复高分辨率图像。通过分析整个流程，我们发现变分自动编码器（VAE）是延迟和内存的瓶颈所在。为彻底解决问题，我们利用像素重排操作来消除VAE的影响，并将基于潜在空间的GenDR逆转到像素空间的GenDR-Pix中。然而，使用8倍像素重排进行上采样可能会导致重复图案的伪影出现。 为了减轻这种失真现象，我们提出了一种多阶段对抗蒸馏方法，逐步移除编码器和解码器，并利用前一阶段模型生成的特征来指导对抗判别过程。此外，我们还提出了随机填充策略以增强生成特性并避免鉴别器崩溃问题。同时，引入了掩蔽傅立叶空间损失函数，用于惩罚幅度异常值。 为了提高推理性能，我们基于填充自集成方法结合无分类引导技术进行了经验整合，从而改进了推理扩展性。实验结果表明，与GenDR相比，GenDR-Pix在视觉质量几乎不变的情况下实现了2.8倍的加速和60%的内存节省，并且优于其他一步扩散SR模型。令人惊讶的是，即使面对逆境，GenDR-Pix也能够在1秒内仅用6GB内存恢复4K图像。
URL
https://arxiv.org/abs/2602.10630
