Abstract
Latent diffusion models such as Stable Diffusion 1.5 offer strong generative priors that are highly valuable for image restoration, yet their full pipelines remain too computationally heavy for deployment on edge devices. Existing lightweight variants predominantly compress the denoising U-Net or reduce the diffusion trajectory, which disrupts the underlying latent manifold and limits generalization beyond a single task. We introduce NanoSD, a family of Pareto-optimal diffusion foundation models distilled from Stable Diffusion 1.5 through network surgery, feature-wise generative distillation, and structured architectural scaling jointly applied to the U-Net and the VAE encoder-decoder. This full-pipeline co-design preserves the generative prior while producing models that occupy distinct operating points along the accuracy-latency-size frontier (e.g., 130M-315M parameters, achieving real-time inference down to 20ms on mobile-class NPUs). We show that parameter reduction alone does not correlate with hardware efficiency, and we provide an analysis revealing how architectural balance, feature routing, and latent-space preservation jointly shape true on-device latency. When used as a drop-in backbone, NanoSD enables state-of-the-art performance across image super-resolution, image deblurring, face restoration, and monocular depth estimation, outperforming prior lightweight diffusion models in both perceptual quality and practical deployability. NanoSD establishes a general-purpose diffusion foundation model family suitable for real-time visual generation and restoration on edge devices.
Abstract (translated)
潜扩散模型（如Stable Diffusion 1.5）提供强大的生成先验，这对图像恢复非常有价值。然而，它们的完整管道对于在边缘设备上部署来说计算负担过重。现有的轻量级变体主要通过压缩去噪U-Net或减少扩散轨迹来实现，这会破坏潜在流形并限制单一任务之外的一般化能力。我们引入了NanoSD，这是一个由Stable Diffusion 1.5通过网络手术、特征生成蒸馏和结构化架构扩展共同应用于U-Net和VAE编码器解码器的Pareto最优扩散基础模型家族。这种全管道协同设计在保持生成先验的同时，产生了位于精度延迟大小前沿的不同操作点上的模型（例如，1.3亿至3.15亿参数，在移动类NPUs上实现低至20ms的实时推理）。我们展示了仅减少参数数量与硬件效率之间不存在相关性，并提供了分析揭示架构平衡、特征路由和潜在空间保留如何共同塑造真正的设备延迟。作为现成骨干使用时，NanoSD在图像超分辨率、图像去模糊、面部恢复以及单目深度估计等任务中实现了最先进的性能，在感知质量和实际部署能力上均超过了先前的轻量级扩散模型。NanoSD为实时视觉生成和边缘设备上的图像恢复奠定了通用型扩散基础模型家族的基础。
URL
https://arxiv.org/abs/2601.09823
