Abstract
Recent advances in visual synthesis have leveraged diffusion models and attention mechanisms to achieve high-fidelity artistic style transfer and photorealistic text-to-image generation. However, real-time deployment on edge devices remains challenging due to computational and memory constraints. We propose Muon-AD, a co-designed framework that integrates the Muon optimizer with attention distillation for real-time edge synthesis. By eliminating gradient conflicts through orthogonal parameter updates and dynamic pruning, Muon-AD achieves 3.2 times faster convergence compared to Stable Diffusion-TensorRT, while maintaining synthesis quality (15% lower FID, 4% higher SSIM). Our framework reduces peak memory to 7GB on Jetson Orin and enables 24FPS real-time generation through mixed-precision quantization and curriculum learning. Extensive experiments on COCO-Stuff and ImageNet-Texture demonstrate Muon-AD's Pareto-optimal efficiency-quality trade-offs. Here, we show a 65% reduction in communication overhead during distributed training and real-time 10s/image generation on edge GPUs. These advancements pave the way for democratizing high-quality visual synthesis in resource-constrained environments.
Abstract (translated)
最近在视觉合成领域取得的进展利用了扩散模型和注意力机制，实现了高质量的艺术风格转换以及逼真的文本到图像生成。然而，由于计算资源和内存限制，在边缘设备上实现实时部署仍然面临挑战。我们提出了一种名为Muon-AD的框架，该框架结合了Muon优化器与注意力蒸馏技术，旨在为边缘设备上的实时合成提供解决方案。 通过正交参数更新和动态修剪消除梯度冲突，Muon-AD实现了比Stable Diffusion-TensorRT快3.2倍的收敛速度，同时保持了合成质量（FID降低了15%，SSIM提高了4%）。我们的框架将Jetson Orin上的峰值内存减少到7GB，并通过混合精度量化和课程学习实现实时生成速率高达24FPS。在COCO-Stuff和ImageNet-Texture数据集上进行的广泛实验表明，Muon-AD在效率与质量之间实现了Pareto最优权衡。 此外，我们的方法展示了分布式训练期间通信开销减少了65%，并且能够在边缘GPU上实现实时每10秒生成一张图像。这些改进为在资源受限环境中实现高质量视觉合成铺平了道路。
URL
https://arxiv.org/abs/2504.08451
