Abstract
Optimizing a text-to-image diffusion model with a given reward function is an important but underexplored research area. In this study, we propose Deep Reward Tuning (DRTune), an algorithm that directly supervises the final output image of a text-to-image diffusion model and back-propagates through the iterative sampling process to the input noise. We find that training earlier steps in the sampling process is crucial for low-level rewards, and deep supervision can be achieved efficiently and effectively by stopping the gradient of the denoising network input. DRTune is extensively evaluated on various reward models. It consistently outperforms other algorithms, particularly for low-level control signals, where all shallow supervision methods fail. Additionally, we fine-tune Stable Diffusion XL 1.0 (SDXL 1.0) model via DRTune to optimize Human Preference Score v2.1, resulting in the Favorable Diffusion XL 1.0 (FDXL 1.0) model. FDXL 1.0 significantly enhances image quality compared to SDXL 1.0 and reaches comparable quality compared with Midjourney v5.2.
Abstract (translated)
优化一个文本到图像扩散模型的奖励函数是一个重要的但尚未深入研究的研究领域。在这项研究中，我们提出了Deep Reward Tuning（DRTune）算法，这是一种直接监督文本到图像扩散模型最终输出图像的算法，并通过迭代采样过程反向传播。我们发现，训练采样过程的早期步骤对低级奖励至关重要，而通过停止噪声网络输入的梯度可以实现深度监督，从而提高奖励的质量和效果。DRTune在各种奖励模型上进行了广泛评估。它 consistently优于其他算法，尤其是在低级控制信号上。此外，通过DRTune对Stable Diffusion XL 1.0（SDXL 1.0）模型进行微调，以优化Human Preference Score v2.1，最终得到Favorable Diffusion XL 1.0（FDXL 1.0）模型。FDXL 1.0在图像质量方面显著提高，与SDXL 1.0相比，图像质量有显著提高，与Midjourney v5.2相当。
URL
https://arxiv.org/abs/2405.00760
