Abstract
Diffusion models have demonstrated great success in the field of text-to-image generation. However, alleviating the misalignment between the text prompts and images is still challenging. The root reason behind the misalignment has not been extensively investigated. We observe that the misalignment is caused by inadequate token attention activation. We further attribute this phenomenon to the diffusion model's insufficient condition utilization, which is caused by its training paradigm. To address the issue, we propose CoMat, an end-to-end diffusion model fine-tuning strategy with an image-to-text concept matching mechanism. We leverage an image captioning model to measure image-to-text alignment and guide the diffusion model to revisit ignored tokens. A novel attribute concentration module is also proposed to address the attribute binding problem. Without any image or human preference data, we use only 20K text prompts to fine-tune SDXL to obtain CoMat-SDXL. Extensive experiments show that CoMat-SDXL significantly outperforms the baseline model SDXL in two text-to-image alignment benchmarks and achieves start-of-the-art performance.
Abstract (translated)
扩散模型在文本到图像生成领域取得了巨大的成功。然而，减轻文本提示和图像之间的不匹配仍然是具有挑战性的。导致不匹配的根本原因尚未进行深入研究。我们观察到，不匹配是由不足的词注意激活引起的。我们进一步将这种现象归因于扩散模型的不足条件利用率，这是由于其训练范式造成的。为了应对这个问题，我们提出了CoMat，一种端到端的扩散模型微调策略，具有图像到文本的概念匹配机制。我们利用图像标题模型测量图像到文本的匹配，并指导扩散模型重新关注被忽略的词。还提出了一个新的属性绑定模块来解决属性绑定问题。在没有图像或人类偏好数据的情况下，我们仅使用20K个文本提示微调SDXL以获得CoMat-SDXL。大量实验证明，CoMat-SDXL在两个文本到图像对齐基准测试中的表现明显优于基线模型SDXL，并实现了最先进的性能。
URL
https://arxiv.org/abs/2404.03653
