Abstract
Given a style-reference image as the additional image condition, text-to-image diffusion models have demonstrated impressive capabilities in generating images that possess the content of text prompts while adopting the visual style of the reference image. However, current state-of-the-art methods often struggle to disentangle content and style from style-reference images, leading to issues such as content leakages. To address this issue, we propose a masking-based method that efficiently decouples content from style without the need of tuning any model parameters. By simply masking specific elements in the style reference's image features, we uncover a critical yet under-explored principle: guiding with appropriately-selected fewer conditions (e.g., dropping several image feature elements) can efficiently avoid unwanted content flowing into the diffusion models, enhancing the style transfer performances of text-to-image diffusion models. In this paper, we validate this finding both theoretically and experimentally. Extensive experiments across various styles demonstrate the effectiveness of our masking-based method and support our theoretical results.
Abstract (translated)
在给定风格参考图像作为附加条件的情况下，文本到图像的扩散模型展示出了生成同时具备文本提示内容并采用参考图像视觉风格的图片的强大能力。然而，目前最先进的方法常常难以从风格参考图中分离出内容和风格，导致诸如内容泄露等问题的发生。为解决这一问题，我们提出了一种基于掩码的方法，能够在不调整任何模型参数的情况下有效分离内容与风格。通过简单地在风格参考图像特征的特定元素上进行掩码处理，我们揭示了一个重要但尚未充分探索的原则：使用适当选择的较少条件（例如，删除几个图像特征元素）可以有效地避免不必要的内容流入扩散模型，从而提升文本到图像扩散模型的风格迁移性能。本文中，我们在理论和实验两方面验证了这一发现的有效性。跨多种风格进行的大量实验展示了我们基于掩码的方法的有效性，并支持我们的理论结果。
URL
https://arxiv.org/abs/2502.07466
