Abstract
Art reinterpretation is the practice of creating a variation of a reference work, making a paired artwork that exhibits a distinct artistic style. We ask if such an image pair can be used to customize a generative model to capture the demonstrated stylistic difference. We propose Pair Customization, a new customization method that learns stylistic difference from a single image pair and then applies the acquired style to the generation process. Unlike existing methods that learn to mimic a single concept from a collection of images, our method captures the stylistic difference between paired images. This allows us to apply a stylistic change without overfitting to the specific image content in the examples. To address this new task, we employ a joint optimization method that explicitly separates the style and content into distinct LoRA weight spaces. We optimize these style and content weights to reproduce the style and content images while encouraging their orthogonality. During inference, we modify the diffusion process via a new style guidance based on our learned weights. Both qualitative and quantitative experiments show that our method can effectively learn style while avoiding overfitting to image content, highlighting the potential of modeling such stylistic differences from a single image pair.
Abstract (translated)
艺术再诠释是对参考作品的一种变体，创作了一对对比艺术作品，展示了独特的艺术风格。我们询问，这样的图像对是否可以用于定制生成模型来捕捉演示的文体差异。我们提出了Pair Customization，一种新的定制方法，它从单个图像对中学习文体差异，然后将获得的风格应用于生成过程。与现有的方法不同，我们的方法从单个图像对中捕获文体差异。这使我们能够在不需要对实例的具体内容进行过拟合的情况下应用文体变化。为了应对这项新任务，我们采用了一种联合优化方法，将风格和内容明确地分离到两个LORA权重空间中。我们优化了这些风格和内容权重，以复制风格和内容图像，同时鼓励它们的正交性。在推理过程中，我们通过基于我们学习到的权重的新的样式指导来修改扩散过程。所有定性和定量实验都表明，我们的方法可以有效地学习风格，同时避免对图像内容的过拟合，突出了从单个图像对中建模这些文体差异的潜在可能性。
URL
https://arxiv.org/abs/2405.01536
