Abstract
Pre-trained large text-to-image (T2I) models with an appropriate text prompt has attracted growing interests in customized images generation field. However, catastrophic forgetting issue make it hard to continually synthesize new user-provided styles while retaining the satisfying results amongst learned styles. In this paper, we propose MuseumMaker, a method that enables the synthesis of images by following a set of customized styles in a never-end manner, and gradually accumulate these creative artistic works as a Museum. When facing with a new customization style, we develop a style distillation loss module to transfer the style of the whole dataset into generation of images. It can minimize the learning biases caused by content of images, and address the catastrophic overfitting issue induced by few-shot images. To deal with catastrophic forgetting amongst past learned styles, we devise a dual regularization for shared-LoRA module to optimize the direction of model update, which could regularize the diffusion model from both weight and feature aspects, respectively. Meanwhile, a unique token embedding corresponding to this new style is learned by a task-wise token learning module, which could preserve historical knowledge from past styles with the limitation of LoRA parameter quantity. As any new user-provided style come, our MuseumMaker can capture the nuances of the new styles while maintaining the details of learned styles. Experimental results on diverse style datasets validate the effectiveness of our proposed MuseumMaker method, showcasing its robustness and versatility across various scenarios.
Abstract (translated)
预训练的大型文本-图像（T2I）模型，特别是适当的文本提示，在定制图像生成领域引起了越来越多的兴趣。然而，灾难性遗忘问题使得在保留学习到的样式的同时，持续生成新的用户提供样式变得困难。在本文中，我们提出了一种名为MuseumMaker的方法，使您能够在无尽的方式下根据一组自定义样式合成图像，并逐渐积累这些创意艺术作品作为一个博物馆。当面临新的定制样式时，我们开发了一种风格蒸馏损失模块，将整个数据集的风格传递给图像生成。它可以减小由于图片内容而产生的学习偏移，并解决由少样本图像引起的灾难性过拟合问题。为了处理过去学习到的样式中的灾难性遗忘，我们为共享LoRA模块设计了双重正则化，以优化模型更新方向，分别从权重和特征方面对扩散模型进行正则。同时，通过任务级别的标记学习模块，学习到一个与新样式对应的独特标记嵌入，这可以保留过去样式的历史知识，同时限制LoRA参数的数量。随着任何新的用户提供样式，我们的MuseumMaker可以捕捉到新风格的细微差别，同时保留学习到的样式的细节。在多样风格数据集上的实验结果证实了我们对MuseumMaker方法的有效性，展示了其在各种场景的稳健性和多样性。
URL
https://arxiv.org/abs/2404.16612
