Abstract
Recent work has demonstrated a remarkable ability to customize text-to-image diffusion models to multiple, fine-grained concepts in a sequential (i.e., continual) manner while only providing a few example images for each concept. This setting is known as continual diffusion. Here, we ask the question: Can we scale these methods to longer concept sequences without forgetting? Although prior work mitigates the forgetting of previously learned concepts, we show that its capacity to learn new tasks reaches saturation over longer sequences. We address this challenge by introducing a novel method, STack-And-Mask INcremental Adapters (STAMINA), which is composed of low-ranked attention-masked adapters and customized MLP tokens. STAMINA is designed to enhance the robust fine-tuning properties of LoRA for sequential concept learning via learnable hard-attention masks parameterized with low rank MLPs, enabling precise, scalable learning via sparse adaptation. Notably, all introduced trainable parameters can be folded back into the model after training, inducing no additional inference parameter costs. We show that STAMINA outperforms the prior SOTA for the setting of text-to-image continual customization on a 50-concept benchmark composed of landmarks and human faces, with no stored replay data. Additionally, we extended our method to the setting of continual learning for image classification, demonstrating that our gains also translate to state-of-the-art performance in this standard benchmark.
Abstract (translated)
近年来，工作展示了将文本到图像扩散模型定制到多个细粒度概念并在序列（即连续）方式下进行定制，而只需为每个概念提供几张示例图像的显著能力。这个设置被称为连续扩散。在这里，我们问一个问题：我们能否将这种方法扩展到更长的概念序列，而不遗忘？尽管先前的研究减轻了之前学习概念的遗忘，但我们证明了在更长的序列中学习新任务的能力已经达到了饱和。为了解决这个问题，我们引入了一种新的方法：STack-And-Mask INcremental Adapters（STAMINA），它由低秩注意力掩码的适应器组成，并定制了自定义的MLP标记。STAMINA旨在通过可学习的高注意度掩码参数化低秩MLP，增强LoRA在序列概念学习中的鲁棒性，实现精确、可扩展的学习，通过稀疏适应。值得注意的是，所有引入的可训练参数在训练后都可以折叠回到模型中，从而不会产生额外的推理参数成本。我们在包含50个概念的文本到图像连续定制设置上评估了STAMINA，发现它在该设置上超过了先前的SOTA。此外，我们还将我们的方法扩展到图像分类的连续学习设置中，证明了我们的收获同样可以应用于这个标准基准。
URL
https://arxiv.org/abs/2311.18763
