Abstract
Recent advances in large-scale text-to-image generation models have led to a surge in subject-driven text-to-image generation, which aims to produce customized images that align with textual descriptions while preserving the identity of specific subjects. Despite significant progress, current methods struggle to disentangle identity-relevant information from identity-irrelevant details in the input images, resulting in overfitting or failure to maintain subject identity. In this work, we propose a novel framework that improves the separation of identity-related and identity-unrelated features and introduces an innovative feature fusion mechanism to improve the quality and text alignment of generated images. Our framework consists of two key components: an Implicit-Explicit foreground-background Decoupling Module (IEDM) and a Feature Fusion Module (FFM) based on a Mixture of Experts (MoE). IEDM combines learnable adapters for implicit decoupling at the feature level with inpainting techniques for explicit foreground-background separation at the image level. FFM dynamically integrates identity-irrelevant features with identity-related features, enabling refined feature representations even in cases of incomplete decoupling. In addition, we introduce three complementary loss functions to guide the decoupling process. Extensive experiments demonstrate the effectiveness of our proposed method in enhancing image generation quality, improving flexibility in scene adaptation, and increasing the diversity of generated outputs across various textual descriptions.
Abstract (translated)
近期在大规模文本到图像生成模型方面的进展引发了以主题为导向的文本到图像生成领域的激增，该领域旨在根据文本描述生成定制化且与特定主体身份一致的图像。尽管已有显著进步，但当前方法仍难以从输入图像中分离出与身份相关的信息和无关信息，导致过度拟合或无法保持主体的身份特征。为此，我们提出了一种新的框架，以改进身份相关和不相关的特性的分离，并引入了一个基于专家混合（MoE）的创新特性融合机制来提高生成图像的质量及其与文本描述的一致性。 该框架包含两个关键组件：隐式-显式前景背景解耦模块（IEDM）以及特征融合模块（FFM）。IEDM结合了可学习适配器用于在特征级别上进行隐式分离，同时采用修复技术进行图像级别的显式前景背景分割。FFM动态地将身份无关的特性与身份相关的特性相结合，在不完全分离的情况下也能实现精细的特征求解表示。 此外，我们还引入了三种互补的损失函数来指导整个解耦过程。大量的实验展示了我们的方法在提高图像生成质量、增强场景适应性灵活性以及增加根据各种文本描述产生的输出多样性方面具有显著效果。
URL
https://arxiv.org/abs/2505.22360
