Abstract
Existing diffusion models show great potential for identity-preserving generation. However, personalized portrait generation remains challenging due to the diversity in user profiles, including variations in appearance and lighting conditions. To address these challenges, we propose IC-Portrait, a novel framework designed to accurately encode individual identities for personalized portrait generation. Our key insight is that pre-trained diffusion models are fast learners (e.g.,100 ~ 200 steps) for in-context dense correspondence matching, which motivates the two major designs of our IC-Portrait framework. Specifically, we reformulate portrait generation into two sub-tasks: 1) Lighting-Aware Stitching: we find that masking a high proportion of the input image, e.g., 80%, yields a highly effective self-supervisory representation learning of reference image lighting. 2) View-Consistent Adaptation: we leverage a synthetic view-consistent profile dataset to learn the in-context correspondence. The reference profile can then be warped into arbitrary poses for strong spatial-aligned view conditioning. Coupling these two designs by simply concatenating latents to form ControlNet-like supervision and modeling, enables us to significantly enhance the identity preservation fidelity and stability. Extensive evaluations demonstrate that IC-Portrait consistently outperforms existing state-of-the-art methods both quantitatively and qualitatively, with particularly notable improvements in visual qualities. Furthermore, IC-Portrait even demonstrates 3D-aware relighting capabilities.
Abstract (translated)
现有的扩散模型在保持身份特性的生成方面展现出了巨大的潜力。然而，由于用户资料的多样性（包括外观和光照条件的变化），个性化肖像生成仍然面临挑战。为了解决这些问题，我们提出了IC-Portrait，这是一个全新的框架，旨在准确编码个人身份以进行个性化的肖像生成。我们的关键见解是，预训练的扩散模型在上下文中密集对应匹配方面学习迅速（例如100到200步），这激励了我们IC-Portrait框架的主要设计。 具体而言，我们将肖像生成重新表述为两个子任务： 1. **光照感知拼接**：我们发现对输入图像进行高度遮挡处理（例如80%），可以非常有效地学习参考图的自监督表示，从而更好地理解光照条件。 2. **视图一致性适应**：利用合成的视图一致资料集来学习上下文中的对应关系。这使得参考轮廓能够被变形到任意姿势，从而提供强大的空间对齐视图调节。 通过简单地连接潜在变量形成类似ControlNet的监督和建模方式，将这两种设计结合起来，我们显著增强了身份保持的准确度和稳定性。广泛的评估显示，IC-Portrait在定量和定性评价中均优于现有的最先进的方法，并且在视觉质量方面有了特别明显的改进。此外，IC-Portrait还展示了3D感知重光照的能力。
URL
https://arxiv.org/abs/2501.17159
