Abstract
Recent advances in diffusion models have significantly improved text-to-face generation, but achieving fine-grained control over facial features remains a challenge. Existing methods often require training additional modules to handle specific controls such as identity, attributes, or age, making them inflexible and resource-intensive. We propose ExpertGen, a training-free framework that leverages pre-trained expert models such as face recognition, facial attribute recognition, and age estimation networks to guide generation with fine control. Our approach uses a latent consistency model to ensure realistic and in-distribution predictions at each diffusion step, enabling accurate guidance signals to effectively steer the diffusion process. We show qualitatively and quantitatively that expert models can guide the generation process with high precision, and multiple experts can collaborate to enable simultaneous control over diverse facial aspects. By allowing direct integration of off-the-shelf expert models, our method transforms any such model into a plug-and-play component for controllable face generation.
Abstract (translated)
最近在扩散模型方面的进展显著提升了文本到人脸生成的效果，但实现对面部特征的精细控制仍是一个挑战。现有方法通常需要训练额外模块来处理特定控制因素，如身份、属性或年龄等，这使得它们变得不够灵活且资源消耗大。我们提出了一种名为ExpertGen的新框架，这是一个无需重新训练的方法，它利用预训练的专业模型（例如面部识别、面部属性识别和年龄估算网络）来指导生成过程，并实现精细的控制。 我们的方法使用一个潜在一致性模型，在每次扩散步骤中确保预测的真实性和分布合理性，从而使准确的引导信号能够有效地调节扩散过程。我们从定性和定量两方面展示了专家模型可以以高精度指导生成过程，并且多个专家模型可以协作同时对多方面的面部特征进行控制。通过允许直接集成现成的专业模型，我们的方法将任何这样的模型转化为可插拔组件，用于可控的人脸生成。 这种方法不仅提高了生成人脸图像的灵活性和效率，还为研究者提供了更方便、高效的方式来探索更多关于面部识别与合成的研究方向。
URL
https://arxiv.org/abs/2505.17256
