Abstract
Compared to the prosperity of pre-training models in natural image understanding, the research on large-scale pre-training models for facial knowledge learning is still limited. Current approaches mainly rely on manually assembled and annotated face datasets for training, but labeling such datasets is labor-intensive and the trained models have limited scalability beyond the training data. To address these limitations, we present a generative pre-training model for facial knowledge learning that leverages large-scale web-built data for training. We use texts and images containing human faces crawled from the internet and conduct pre-training on self-supervised tasks, including masked image/language modeling (MILM) and image-text matching (ITM). During the generation stage, we further utilize the image-text matching loss to pull the generation distribution towards the control signal for controllable image/text generation. Experimental results demonstrate that our model achieves comparable performance to state-of-the-art pre-training models for various facial downstream tasks, such as attribution classification and expression recognition. Furthermore, our approach is also applicable to a wide range of face editing tasks, including face attribute editing, expression manipulation, mask removal, and photo inpainting.
Abstract (translated)
与自然图像理解中预训练模型的繁荣相比，针对面部知识学习的大规模预训练研究仍处于初级阶段。目前的方法主要依赖于手动收集和标注的人脸数据集进行训练，但这种标注过程费时费力，并且所训练出的模型在超出训练数据范围后可扩展性有限。为了解决这些问题，我们提出了一种基于大规模网络构建的数据来训练的生成式预训练模型，用于面部知识学习。我们从互联网上抓取包含人脸的文本和图像，并进行包括掩码图像/语言建模（MILM）和图像-文本匹配（ITM）在内的自监督任务的预训练。在生成阶段，进一步利用图像-文本匹配损失将生成分布拉向控制信号，以实现可控的图像/文本生成。 实验结果表明，在各种面部下游任务中，如属性分类和表情识别方面，我们的模型实现了与当前最先进的预训练模型相当的表现。此外，该方法还适用于广泛的面部编辑任务，包括人脸属性编辑、表情操作、口罩去除以及照片修补等。
URL
https://arxiv.org/abs/2510.18345
