Abstract
We introduce GaussianSwap, a novel video face swapping framework that constructs a 3D Gaussian Splatting based face avatar from a target video while transferring identity from a source image to the avatar. Conventional video swapping frameworks are limited to generating facial representations in pixel-based formats. The resulting swapped faces exist merely as a set of unstructured pixels without any capacity for animation or interactive manipulation. Our work introduces a paradigm shift from conventional pixel-based video generation to the creation of high-fidelity avatar with swapped faces. The framework first preprocesses target video to extract FLAME parameters, camera poses and segmentation masks, and then rigs 3D Gaussian splats to the FLAME model across frames, enabling dynamic facial control. To ensure identity preserving, we propose an compound identity embedding constructed from three state-of-the-art face recognition models for avatar finetuning. Finally, we render the face-swapped avatar on the background frames to obtain the face-swapped video. Experimental results demonstrate that GaussianSwap achieves superior identity preservation, visual clarity and temporal consistency, while enabling previously unattainable interactive applications.
Abstract (translated)
我们介绍了一种新的视频面部交换框架——GaussianSwap，该框架通过从目标视频构建基于3D高斯点阵的脸部虚拟形象，并将源图像的身份信息转移到虚拟形象中来实现面部替换。传统的视频面部交换框架仅限于生成像素格式的面部表示形式，而置换后的脸部仅仅是无结构的一组像素集合，缺乏动画和互动操作的能力。 我们的工作打破了传统基于像素的视频生成模式，实现了高保真度、具备身份交换功能的虚拟形象创建。该框架首先对目标视频进行预处理以提取FLAME参数（面部模型）、相机姿态以及分割掩码，并在不同帧之间将3D高斯点阵绑定到FLAME模型上，从而实现动态面部控制。 为了确保身份一致性，我们提出了一种由三种最先进的面部识别模型组成的复合身份嵌入方法，用于虚拟形象的微调。最后，我们将面部交换后的虚拟形象渲染回背景帧中以生成最终的面部交换视频。 实验结果表明，GaussianSwap在身份保持、视觉清晰度和时间一致性方面表现出色，并且实现了以前难以实现的互动应用功能。
URL
https://arxiv.org/abs/2601.05511
