Abstract
Speech-driven talking heads have recently emerged and enable interactive avatars. However, real-world applications are limited, as current methods achieve high visual fidelity but slow or fast yet temporally unstable. Diffusion methods provide realistic image generation, yet struggle with oneshot settings. Gaussian Splatting approaches are real-time, yet inaccuracies in facial tracking, or inconsistent Gaussian mappings, lead to unstable outputs and video artifacts that are detrimental to realistic use cases. We address this problem by mapping Gaussian Splatting using 3D Morphable Models to generate person-specific avatars. We introduce transformer-based prediction of model parameters, directly from audio, to drive temporal consistency. From monocular video and independent audio speech inputs, our method enables generation of real-time talking head videos where we report competitive quantitative and qualitative performance.
Abstract (translated)
基于语音驱动的虚拟头像（说话头部）技术最近发展迅速，使互动式化身成为可能。然而，现实世界中的应用仍然受到限制，因为目前的方法虽然能实现高视觉保真度，但存在速度慢或快而不稳定的问题。扩散方法能够生成逼真的图像，但在一次性的设置中表现不佳。Gaussian Splatting 方法实现了实时性能，但由于面部跟踪不准确或者高斯映射的一致性问题，会导致输出不稳定和视频伪影，这对真实使用场景不利。 为了解决这些问题，我们通过将3D Morphable Models（三维形态模型）与 Gaussian Splatting 结合起来生成特定于个人的头像。此外，我们引入了基于变压器的预测方法，直接从音频中预测模型参数，从而驱动时间上的稳定性。我们的方法可以从单目视频和独立的语音输入中生成实时说话头部视频，并在定量和定性评估上都表现出竞争力。
URL
https://arxiv.org/abs/2512.10939
