Abstract
We propose GaussianTalker, a novel framework for real-time generation of pose-controllable talking heads. It leverages the fast rendering capabilities of 3D Gaussian Splatting (3DGS) while addressing the challenges of directly controlling 3DGS with speech audio. GaussianTalker constructs a canonical 3DGS representation of the head and deforms it in sync with the audio. A key insight is to encode the 3D Gaussian attributes into a shared implicit feature representation, where it is merged with audio features to manipulate each Gaussian attribute. This design exploits the spatial-aware features and enforces interactions between neighboring points. The feature embeddings are then fed to a spatial-audio attention module, which predicts frame-wise offsets for the attributes of each Gaussian. It is more stable than previous concatenation or multiplication approaches for manipulating the numerous Gaussians and their intricate parameters. Experimental results showcase GaussianTalker's superiority in facial fidelity, lip synchronization accuracy, and rendering speed compared to previous methods. Specifically, GaussianTalker achieves a remarkable rendering speed of 120 FPS, surpassing previous benchmarks. Our code is made available at this https URL .
Abstract (translated)
我们提出了GaussianTalker，一种用于实时生成具有姿态控制的说话头的新框架。它利用了3D高斯平滑（3DGS）的快速渲染能力，同时解决了直接用语音音频控制3DGS的挑战。GaussianTalker构建了一个规范的3DGS头部的表示，并与其同步变形。一个关键的见解是将3D高斯属性编码成一个共享的隐式特征表示，其中它与音频特征合并以操纵每个高斯属性。这种设计利用了空间感知特征，并强制处理相邻点之间的交互。然后将特征嵌入 feed 到一个空间-音频关注模块，该模块预测每个高斯属性的时偏移。与之前的方法相比，GaussianTalker在面部保真度、 lip 同步准确性和渲染速度方面具有优越性。具体来说，GaussianTalker实现了令人印象深刻的120 FPS的渲染速度，超过了之前的基准。我们的代码可在此处访问的 URL 下载。
URL
https://arxiv.org/abs/2404.16012
