Abstract
With the rising interest from the community in digital avatars coupled with the importance of expressions and gestures in communication, modeling natural avatar behavior remains an important challenge across many industries such as teleconferencing, gaming, and AR/VR. Human hands are the primary tool for interacting with the environment and essential for realistic human behavior modeling, yet existing 3D hand and head avatar models often overlook the crucial aspect of hand-body interactions, such as between hand and face. We present InteracttAvatar, the first model to faithfully capture the photorealistic appearance of dynamic hand and non-rigid hand-face interactions. Our novel Dynamic Gaussian Hand model, combining template model and 3D Gaussian Splatting as well as a dynamic refinement module, captures pose-dependent change, e.g. the fine wrinkles and complex shadows that occur during articulation. Importantly, our hand-face interaction module models the subtle geometry and appearance dynamics that underlie common gestures. Through experiments of novel view synthesis, self reenactment and cross-identity reenactment, we demonstrate that InteracttAvatar can reconstruct hand and hand-face interactions from monocular or multiview videos with high-fidelity details and be animated with novel poses.
Abstract (translated)
随着社区对数字虚拟形象的兴趣日益增加，加之表情和手势在交流中的重要性，模拟自然的虚拟形象行为对于电信会议、游戏以及AR/VR等行业而言仍是一个重要的挑战。人类的手是与环境互动的主要工具，同时也是现实主义的人类行为建模中不可或缺的一部分，然而现有的3D手部和头部虚拟模型往往忽视了手部与身体其他部位（如脸部）之间的重要交互作用。我们提出了一种名为InteracttAvatar的模型，这是首个能够忠实捕捉动态手部及非刚性手脸互动的逼真外观的技术。 我们的新Dynamic Gaussian Hand模型结合了模板模型和3D高斯点阵技术，并且包括了一个动态细化模块，可以捕获姿势相关的变形，比如在手指关节活动时产生的细微皱纹和复杂的阴影。尤为重要的是，我们设计的手部与脸部交互模块能够模拟出常见手势背后的微妙几何变化和外观动态。 通过新颖视角合成、自我再现以及跨身份再现的实验，我们证明了InteracttAvatar可以从单目或多视图视频中重建手部及手脸互动，并且可以以新的姿态进行动画化，同时保持高保真的细节。
URL
https://arxiv.org/abs/2504.07949
