Abstract
Generating full-body human gestures encompassing face, body, hands, and global movements from audio is a valuable yet challenging task in virtual avatar creation. Previous systems focused on tokenizing the human gestures framewisely and predicting the tokens of each frame from the input audio. However, one observation is that the number of frames required for a complete expressive human gesture, defined as granularity, varies among different human gesture patterns. Existing systems fail to model these gesture patterns due to the fixed granularity of their gesture tokens. To solve this problem, we propose a novel framework named Multi-Granular Gesture Generator (M3G) for audio-driven holistic gesture generation. In M3G, we propose a novel Multi-Granular VQ-VAE (MGVQ-VAE) to tokenize motion patterns and reconstruct motion sequences from different temporal granularities. Subsequently, we proposed a multi-granular token predictor that extracts multi-granular information from audio and predicts the corresponding motion tokens. Then M3G reconstructs the human gestures from the predicted tokens using the MGVQ-VAE. Both objective and subjective experiments demonstrate that our proposed M3G framework outperforms the state-of-the-art methods in terms of generating natural and expressive full-body human gestures.
Abstract (translated)
从音频生成包括面部、身体和手部动作在内的全身人类姿态是一项在虚拟化身创建中既有价值又具挑战性的任务。以往的系统主要侧重于逐帧地将人体姿态进行标记，并根据输入的音频预测每帧的姿态标记。然而，一个值得注意的现象是，完成一个具有表现力的人体姿态所需的帧数（定义为粒度）因不同的姿态模式而异。现有的系统由于其固定化的姿态标记粒度无法建模这些多样的姿态模式。为了克服这个问题，我们提出了一种名为多重粒度手势生成器（M3G）的全新框架，用于基于音频驱动的整体手势生成。 在M3G中，我们提出了一个创新性的多重粒度矢量量化变分自编码器（MGVQ-VAE），该模型能够以不同的时间粒度来标记运动模式并重构运动序列。接着，我们设计了一种多粒度令牌预测器，它可以从音频中提取多种不同时间尺度的信息，并预测相应的动作令牌。然后，M3G利用MGVQ-VAE从预测的令牌重建人类姿态。 无论是客观测试还是主观评估都表明，我们的M3G框架在生成自然且富有表现力的整体人体姿势方面优于现有的最先进技术。
URL
https://arxiv.org/abs/2505.08293
