Abstract
Synthesizing synchronized and natural co-speech gesture videos remains a formidable challenge. Recent approaches have leveraged motion graphs to harness the potential of existing video data. To retrieve an appropriate trajectory from the graph, previous methods either utilize the distance between features extracted from the input audio and those associated with the motions in the graph or embed both the input audio and motion into a shared feature space. However, these techniques may not be optimal due to the many-to-many mapping nature between audio and gestures, which cannot be adequately addressed by one-to-one mapping. To alleviate this limitation, we propose a novel framework that initially employs a diffusion model to generate gesture motions. The diffusion model implicitly learns the joint distribution of audio and motion, enabling the generation of contextually appropriate gestures from input audio sequences. Furthermore, our method extracts both low-level and high-level features from the input audio to enrich the training process of the diffusion model. Subsequently, a meticulously designed motion-based retrieval algorithm is applied to identify the most suitable path within the graph by assessing both global and local similarities in motion. Given that not all nodes in the retrieved path are sequentially continuous, the final step involves seamlessly stitching together these segments to produce a coherent video output. Experimental results substantiate the efficacy of our proposed method, demonstrating a significant improvement over prior approaches in terms of synchronization accuracy and naturalness of generated gestures.
Abstract (translated)
合成同步且自然的伴随言语的手势视频仍然是一个艰巨的任务。近期的研究方法利用了运动图，以发掘现有视频数据的潜力。为了从图中检索合适的轨迹，先前的方法要么采用输入音频与图中动作特征之间的距离来获取相应的路径，要么将输入音频和动作嵌入到共享的特征空间中。然而，由于声音与手势之间存在多对多映射关系，这两种技术可能并非最优选择，因为这种复杂性不能仅通过一对一映射解决。 为了解决这一限制，我们提出了一种新的框架，首先利用扩散模型生成手势运动。该扩散模型隐式地学习了音频和动作的联合分布，从而能够从输入音频序列中产生语境相关的手势。此外，我们的方法同时提取输入音频中的低级和高级特征，以丰富扩散模型的训练过程。 随后，我们设计了一个精细的动作基础检索算法，通过评估全局和局部运动相似性来识别图中最合适的路径。考虑到在检索到的路径上并非所有节点都连续相连，最后一步需要将这些片段无缝拼接起来，生成一个连贯的视频输出。 实验结果证实了我们方法的有效性，展示了相比之前的方法，在同步准确性和手势自然度方面有显著改进。
URL
https://arxiv.org/abs/2512.02576
