Abstract
Generating gestures from human speech has gained tremendous progress in animating virtual avatars. While the existing methods enable synthesizing gestures cooperated by individual self-talking, they overlook the practicality of concurrent gesture modeling with two-person interactive conversations. Moreover, the lack of high-quality datasets with concurrent co-speech gestures also limits handling this issue. To fulfill this goal, we first construct a large-scale concurrent co-speech gesture dataset that contains more than 7M frames for diverse two-person interactive posture sequences, dubbed GES-Inter. Additionally, we propose Co$^3$Gesture, a novel framework that enables coherent concurrent co-speech gesture synthesis including two-person interactive movements. Considering the asymmetric body dynamics of two speakers, our framework is built upon two cooperative generation branches conditioned on separated speaker audio. Specifically, to enhance the coordination of human postures with respect to corresponding speaker audios while interacting with the conversational partner, we present a Temporal Interaction Module (TIM). TIM can effectively model the temporal association representation between two speakers' gesture sequences as interaction guidance and fuse it into the concurrent gesture generation. Then, we devise a mutual attention mechanism to further holistically boost learning dependencies of interacted concurrent motions, thereby enabling us to generate vivid and coherent gestures. Extensive experiments demonstrate that our method outperforms the state-of-the-art models on our newly collected GES-Inter dataset. The dataset and source code are publicly available at \href{this https URL}{\textit{this https URL}}.
Abstract (translated)
从人类言语生成手势在虚拟角色动画领域取得了重大进展。尽管现有的方法能够合成个人自言自语时的手势，但它们忽视了同时模拟双人互动对话中的手势模型的实际应用性。此外，缺乏高质量的包含同步共话语音手势的数据集也限制了这一问题的处理能力。为了实现这一目标，我们首先构建了一个大规模的并发共话语音手势数据集——GES-Inter，该数据集中包含了超过700万帧用于多样化的双人互动姿势序列。另外，我们提出了Co$^3$Gesture，这是一个全新的框架，能够生成包括两人互动动作在内的连贯的同步共话语音手势。 考虑到两位说话者身体动态的不对称性，我们的框架基于两个根据各自语音条件化生成的合作分支建立而成。具体而言，为了增强人体姿态与相应讲话人音频之间的协调，在与对话伙伴互动时，我们提出了一种时间交互模块（TIM）。TIM可以有效建模两名演讲者手势序列之间的时间关联表示作为交互指导，并将其融入到同步手势生成中。此外，我们设计了一个互注意机制，进一步整体提升相互作用的并发动作的学习依赖性，从而能够生成生动且连贯的手势。 广泛的实验表明，我们的方法在我们新收集的GES-Inter数据集上优于当前最佳模型。该数据集和源代码可以在\href{this https URL}{此链接}公开获取。
URL
https://arxiv.org/abs/2505.01746
