Abstract
In conversation, humans use multimodal cues, such as speech, gestures, and gaze, to manage turn-taking. While linguistic and acoustic features are informative, gestures provide complementary cues for modeling these transitions. To study this, we introduce DnD Gesture++, an extension of the multi-party DnD Gesture corpus enriched with 2,663 semantic gesture annotations spanning iconic, metaphoric, deictic, and discourse types. Using this dataset, we model turn-taking prediction through a Mixture-of-Experts framework integrating text, audio, and gestures. Experiments show that incorporating semantically guided gestures yields consistent performance gains over baselines, demonstrating their complementary role in multimodal turn-taking.
Abstract (translated)
在对话中，人类使用多种模态的线索（如言语、手势和目光交流）来管理轮流发言。虽然语言和声学特征具有信息性，但手势提供了互补性的线索，有助于建模这些转换过程。为了研究这一问题，我们引入了DnD Gesture++，这是一个扩展版的多方对话DnD Gesture语料库，其中新增了2,663个语义手势标注，涵盖了象形、隐喻、指示和话语类型的手势。利用这个数据集，我们通过混合专家框架整合文本、音频和手势来建模轮流发言预测。实验表明，引入由语义指导的手势能持续地优于基线模型，在多模态轮流发言中展示了它们的互补作用。
URL
https://arxiv.org/abs/2510.19350
