Abstract
Generating realistic conversational gestures are essential for achieving natural, socially engaging interactions with digital humans. However, existing methods typically map a single audio stream to a single speaker's motion, without considering social context or modeling the mutual dynamics between two people engaging in conversation. We present DyaDiT, a multi-modal diffusion transformer that generates contextually appropriate human motion from dyadic audio signals. Trained on Seamless Interaction Dataset, DyaDiT takes dyadic audio with optional social-context tokens to produce context-appropriate motion. It fuses information from both speakers to capture interaction dynamics, uses a motion dictionary to encode motion priors, and can optionally utilize the conversational partner's gestures to produce more responsive motion. We evaluate DyaDiT on standard motion generation metrics and conduct quantitative user studies, demonstrating that it not only surpasses existing methods on objective metrics but is also strongly preferred by users, highlighting its robustness and socially favorable motion generation. Code and models will be released upon acceptance.
Abstract (translated)
生成逼真的对话手势对于实现自然、社交互动的数字人类交互至关重要。然而，现有方法通常将单一音频流映射到单个发言人的动作上，而不考虑社会背景或建模两个参与对话的人之间的相互动态关系。我们提出了DyaDiT，这是一种多模式扩散变压器，可以从双人音频信号生成上下文相关的肢体动作。在Seamless Interaction Dataset（无缝交互数据集）上训练后，DyaDiT可以接收带有可选社交背景标记的双人音频，并产生相应的适当动作。它融合了双方的信息以捕捉互动动态，使用动作词典来编码动作先验知识，并且可以选择利用对话伙伴的手势生成更灵敏的动作。 我们在标准动作生成指标上评估了DyaDiT，并进行了定量用户研究，证明了它不仅在客观指标上超越现有方法，而且也因其实用性和社交友好性而被用户强烈偏爱。这表明其稳健的上下文适应能力以及能够产生高度自然的社会互动行为。代码和模型将在接受后发布。 这段文字描述了一种新的人工智能系统DyaDiT的功能与优势，该系统能够生成更符合社会情境的动作，并且在用户体验方面有着显著的优势。
URL
https://arxiv.org/abs/2602.23165
