Abstract
Full-body gestures play a pivotal role in natural interactions and are crucial for achieving effective communication. Nevertheless, most existing studies primarily focus on the gesture generation of speakers, overlooking the vital role of listeners in the interaction process and failing to fully explore the dynamic interaction between them. This paper innovatively proposes an Inter-Diffusion Generation Model of Speakers and Listeners for Effective Communication. For the first time, we integrate the full-body gestures of listeners into the generation framework. By devising a novel inter-diffusion mechanism, this model can accurately capture the complex interaction patterns between speakers and listeners during communication. In the model construction process, based on the advanced diffusion model architecture, we innovatively introduce interaction conditions and the GAN model to increase the denoising step size. As a result, when generating gesture sequences, the model can not only dynamically generate based on the speaker's speech information but also respond in realtime to the listener's feedback, enabling synergistic interaction between the two. Abundant experimental results demonstrate that compared with the current state-of-the-art gesture generation methods, the model we proposed has achieved remarkable improvements in the naturalness, coherence, and speech-gesture synchronization of the generated gestures. In the subjective evaluation experiments, users highly praised the generated interaction scenarios, believing that they are closer to real life human communication situations. Objective index evaluations also show that our model outperforms the baseline methods in multiple key indicators, providing more powerful support for effective communication.
Abstract (translated)
全身手势在自然互动中扮演着关键角色，并且对于实现有效沟通至关重要。然而，大多数现有研究主要关注说话者的手势生成，忽略了倾听者在互动过程中的重要性，并未能充分探索双方之间的动态交互作用。本文创新地提出了一种用于有效沟通的演讲者与倾听者之间相互扩散生成模型（Inter-Diffusion Generation Model of Speakers and Listeners）。首次将听众的全身手势融入到生成框架中，通过设计一种新颖的互扩散机制，该模型能够准确捕捉说话人和倾听人在交流过程中复杂的交互模式。在模型构建过程中，基于先进的扩散模型架构，我们创新性地引入了互动条件以及GAN（生成对抗网络）模型，以增加去噪步骤大小。因此，在生成手势序列时，该模型不仅能根据发言人的语言信息动态生成，还能实时回应听者的反馈，从而实现两者之间的协同交互作用。 大量的实验结果表明，与当前最先进的手势生成方法相比，我们提出的模型在所生成手势的自然性、连贯性和话语-手势同步性方面取得了显著改进。在主观评价实验中，用户对生成的互动场景给予了高度赞扬，认为它们更接近现实生活中的交流情况。客观指标评估也显示，我们的模型在多个关键指标上超越了基准方法，为有效沟通提供了更为强大的支持。
URL
https://arxiv.org/abs/2505.04996
