Abstract
We present Social Agent, a novel framework for synthesizing realistic and contextually appropriate co-speech nonverbal behaviors in dyadic conversations. In this framework, we develop an agentic system driven by a Large Language Model (LLM) to direct the conversation flow and determine appropriate interactive behaviors for both participants. Additionally, we propose a novel dual-person gesture generation model based on an auto-regressive diffusion model, which synthesizes coordinated motions from speech signals. The output of the agentic system is translated into high-level guidance for the gesture generator, resulting in realistic movement at both the behavioral and motion levels. Furthermore, the agentic system periodically examines the movements of interlocutors and infers their intentions, forming a continuous feedback loop that enables dynamic and responsive interactions between the two participants. User studies and quantitative evaluations show that our model significantly improves the quality of dyadic interactions, producing natural, synchronized nonverbal behaviors.
Abstract (translated)
我们介绍了Social Agent，这是一个用于生成真实且符合情境的对话语音非言语行为的创新框架。在该框架中，我们开发了一个由大型语言模型（LLM）驱动的代理系统来引导对话流程并确定双方参与者适当的互动行为。此外，我们提出了一种基于自回归扩散模型的新颖双人手势生成模型，该模型可根据语音信号合成协调的动作。代理系统的输出被转换为对动作生成器的高层次指导，从而在行为和运动层面产生真实感的动作。此外，代理系统会定期检查对话双方的动作并推断其意图，形成一个持续反馈循环，使两者的互动更加动态且响应迅速。用户研究和定量评估表明，我们的模型显著提升了双边交互的质量，产生了自然、同步的非言语行为。
URL
https://arxiv.org/abs/2510.04637
