Abstract
Human communication involves a complex interplay of verbal and nonverbal signals, essential for conveying meaning and achieving interpersonal goals. To develop socially intelligent AI technologies, it is crucial to develop models that can both comprehend and generate dyadic behavioral dynamics. To this end, we introduce the Seamless Interaction Dataset, a large-scale collection of over 4,000 hours of face-to-face interaction footage from over 4,000 participants in diverse contexts. This dataset enables the development of AI technologies that understand dyadic embodied dynamics, unlocking breakthroughs in virtual agents, telepresence experiences, and multimodal content analysis tools. We also develop a suite of models that utilize the dataset to generate dyadic motion gestures and facial expressions aligned with human speech. These models can take as input both the speech and visual behavior of their interlocutors. We present a variant with speech from an LLM model and integrations with 2D and 3D rendering methods, bringing us closer to interactive virtual agents. Additionally, we describe controllable variants of our motion models that can adapt emotional responses and expressivity levels, as well as generating more semantically-relevant gestures. Finally, we discuss methods for assessing the quality of these dyadic motion models, which are demonstrating the potential for more intuitive and responsive human-AI interactions.
Abstract (translated)
人类沟通涉及复杂的言语和非言语信号的互动，这对传达意义和实现人际目标至关重要。为了开发具有社会智能的人工智能技术，需要建立既能理解又能生成二元行为动态（即两人之间的互动）的模型。为此，我们引入了“无缝交互数据集”，这是一个大规模的数据集合，包含了超过4,000名参与者在各种情境下面对面交流的超过4,000小时的视频记录。该数据集有助于开发能够理解二元身体互动的人工智能技术，从而推动虚拟代理、远程呈现体验和多模态内容分析工具的发展。 我们还开发了一系列模型，这些模型利用数据集生成与人类语言相匹配的一对一动作手势和面部表情。这些模型可以接受对话伙伴的言语和视觉行为作为输入。我们提出了一种变体，其中使用大型语言模型（LLM）产生的语音，并结合了2D和3D渲染方法，这使我们更接近于实现交互式的虚拟代理。 此外，我们还描述了我们的动作模型可控制的变体版本，这些版本可以适应情感反应的不同水平并调整表情强度，同时生成更具语义相关性的手势。最后，我们讨论了评估这些二元动作模型质量的方法，展示了人与AI之间更直观和响应互动的巨大潜力。
URL
https://arxiv.org/abs/2506.22554
