Abstract
We address the problem of accurate capture of interactive behaviors between two people in daily scenarios. Most previous works either only consider one person or solely focus on conversational gestures of two people, assuming the body orientation and/or position of each actor are constant or barely change over each interaction. In contrast, we propose to simultaneously model two people's activities, and target objective-driven, dynamic, and semantically consistent interactions which often span longer duration and cover bigger space. To this end, we capture a new multi-modal dataset dubbed InterAct, which is composed of 241 motion sequences where two people perform a realistic and coherent scenario for one minute or longer over a complete interaction. For each sequence, two actors are assigned different roles and emotion labels, and collaborate to finish one task or conduct a common interaction activity. The audios, body motions, and facial expressions of both persons are captured. InterAct contains diverse and complex motions of individuals and interesting and relatively long-term interaction patterns barely seen before. We also demonstrate a simple yet effective diffusion-based method that estimates interactive face expressions and body motions of two people from speech inputs. Our method regresses the body motions in a hierarchical manner, and we also propose a novel fine-tuning mechanism to improve the lip accuracy of facial expressions. To facilitate further research, the data and code is made available at this https URL .
Abstract (translated)
我们解决了在日常场景中准确捕捉两人互动行为的问题。大多数先前的工作要么只考虑一个人，要么仅关注两个人之间的对话手势，假设每个演员的身体朝向和/或位置保持不变或变化很小。相比之下，我们提出同时建模两人的活动，并针对目标驱动的、动态且语义一致的交互，这些交互通常持续时间较长并覆盖更大的空间范围。为此，我们收集了一个新的多模态数据集，名为InterAct，该数据集包含241个动作序列，在每个序列中两人共同表演一个真实且连贯的情景超过一分钟或更长时间直至完成整个互动过程。对于每一个序列，两名演员被分配不同的角色和情感标签，并合作完成一项任务或者进行一种常见的交互活动。捕捉了两个人的声音、身体动作和面部表情。 InterAct数据集包含个人多样化且复杂的动作以及以前很少见的有趣而相对长期的人际互动模式。我们还演示了一种简单但有效的基于扩散的方法，该方法能够根据语音输入估算出两人的互动面部表情和身体动作。我们的方法以分层的方式回归身体运动，并提出了一种新颖的微调机制来提高面部表情的唇部精度。 为了促进进一步的研究，数据集和代码可在以下链接获得：[此URL]。
URL
https://arxiv.org/abs/2509.05747
