Abstract
Multimodal social interaction understanding (MMSI) is critical in human-robot interaction systems. In real-world scenarios, AI agents are required to provide real-time feedback. However, existing models often depend on both past and future contexts, which hinders them from applying to real-world problems. To bridge this gap, we propose an online MMSI setting, where the model must resolve MMSI tasks using only historical information, such as recorded dialogues and video streams. To address the challenges of missing the useful future context, we develop a novel framework, named Online-MMSI-VLM, that leverages two complementary strategies: multi-party conversation forecasting and social-aware visual prompting with multi-modal large language models. First, to enrich linguistic context, the multi-party conversation forecasting simulates potential future utterances in a coarse-to-fine manner, anticipating upcoming speaker turns and then generating fine-grained conversational details. Second, to effectively incorporate visual social cues like gaze and gesture, social-aware visual prompting highlights the social dynamics in video with bounding boxes and body keypoints for each person and frame. Extensive experiments on three tasks and two datasets demonstrate that our method achieves state-of-the-art performance and significantly outperforms baseline models, indicating its effectiveness on Online-MMSI. The code and pre-trained models will be publicly released at: this https URL.
Abstract (translated)
多模态社会交互理解（MMSI）在人机交互系统中至关重要。在现实场景下，AI代理需要提供实时反馈。然而，现有模型往往依赖于过去的和将来的上下文信息，这阻碍了它们应用于实际问题的能力。为了弥合这一差距，我们提出了一种在线多模态社会交互理解（Online-MMSI）设置，在这种设置中，模型必须仅利用历史信息（如记录的对话和视频流）来解决MMSI任务。 为了解决缺乏有用未来上下文的问题，我们开发了一个新颖的框架，命名为Online-MMSI-VLM，该框架采用了两种互补策略：多党会议预测和基于多人模态大型语言模型的社会感知视觉提示。首先，为了丰富语言背景，多党会议预测以从粗到细的方式模拟潜在的未来的言语行为，在预测即将到来的说话者回合后生成详细的对话细节。其次，为有效整合诸如凝视和手势等视觉社会线索，社会感知视觉提示通过为人像中的每一帧标注边界框和身体关键点来突出视频中的社交动态。 在三个任务和两个数据集上的广泛实验表明，我们的方法达到了最先进的性能，并显著优于基线模型，这证明了它在线MMSI的有效性。代码及预训练模型将在以下网址公开发布：[此链接](this https URL)。
URL
https://arxiv.org/abs/2503.19851
