Abstract
The need for social robots and agents to interact and assist humans is growing steadily. To be able to successfully interact with humans, they need to understand and analyse socially interactive scenes from their (robot's) perspective. Works that model social situations between humans and agents are few; and even those existing ones are often too computationally intensive to be suitable for deployment in real time or on real world scenarios with limited available information. We propose a robust knowledge distillation framework that models social interactions through various multimodal cues, yet is robust against incomplete and noisy information during inference. Our teacher model is trained with multimodal input (body, face and hand gestures, gaze, raw images) that transfers knowledge to a student model that relies solely on body pose. Extensive experiments on two publicly available human-robot interaction datasets demonstrate that the our student model achieves an average accuracy gain of 14.75\% over relevant baselines on multiple downstream social understanding task even with up to 51\% of its input being corrupted. The student model is highly efficient: it is $<1$\% in size of the teacher model in terms of parameters and uses $\sim 0.5$\textperthousand~FLOPs of that in the teacher model. Our code will be made public during publication.
Abstract (translated)
社会机器人和代理与人类互动并提供帮助的需求正在稳步增长。为了能够成功地与人类交互，它们需要从自身的（机器人的）视角理解并分析社交场景。然而，模拟人机之间社交情景的作品并不多；即使现有的作品也往往计算复杂度高，不适合在实际场景中实时部署或处理信息有限的情况。我们提出了一种鲁棒的知识蒸馏框架，该框架通过多种多模态线索来建模社会互动，并且能够应对推理过程中的不完整和噪音信息。我们的教师模型使用多模态输入（身体、面部及手部动作、视线以及原始图像）进行训练，并将知识传递给仅依赖于身体姿态的学徒模型。在两个公开的人机交互数据集上的大量实验表明，即使有高达51%的输入被破坏的情况下，我们的学徒模型在多个下游社交理解任务上仍比相关基准提高了平均14.75%的准确度。该学生模型非常高效：它的参数量仅为教师模型的不到1%，并且使用的计算资源（FLOPs）约为教师模型的0.5‰左右。我们将在论文发表时公开代码。
URL
https://arxiv.org/abs/2505.06278
