Abstract
Effective human-robot interaction requires emotionally rich multimodal expressions, yet most humanoid robots lack coordinated speech, facial expressions, and gestures. Meanwhile, real-world deployment demands on-device solutions that can operate autonomously without continuous cloud connectivity. To bridging \underline{\textit{S}}peech, \underline{\textit{E}}motion, and \underline{\textit{M}}otion, we present \textit{SeM$^2$}, a Vision Language Model-based framework that orchestrates emotionally coherent multimodal interactions through three key components: a multimodal perception module capturing user contextual cues, a Chain-of-Thought reasoning for response planning, and a novel Semantic-Sequence Aligning Mechanism (SSAM) that ensures precise temporal coordination between verbal content and physical expressions. We implement both cloud-based and \underline{\textit{e}}dge-deployed versions (\textit{SeM$^2_e$}), with the latter knowledge distilled to operate efficiently on edge hardware while maintaining 95\% of the relative performance. Comprehensive evaluations demonstrate that our approach significantly outperforms unimodal baselines in naturalness, emotional clarity, and modal coherence, advancing socially expressive humanoid robotics for diverse real-world environments.
Abstract (translated)
有效的类人机器人交互需要丰富的情感多模态表达，然而大多数的类人机器人在语音、面部表情和手势方面的协调能力不足。同时，实际部署要求设备能够在没有持续云连接的情况下自主运行。为了填补**S**peech（言语）、**E**motion（情感）和**M**otion（动作）之间的空白，我们提出了基于视觉语言模型框架的SeM$^2$，该框架通过三个关键组件协调情感一致性的多模态交互：一个多模态感知模块捕捉用户上下文线索、一个Chain-of-Thought推理用于响应规划以及一种新颖的语义序列对齐机制（SSAM），确保口头内容和物理表达在时间上精确同步。我们实现了基于云和**e**dge部署版本（SeM$^2_e$），后者通过知识蒸馏可以在边缘硬件上高效运行，同时保持95%以上的相对性能。全面的评估表明，我们的方法在自然性、情感清晰度以及模态一致性方面显著优于单模基线，在各种实际环境中推进了社会表达型类人机器人的发展。
URL
https://arxiv.org/abs/2602.07434
