Abstract
Sign language is a natural and visual form of language that uses movements and expressions to convey meaning, serving as a crucial means of communication for individuals who are deaf or hard-of-hearing (DHH). However, the number of people proficient in sign language remains limited, highlighting the need for technological advancements to bridge communication gaps and foster interactions with minorities. Based on recent advancements in embodied humanoid robots, we propose SignBot, a novel framework for human-robot sign language interaction. SignBot integrates a cerebellum-inspired motion control component and a cerebral-oriented module for comprehension and interaction. Specifically, SignBot consists of: 1) Motion Retargeting, which converts human sign language datasets into robot-compatible kinematics; 2) Motion Control, which leverages a learning-based paradigm to develop a robust humanoid control policy for tracking sign language gestures; and 3) Generative Interaction, which incorporates translator, responser, and generator of sign language, thereby enabling natural and effective communication between robots and humans. Simulation and real-world experimental results demonstrate that SignBot can effectively facilitate human-robot interaction and perform sign language motions with diverse robots and datasets. SignBot represents a significant advancement in automatic sign language interaction on embodied humanoid robot platforms, providing a promising solution to improve communication accessibility for the DHH community.
Abstract (translated)
手语是一种自然且视觉化的语言形式，通过手势和面部表情来传达意义，是聋人或听力受损者的重要沟通方式。然而，精通手语的人数仍然有限，这凸显了技术进步在弥合沟通差距、促进与少数群体互动方面的必要性。基于近期有关具身类人机器人的进展，我们提出了一种名为SignBot的新框架，旨在实现人类与机器人之间的手语交流。SignBot集成了一个受小脑启发的运动控制组件和一个面向大脑的理解与互动模块。 具体而言，SignBot包括以下三个部分： 1. 运动重定向：将人类手语数据集转换为机器人兼容的动力学模型。 2. 运动控制：利用基于学习的方法来开发一种稳健的人形机器人控制策略，用于跟踪手语手势。 3. 生成互动：整合翻译器、响应者和手语生成器，从而实现机器人与人之间的自然且有效的沟通。 模拟实验和真实世界实验的结果表明，SignBot能够有效促进人类与机器人的交互，并能在不同类型的机器人和数据集中执行各种手语动作。SignBot在具身类人机器人平台上实现了自动手语互动的重大进展，为提高听力受损社区的通信可访问性提供了有前景的解决方案。
URL
https://arxiv.org/abs/2505.24266
