Abstract
As human-robot collaboration advances, natural and flexible communication methods are essential for effective robot control. Traditional methods relying on a single modality or rigid rules struggle with noisy or misaligned data as well as with object descriptions that do not perfectly fit the predefined object names (e.g. 'Pick that red object'). We introduce TransforMerger, a transformer-based reasoning model that infers a structured action command for robotic manipulation based on fused voice and gesture inputs. Our approach merges multimodal data into a single unified sentence, which is then processed by the language model. We employ probabilistic embeddings to handle uncertainty and we integrate contextual scene understanding to resolve ambiguous references (e.g., gestures pointing to multiple objects or vague verbal cues like "this"). We evaluate TransforMerger in simulated and real-world experiments, demonstrating its robustness to noise, misalignment, and missing information. Our results show that TransforMerger outperforms deterministic baselines, especially in scenarios requiring more contextual knowledge, enabling more robust and flexible human-robot communication. Code and datasets are available at: this http URL.
Abstract (translated)
随着人机协作的不断发展，自然且灵活的通信方法对于有效的机器人控制至关重要。传统的方法依赖单一模态或刚性规则，在处理嘈杂或对齐不良的数据时以及面对不完全符合预定义对象名称的对象描述（例如“拿起那个红色物品”）时会遇到困难。我们引入了TransforMerger，这是一种基于变压器的推理模型，它能够根据融合的声音和手势输入推断出机器人操作的结构化动作命令。我们的方法将多模态数据合并为一个统一的句子，然后由语言模型进行处理。我们采用概率嵌入来处理不确定性，并整合上下文场景理解以解决模糊引用问题（例如，指向多个对象的手势或含糊不清的语言提示如“这个”）。我们在模拟和现实世界实验中评估了TransforMerger的表现，展示了其在面对噪音、不对齐以及信息缺失时的鲁棒性。我们的结果显示，TransforMerger超越了确定性的基线模型，在需要更多上下文知识的情况下表现尤为出色，从而实现了更加强大且灵活的人机通信。代码和数据集可在以下链接获取：[此网址]（请将[此网址]替换为实际提供的URL）。
URL
https://arxiv.org/abs/2504.01708
