Abstract
Deictic gestures, like pointing, are a fundamental form of non-verbal communication, enabling humans to direct attention to specific objects or locations. This capability is essential in Human-Robot Interaction (HRI), where robots should be able to predict human intent and anticipate appropriate responses. In this work, we propose the Multi-Modality Inter-TransFormer (MM-ITF), a modular architecture to predict objects in a controlled tabletop scenario with the NICOL robot, where humans indicate targets through natural pointing gestures. Leveraging inter-modality attention, MM-ITF maps 2D pointing gestures to object locations, assigns a likelihood score to each, and identifies the most likely target. Our results demonstrate that the method can accurately predict the intended object using monocular RGB data, thus enabling intuitive and accessible human-robot collaboration. To evaluate the performance, we introduce a patch confusion matrix, providing insights into the model's predictions across candidate object locations. Code available at: this https URL.
Abstract (translated)
指示性手势（如指向）是非语言沟通的一种基本形式，能够帮助人类将注意力集中到特定的对象或位置上。这种能力在人机交互（HRI）中至关重要，因为机器人需要具备预测人的意图并作出适当回应的能力。在这项工作中，我们提出了一种模块化架构——多模态互变换器（Multi-Modality Inter-TransFormer, MM-ITF），用于在一个受控的桌面上与NICOL机器人互动的情境下，通过自然的手指指向手势来预测目标对象。利用跨模式注意机制，MM-ITF将2D指向手势映射到物体位置上，并为每个可能的目标分配一个可能性分数，从而确定最有可能的目标。我们的实验结果表明，该方法能够仅使用单目RGB数据准确地预测意图中的对象，从而使人类与机器人的协作更加直观和易于实现。为了评估性能，我们引入了一个补丁混淆矩阵，提供了模型在候选目标位置上预测情况的深入见解。 代码可在以下网址获取：this https URL.
URL
https://arxiv.org/abs/2509.05031
