Abstract
Referring expression comprehension (REC) aims at achieving object localization based on natural language descriptions. However, existing REC approaches are constrained by object category descriptions and single-attribute intention descriptions, hindering their application in real-world scenarios. In natural human-robot interactions, users often express their desires through individual states and intentions, accompanied by guiding gestures, rather than detailed object descriptions. To address this challenge, we propose Multi-ref EC, a novel task framework that integrates state descriptions, derived intentions, and embodied gestures to locate target objects. We introduce the State-Intention-Gesture Attributes Reference (SIGAR) dataset, which combines state and intention expressions with embodied references. Through extensive experiments with various baseline models on SIGAR, we demonstrate that properly ordered multi-attribute references contribute to improved localization performance, revealing that single-attribute reference is insufficient for natural human-robot interaction scenarios. Our findings underscore the importance of multi-attribute reference expressions in advancing visual-language understanding.
Abstract (translated)
参考表达理解（REC）旨在根据自然语言描述实现对象定位。然而，现有的REC方法受到物体类别描述和单一属性意图描述的限制，这阻碍了它们在现实场景中的应用。在自然的人机交互中，用户通常通过个体状态和意图来表达他们的需求，并伴有引导手势，而不是详细的物体描述。为了解决这一挑战，我们提出了一种新的任务框架——Multi-ref EC，该框架将状态描述、推导出的意图以及身体动作整合起来以定位目标对象。我们引入了State-Intention-Gesture Attributes Reference（SIGAR）数据集，结合了状态和意图表达与身体引用。通过在SIGAR上进行广泛的基线模型实验，我们证明了适当排序的多属性参考有助于提高定位性能，并揭示了单一属性参考对于自然人机交互场景是不足的。我们的研究结果强调了多属性参考表达在推进视觉-语言理解中的重要性。
URL
https://arxiv.org/abs/2503.19240
