Abstract
We introduce Speech-to-Spatial, a referent disambiguation framework that converts verbal remote-assistance instructions into spatially grounded AR guidance. Unlike prior systems that rely on additional cues (e.g., gesture, gaze) or manual expert annotations, Speech-to-Spatial infers the intended target solely from spoken references (speech input). Motivated by our formative study of speech referencing patterns, we characterize recurring ways people specify targets (Direct Attribute, Relational, Remembrance, and Chained) and ground them to our object-centric relational graph. Given an utterance, referent cues are parsed and rendered as persistent in-situ AR visual guidance, reducing iterative micro-guidance ("a bit more to the right", "now, stop.") during remote guidance. We demonstrate the use cases of our system with remote guided assistance and intent disambiguation scenarios. Our evaluation shows that Speechto-Spatial improves task efficiency, reduces cognitive load, and enhances usability compared to a conventional voice-only baseline, transforming disembodied verbal instruction into visually explainable, actionable guidance on a live shared view.
Abstract (translated)
我们介绍了一种名为Speech-to-Spatial的指代消歧框架，该框架将口头远程协助指令转换为基于空间定位的增强现实（AR）指导。与以往依赖额外线索（如手势、目光）或手动专家注释的系统不同，Speech-to-Spatial仅根据口语化的参考信息推断目标对象。受我们对言语引用模式初步研究的启发，我们将人们指定目标的方式进行了分类（直接属性、关系性、记忆式和链式），并将其与以物体为中心的关系图关联起来。对于每条语音输入，指代线索被解析并渲染为持久性的现场AR视觉指导，从而减少了远程指导过程中反复微调指引的需求（如“再往右一点”，“停。”）。 我们通过远程协助和意图消歧场景演示了系统的应用案例。我们的评估表明，与传统的仅依赖声音的基准系统相比，Speech-to-Spatial在提高任务效率、减轻认知负荷以及增强用户体验方面表现出明显优势，将抽象的语言指令转化为在实时共享视图中可视且可操作的指导。
URL
https://arxiv.org/abs/2602.03059
