Abstract
Enabling robots to understand human gaze target is a crucial step to allow capabilities in downstream tasks, for example, attention estimation and movement anticipation in real-world human-robot interactions. Prior works have addressed the in-frame target localization problem with data-driven approaches by carefully removing out-of-frame samples. Vision-based gaze estimation methods, such as OpenFace, do not effectively absorb background information in images and cannot predict gaze target in situations where subjects look away from the camera. In this work, we propose a system to address the problem of 360-degree gaze target estimation from an image in generalized visual scenes. The system, named GazeTarget360, integrates conditional inference engines of an eye-contact detector, a pre-trained vision encoder, and a multi-scale-fusion decoder. Cross validation results show that GazeTarget360 can produce accurate and reliable gaze target predictions in unseen scenarios. This makes a first-of-its-kind system to predict gaze targets from realistic camera footage which is highly efficient and deployable. Our source code is made publicly available at: this https URL.
Abstract (translated)
让机器人理解人类的目光注视目标是实现在下游任务中具备能力的关键一步，例如在现实世界的人机交互中的注意力估计和动作预测。先前的研究通过仔细移除超出画面范围的数据样本，采用数据驱动的方法解决了画面内目标定位的问题。基于视觉的视线估计方法（如OpenFace）无法有效地吸收图像背景信息，并且不能预测当主体目光偏离相机时的目标注视点。在这项工作中，我们提出了一种系统来解决从一般化视觉场景中的图像进行360度视野内的注视目标估算问题。该系统名为GazeTarget360，集成了眼部接触检测器的条件推理引擎、预训练的视觉编码器以及多尺度融合解码器。 交叉验证结果显示，GazeTarget360可以在未见过的情景中生成准确且可靠的视线目标预测结果。这使得它成为首个能够从现实摄像机拍摄的画面中预测注视点的目标系统，并且具有高效率和可部署性。我们的源代码可在以下链接公开获取：[这里插入链接] (请将此链接替换为实际提供的URL)。
URL
https://arxiv.org/abs/2507.00253
