Abstract
We present HOIGaze - a novel learning-based approach for gaze estimation during hand-object interactions (HOI) in extended reality (XR). HOIGaze addresses the challenging HOI setting by building on one key insight: The eye, hand, and head movements are closely coordinated during HOIs and this coordination can be exploited to identify samples that are most useful for gaze estimator training - as such, effectively denoising the training data. This denoising approach is in stark contrast to previous gaze estimation methods that treated all training samples as equal. Specifically, we propose: 1) a novel hierarchical framework that first recognises the hand currently visually attended to and then estimates gaze direction based on the attended hand; 2) a new gaze estimator that uses cross-modal Transformers to fuse head and hand-object features extracted using a convolutional neural network and a spatio-temporal graph convolutional network; and 3) a novel eye-head coordination loss that upgrades training samples belonging to the coordinated eye-head movements. We evaluate HOIGaze on the HOT3D and Aria digital twin (ADT) datasets and show that it significantly outperforms state-of-the-art methods, achieving an average improvement of 15.6% on HOT3D and 6.0% on ADT in mean angular error. To demonstrate the potential of our method, we further report significant performance improvements for the sample downstream task of eye-based activity recognition on ADT. Taken together, our results underline the significant information content available in eye-hand-head coordination and, as such, open up an exciting new direction for learning-based gaze estimation.
Abstract (translated)
我们提出了HOIGaze——一种针对扩展现实（XR）中手部与物体交互（HOI）过程中注视点估计的新型基于学习的方法。HOIGaze通过一个关键洞察力解决了这一具有挑战性的HOI场景：在HOIs期间，眼睛、手和头部的动作紧密协调，并且可以利用这种协调来识别对训练注视点估计算法最有用的数据样本——从而有效去噪训练数据。这种方法与以往将所有训练样本同等对待的注视估计方法形成了鲜明对比。 具体而言，我们提出了： 1）一种新颖的分层框架，该框架首先识别当前视觉注意的手部，然后基于所注意的手来估算注视方向； 2）一个新的注视点估计算法，利用跨模态Transformer融合由卷积神经网络提取的头部特征和空间-时间图卷积网络提取的手部与物体特征； 3）一种新颖的眼头协调损失函数，用于提升属于眼头协同运动的训练样本的质量。 我们在HOT3D及Aria数字孪生（ADT）数据集上评估了HOIGaze，并显示它显著优于现有最先进的方法，在HOT3D和ADT上的平均角度误差分别减少了15.6% 和6.0%。为了展示我们方法的潜力，我们也报告了在ADT上的基于眼睛活动识别下游任务中的显著性能改进。 综上所述，我们的结果强调了眼手头协调中包含的重要信息量，并因此为基于学习的注视点估计开辟了一个令人兴奋的新方向。
URL
https://arxiv.org/abs/2504.19828
