Abstract
We propose a novel 3D gaze estimation approach that learns spatial relationships between the subject and objects in the scene, and outputs 3D gaze direction. Our method targets unconstrained settings, including cases where close-up views of the subject's eyes are unavailable, such as when the subject is distant or facing away. Previous approaches typically rely on either 2D appearance alone or incorporate limited spatial cues using depth maps in the non-learnable post-processing step. Estimating 3D gaze direction from 2D observations in these scenarios is challenging; variations in subject pose, scene layout, and gaze direction, combined with differing camera poses, yield diverse 2D appearances and 3D gaze directions even when targeting the same 3D scene. To address this issue, we propose GA3CE: Gaze-Aware 3D Context Encoding. Our method represents subject and scene using 3D poses and object positions, treating them as 3D context to learn spatial relationships in 3D space. Inspired by human vision, we align this context in an egocentric space, significantly reducing spatial complexity. Furthermore, we propose D$^3$ (direction-distance-decomposed) positional encoding to better capture the spatial relationship between 3D context and gaze direction in direction and distance space. Experiments demonstrate substantial improvements, reducing mean angle error by 13%-37% compared to leading baselines on benchmark datasets in single-frame settings.
Abstract (translated)
我们提出了一种新颖的3D注视估计方法，该方法通过学习场景中主体与物体之间的空间关系，并输出3D注视方向。我们的方法针对非约束环境，包括当拍摄对象距离较远或背对镜头时无法获得其眼睛特写的情况。以往的方法通常依赖于2D外观特征或者在不可学习的后处理步骤中使用深度图来融合有限的空间线索。在这种情况下，从2D观察结果推断3D注视方向具有挑战性；由于主体姿态、场景布局和注视方向的变化以及不同摄像机位置的影响，即便面对相同的3D场景，也会产生多样化的2D外观特征和3D注视方向。 为解决这一问题，我们提出了GA3CE（Gaze-Aware 3D Context Encoding）。该方法利用3D姿势及物体位置来表示主体与场景，并将其视为用于学习三维空间中空间关系的3D上下文。受人类视觉系统的启发，我们在以自我为中心的空间内对齐此上下文，大大减少了空间复杂度。此外，我们提出了D$^3$（方向-距离分解）的位置编码方法，以便更准确地捕捉3D上下文和注视方向在方向与距离上的空间关系。 实验结果显示，在单帧设置下，我们的方法相对于基准数据集上领先的基线方法显著降低了平均角度误差，减少了13%-37%。
URL
https://arxiv.org/abs/2505.10671
