Abstract
Accurate 3D gaze estimation in unconstrained real-world environments remains a significant challenge due to variations in appearance, head pose, occlusion, and the limited availability of in-the-wild 3D gaze datasets. To address these challenges, we introduce a novel Self-Training Weakly-Supervised Gaze Estimation framework (ST-WSGE). This two-stage learning framework leverages diverse 2D gaze datasets, such as gaze-following data, which offer rich variations in appearances, natural scenes, and gaze distributions, and proposes an approach to generate 3D pseudo-labels and enhance model generalization. Furthermore, traditional modality-specific models, designed separately for images or videos, limit the effective use of available training data. To overcome this, we propose the Gaze Transformer (GaT), a modality-agnostic architecture capable of simultaneously learning static and dynamic gaze information from both image and video datasets. By combining 3D video datasets with 2D gaze target labels from gaze following tasks, our approach achieves the following key contributions: (i) Significant state-of-the-art improvements in within-domain and cross-domain generalization on unconstrained benchmarks like Gaze360 and GFIE, with notable cross-modal gains in video gaze estimation; (ii) Superior cross-domain performance on datasets such as MPIIFaceGaze and Gaze360 compared to frontal face methods. Code and pre-trained models will be released to the community.
Abstract (translated)
在不受限制的真实世界环境中进行精确的三维注视估计仍然是一个重大挑战，这主要是由于外观变化、头部姿态的变化、遮挡以及缺乏大规模的实际三维注视数据集所致。为了解决这些问题，我们提出了一种新的自我训练弱监督注视估计框架（Self-Training Weakly-Supervised Gaze Estimation, ST-WSGE）。这一双阶段学习框架利用了各种二维注视数据集（如跟随注视的数据），这些数据集提供了丰富的外观变化、自然场景和注视分布的多样性，并提出了生成三维伪标签的方法以增强模型泛化能力。此外，传统的特定模式设计的模型（分别针对图像或视频进行设计）限制了现有训练数据的有效利用。为了解决这一问题，我们提出了一种模态无关架构——注视变换器（Gaze Transformer, GaT），该架构能够同时从图像和视频数据集中学习静态和动态的注视信息。通过结合3D视频数据集与来自跟随注视任务的2D注视目标标签，我们的方法实现了以下关键贡献： (i) 在不受限制基准测试（如Gaze360和GFIE）中的域内和跨域泛化方面取得了显著的最新技术水平改进，并在视频注视估计中获得了显着的跨模态收益； (ii) 与面向正面脸部的方法相比，在MPIIFaceGaze和Gaze360等数据集上展现了优越的跨域性能。 代码和预训练模型将向社区开放。
URL
https://arxiv.org/abs/2502.20249
