Abstract
3D and 2D gaze estimation share the fundamental objective of capturing eye movements but are traditionally treated as two distinct research domains. In this paper, we introduce a novel cross-task few-shot 2D gaze estimation approach, aiming to adapt a pre-trained 3D gaze estimation network for 2D gaze prediction on unseen devices using only a few training images. This task is highly challenging due to the domain gap between 3D and 2D gaze, unknown screen poses, and limited training data. To address these challenges, we propose a novel framework that bridges the gap between 3D and 2D gaze. Our framework contains a physics-based differentiable projection module with learnable parameters to model screen poses and project 3D gaze into 2D gaze. The framework is fully differentiable and can integrate into existing 3D gaze networks without modifying their original architecture. Additionally, we introduce a dynamic pseudo-labelling strategy for flipped images, which is particularly challenging for 2D labels due to unknown screen poses. To overcome this, we reverse the projection process by converting 2D labels to 3D space, where flipping is performed. Notably, this 3D space is not aligned with the camera coordinate system, so we learn a dynamic transformation matrix to compensate for this misalignment. We evaluate our method on MPIIGaze, EVE, and GazeCapture datasets, collected respectively on laptops, desktop computers, and mobile devices. The superior performance highlights the effectiveness of our approach, and demonstrates its strong potential for real-world applications.
Abstract (translated)
3D和2D注视估计虽然有着共同的基本目标，即捕捉眼球运动，但传统上被视为两个独立的研究领域。在本文中，我们提出了一种新颖的跨任务少样本2D注视估计方法，旨在利用少量训练图像，将预训练好的3D注视估算网络适应到未见过设备上的2D注视预测。这项任务由于3D和2D注视之间的域差距、未知屏幕姿态以及有限的训练数据而极具挑战性。 为了解决这些难题，我们提出了一种新颖的框架来弥合3D和2D注视间的鸿沟。该框架包含一个基于物理原理的可学习参数化差分投影模块，用于建模屏幕姿态并将3D注视转换成2D注视。整个框架是全差异化的，并且能够不改变原有架构的情况下整合到现有的3D注视网络中。 此外，我们还引入了一种针对翻转图像的动态伪标签策略，这对未知屏幕姿态下的2D标签来说尤其具有挑战性。为克服这一困难，我们将转换过程逆向进行，即将2D标签转化为3D空间，在那里执行翻转操作。值得注意的是，这个3D空间并未与相机坐标系统对齐，因此我们学习了一个动态变换矩阵来补偿这种错位。 我们在MPIIGaze、EVE和GazeCapture数据集上评估了我们的方法，这些数据分别采集自笔记本电脑、台式计算机和移动设备上。我们的方法表现出色，并突显了其在现实世界应用中的强大潜力。
URL
https://arxiv.org/abs/2502.04074
