Abstract
Video-based gaze estimation methods aim to capture the inherently temporal dynamics of human eye gaze from multiple image frames. However, since models must capture both spatial and temporal relationships, performance is limited by the feature representations within a frame but also between multiple frames. We propose the Spatio-Temporal Gaze Network (ST-Gaze), a model that combines a CNN backbone with dedicated channel attention and self-attention modules to fuse eye and face features optimally. The fused features are then treated as a spatial sequence, allowing for the capture of an intra-frame context, which is then propagated through time to model inter-frame dynamics. We evaluated our method on the EVE dataset and show that ST-Gaze achieves state-of-the-art performance both with and without person-specific adaptation. Additionally, our ablation study provides further insights into the model performance, showing that preserving and modelling intra-frame spatial context with our spatio-temporal recurrence is fundamentally superior to premature spatial pooling. As such, our results pave the way towards more robust video-based gaze estimation using commonly available cameras.
Abstract (translated)
基于视频的眼动估计方法旨在从多帧图像中捕捉人类眼动的内在时间动态。然而，由于模型必须同时捕获空间和时间关系，其性能不仅受限于单帧内的特征表示，还受到多个帧之间特征表示的影响。为此，我们提出了一个结合了卷积神经网络（CNN）骨干网、专用通道注意力机制和自注意模块的时空眼动网络（ST-Gaze），该模型能够将眼睛和面部特征最优地融合在一起。 这些融合后的特征被视为空间序列处理，使得可以捕捉到单帧内的上下文信息。随后，这种上下文信息通过时间传播来建模多帧之间的动态变化。我们在EVE数据集上评估了我们的方法，并展示了ST-Gaze在有无特定个体适应的情况下均达到了最先进的性能。 此外，我们进行的消融研究进一步揭示了模型性能的关键因素，表明利用我们提出的时空递归结构来保留和建模单帧内的空间上下文与过早的空间池化相比具有根本性的优势。因此，我们的研究成果为使用常见摄像头实现更稳健的视频眼动估计开辟了道路。
URL
https://arxiv.org/abs/2512.17673
