Abstract
The complex application scenarios have raised critical requirements for precise and generalizable gaze estimation methods. Recently, the pre-trained CLIP has achieved remarkable performance on various vision tasks, but its potentials have not been fully exploited in gaze estimation. In this paper, we propose a novel CLIP-driven Dual Feature Enhancing Network (CLIP-DFENet), which boosts gaze estimation performance with the help of CLIP under a novel `main-side' collaborative enhancing strategy. Accordingly, a Language-driven Differential Module (LDM) is designed on the basis of the CLIP's text encoder to reveal the semantic difference of gaze. This module could empower our Core Feature Extractor with the capability of characterizing the gaze-related semantic information. Moreover, a Vision-driven Fusion Module (VFM) is introduced to strengthen the generalized and valuable components of visual embeddings obtained via CLIP's image encoder, and utilizes them to further improve the generalization of the features captured by Core Feature Extractor. Finally, a robust Double-head Gaze Regressor is adopted to map the enhanced features to gaze directions. Extensive experimental results on four challenging datasets over within-domain and cross-domain tasks demonstrate the discriminability and generalizability of our CLIP-DFENet.
Abstract (translated)
复杂的应用场景提出了对精确且通用的注视估计方法的关键需求。最近，预训练的CLIP在各种视觉任务上取得了显著性能，但其在注视估计中的潜力尚未被充分挖掘。本文中，我们提出了一种新颖的基于CLIP的双特征增强网络（CLIP-DFENet），该网络利用CLIP并通过一种创新的“主侧”协作增强策略来提升注视估计的表现。为此，设计了一个以CLIP的文字编码器为基础的语言驱动差异模块（LDM）来揭示注视的语义差异。此模块能够赋予我们的核心特征提取器识别与注视相关语义信息的能力。此外，引入了一种视觉驱动融合模块（VFM），通过增强从CLIP图像编码器获得的视觉嵌入中的通用且有价值的组件，并利用这些组件进一步提升由核心特征提取器捕获特征的一般化能力。最后，采用了一个稳健的双头注视回归器来将增强后的特征映射到注视方向上。在四个具有挑战性的数据集上的广泛实验结果显示了我们提出的CLIP-DFENet模型在域内和跨域任务中的区分能力和泛化能力。
URL
https://arxiv.org/abs/2502.20128
