Gaze_Estimation
-
TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues
2026-07-11 05:40:45Junhui She, Fei Wang, Kun Li, Yiqi Nie, Yuxin Liu, Zhangling Duan, Xun YangAbstract
Gaze target estimation aims to infer the position of a person's gaze within a scene. Within mainstream design logic, multi-branch methods require extra supervision and annotations, while streamlined designs prioritize low-level visual saliency over true gaze intent. The former leads to a high annotation burden and hinders domain transfer, whereas the latter causes misalignment between predicted attention and actual gaze targets. To address this issue, we propose TextGaze, a unified cross-modal architecture that leverages a Large Vision-Language Model (LVLM) as scalable semantic guidance to balance the two design paradigms. The model extracts visual features from a frozen encoder and utilizes an LVLM to obtain gaze-aligned textual cues. We design a transformer-based fusion module with hierarchical text supervision to preserve task semantics. Lightweight decoding heads enable the joint prediction of gaze heatmaps and in-/out-of-frame status. We evaluate our method on four mainstream datasets, and the results show competitive performance across key metrics with robust cross-dataset generalisation without extra fine-tuning. Overall, we provide a streamlined alternative to traditional designs and highlight the potential of LVLMs as accessible auxiliary guidance for gaze estimation.
Abstract (translated)
URL
https://arxiv.org/abs/2607.10130
PDF
-
PAGE: Towards Practical Human-level Gaze Target Estimation
2026-07-06 09:33:52Zhoutong Ye, Chengwen Zhang, Zhaibin Cui, Mingze Sun, Jiaqi Liu, Xiangwu Li, Qingyang Wan, Chang Liu, Xutong Wang, Huan-ang Gao, Yu Mei, Chun Yu, Yuanchun ShiAbstract
Gaze target estimation, the task of predicting where a person is looking in a scene, is crucial to understanding human attention and intent. It is a challenging task that combines high-level understanding of global scene semantics and precise spatial reasoning using human appearance (e.g. pose, eye orientation). As a result, human-level performance remains elusive for existing models, limiting their practical application. To this end, we propose PaGE (Practical Gaze Estimator), a gaze estimation model that explicitly models the complex interaction between scene and head features. Using a PaGE model with a large ViT-H+ backbone as the teacher, we further distill student models with lighter backbones on a much larger and more diverse unlabeled dataset. The architectural improvements and novel training recipe allow PaGE to achieve state-of-the-art performance on several gaze estimation tasks, outperforming humans in 7 out of 9 metrics while reducing the human-AI gap by at least 60% in the remaining 2. The distilled student models retain most of the teacher's performance while being lightweight enough for practical deployment on robots and consumer devices. The code and model checkpoints are available at our project page.
Abstract (translated)
URL
https://arxiv.org/abs/2607.04860
PDF
-
Binocular Gaze Estimation with Single Camera and Single Light Source
2026-07-06 09:13:21Tongbing Huang, Yang Fu, Yunfei Wang, Zhaocan WangAbstract
According to commonly consented theories, the minimum hardware requirement for gaze tracker is one camera and two light sources to realize gaze estimation with free head movements. However, in some scenarios such as eye tracking on mobile devices, it is preferable to use less components, especially light sources. We propose a gaze estimation method with one camera and one light source. A "virtual light source" is introduced, which is geometrically placed symmetrically to the real light source with respect to the camera, and generates a "virtual glint" in the acquired image. We estimate the "virtual glint" by exploiting the relationship between the distance between two pupils and two glints in the captured image, and estimate the gaze with polynomial regression assuming two light sources are available. A new normalization factor for regression method is verified, which turns out to be practical for one-glint system. The performance is proved to be acceptable, while degradation is noticed compared to system with two actual light sources.
Abstract (translated)
URL
https://arxiv.org/abs/2607.05473
PDF
-
How Much Future Helps? A Controlled Study of Future-Privileged Supervision for Causal Egocentric Gaze Estimation
2026-07-01 20:00:12Jia Li, Wenjie Zhao, Fnu Atisri, Sanskriti Aripineni, Shijian Deng, Jon E. Froehlich, Yuhang Zhao, Yapeng TianAbstract
Egocentric gaze estimation is commonly studied using models that process the full video with access to future frames, while real-world applications require strictly causal, online prediction. This discrepancy raises key questions: Does future context inherently provide valuable signals for gaze estimation? If so, how much future look-ahead optimally supervises a causal model during training? To investigate, we propose a controlled framework featuring a future-aware branch that accesses a tunable look-ahead horizon during training but is discarded at inference. This design isolates the impact of future context while keeping the inference architecture fixed and strictly causal. Across EGTEA Gaze+ and Ego4D, we find that future-privileged supervision consistently improves causal gaze prediction, confirming its utility. However, performance gains do not increase monotonically with longer look-ahead, but rather peak within a bounded temporal regime. Specifically, optimal performance corresponds to roughly 1.7--3.3 seconds of future context ($H{\in}[5, 10]$) on EGTEA Gaze+ and 2.7 seconds ($H{=}10$) on Ego4D. Our results demonstrate that lightweight causal models can effectively absorb future-aware signals, providing practical guidance for real-time egocentric gaze modeling.
Abstract (translated)
URL
https://arxiv.org/abs/2607.01437
PDF
-
Multi-scale Object-Aware Gaze Estimation via Geometric Reasoning
2026-06-28 11:02:21Jiajie Mi, Xinyu Liu, Mengke Song, Chenglizhao ChenAbstract
Gaze target estimation aims to predict the semantic object an observer fixates upon within an image, a task deeply rooted in the object-oriented nature of human gaze. Observers tend to select a specific semantic entity as the attentional target, rather than responding randomly across arbitrary regions of the image. However, existing methods typically model this task as a direct mapping from global features to gaze heatmaps, essentially treating it as a pixel-level regression problem. This approach fails to explicitly represent the gazed object as a distinct entity, making it difficult to produce stable and semantically consistent predictions in complex scenes. To address this, we propose a two-stage gaze estimation framework guided by object semantics, reformulating gaze target estimation as a hierarchical reasoning process. Our method incorporates object-level representations during feature encoding to align image features with discrete semantic entities, then introduces multi-scale feature fusion and geometric constraints from head pose and gaze direction for fine-grained localization and object-level discrimination. Extensive experiments on GazeFollow, VideoAttentionTarget, ChildPlay, and GOO-Real demonstrate that our method achieves AUC of 0.961, 0.948, 0.987, and 0.977 respectively, delivering strong performance across all benchmarks while maintaining a compact parameter size of 7.1M.
Abstract (translated)
URL
https://arxiv.org/abs/2606.29334
PDF
-
Semi-Supervised Gaze Estimation via Disentangled Subspace Contrastive Learning
2026-05-26 14:31:10Qida Tan, Hongyu Yang, Wenchao DuAbstract
Appearance-based gaze estimation always suffers from poor generalization due to limited annotated samples and insufficient dataset diversity. Leading approaches adopt weakly supervised learning to generate large-scale pseudo-labeled data from unconstrained real-world scenarios, aiming to mitigate the domain shifts. In this work, we devise a simple yet effective semi-supervised learning architecture that leverages unlabeled data to enhance domain generalization, thereby reducing reliance on labor-intensive manual annotations. Our key insight is to impose Jacobian regularization to disentangle feature representations into discriminative subspaces dedicated to specific gaze components, such as pitch and yaw angles. We further exploit the intrinsic ordinal ranking within each subspace for contrastive learning, enabling the model to learn robust gaze representations from a small set of labeled samples and an abundance of unlabeled ones. This ultimately yields our Disentangled Subspace Contrastive Learning (DSCL) framework. Extensive experiments on multiple benchmarks verify that the proposed DSCL is plug-and-play, achieving competitive performance using only 20\%, 10\%, and even 5\% of the annotated data under both in-domain and cross-domain evaluation settings. The public code is available at \href{this https URL}{this https URL}.
Abstract (translated)
URL
https://arxiv.org/abs/2605.27080
PDF
-
Low Latency Gaze Tracking via Latent Optical Sensing
2026-05-18 07:46:25Yidan Zheng, Matheus Souza, Kaizhang Kang, Qiang Fu, Hadi Amata, Wolfgang HeidrichAbstract
We present a real-time gaze tracking system that directly acquires task-relevant latent features using a fully passive optical encoder. Instead of forming and processing full-resolution images, our approach leverages a microlens array with a co-designed binary chromium mask to perform spatially multiplexed optical encoding, producing a compact set of measurements sufficient for gaze estimation. By integrating sensing and feature extraction in the optical domain, the proposed system eliminates the need for high-bandwidth image readout and substantially reduces computational overhead. The encoded measurements are captured by a 4 x 4 phototransistor array and mapped to gaze direction using a lightweight neural network. Our proof-of-concept prototype enables an end-to-end sensing-to-inference latency of 3.4 ms, outperforming published research systems. We demonstrate the effectiveness of our approach on both simulated and real-world data, achieving competitive gaze estimation accuracy while significantly improving latency and energy efficiency compared to conventional camera-based pipelines. This work highlights the potential of task-driven optical sensing for ultra-low-latency, computationally efficient human-computer interaction systems.
Abstract (translated)
URL
https://arxiv.org/abs/2605.17990
PDF
-
LISA: Language-guided Interference-aware Spatial-Frequency Attention for Driver Gaze Estimation
2026-05-17 06:57:31Jun Ma, Zhenye Yang, Ruichen Zhou, Pei Zhang, Huan Li, Jinpeng ChenAbstract
Driver gaze estimation serves as a fundamental metric for evaluating driver attentiveness in modern monitoring systems. Beyond being vulnerable to sudden lighting changes and sensor noise, spatial-domain models struggle to disentangle authentic gaze cues from irrelevant visual attributes. In this paper, we propose LISA, a \textbf{L}anguage-guided \textbf{I}nterference-aware \textbf{S}patial-Frequency \textbf{A}ttention framework that combines frequency-domain priors with vision-language knowledge. Observing that the amplitude spectrum remains relatively stable even under spatial perturbations, we design a dual-domain fusion mechanism. It integrates stable low-frequency semantics into high-frequency details, employing spatial attention to precisely target ocular regions. To reduce semantic ambiguity, we also introduce a training-time disentanglement strategy. Using a frozen CLIP encoder and orthogonal regularization, we explicitly separate gaze features from appearance interference. Experiments on two benchmarks show that LISA achieves state-of-the-art performance, with significantly improved robustness against occlusions and lighting variations. The code repository is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2605.17287
PDF
-
PicoEyes: Unified Gaze Estimation Framework for Mixed Reality with a Large-Scale Multi-View Dataset
2026-05-08 03:34:09Fuxin Duan, Hui WangAbstract
We present PicoEyes, a unified gaze estimation framework that directly predicts all key attributes of gaze, including 3D eye parameters, eye-region segmentation, optical axis, visual axis, and depth maps, from either monocular or binocular inputs. The framework simultaneously addresses calibration, gaze forecasting, and varying device postures, while also supporting 3D eye reconstruction via joint estimation of eye parameters and depth maps in an end-to-end manner. In addition, we introduce a large-scale multi-view near-eye dataset containing comprehensive 2D and 3D annotations under diverse conditions, including train, test, rewear-test, and calibration sessions. Extensive experiments demonstrate that PicoEyes achieves state-ofthe-art performance, consistently outperforming both academic and industrial gaze tracking methods across nocalibration, calibration, rewear-after-calibration, and forecasting settings. This work establishes a practical, end-toend paradigm for robust and generalizable gaze estimation in mixed reality (MR) applications.
Abstract (translated)
URL
https://arxiv.org/abs/2605.07188
PDF
-
Gaze4HRI: Zero-shot Benchmarking Gaze Estimation Neural-Networks for Human-Robot Interaction
2026-05-06 11:17:18Berk Sezer, Ali G\"orkem K\"u\c{c}\"uk, Erol \c{S}ahin, Sinan KalkanAbstract
While zero-shot appearance-based 3D gaze estimation offers significant cost-efficiency by directly mapping RGB images to gaze vectors, its reliability in Human-Robot Interaction (HRI) settings remains uncertain. Existing benchmarks frequently overlook fundamental HRI conditions, such as dynamic camera viewpoints and moving targets in video. Furthermore, current cross-dataset evaluations often suffer from a complexity gap, where methods trained on diverse datasets are tested on significantly smaller and less varied sets, failing to assess true robustness. To bridge these gaps, we introduce Gaze4HRI, a large-scale dataset (50+ subjects, 3,000+ videos, 600,000+ frames) designed to evaluate state-of-the-art performance against critical HRI variables: illumination, head-gaze conflict, as well as the motion of camera and gaze target in video. Our benchmark reveals that all evaluated methods fail in at least one condition, identifying steeply-downward gaze as a universal failure point. Notably, PureGaze trained on the ETH-X-Gaze dataset uniquely maintains resilience across all other conditions. These results challenge the recent focus in the literature on complex spatial-temporal modeling and Transformer-based architectures. Instead, our findings suggest that extensive data diversity, as exemplified by the ETH-X-Gaze dataset, serves as the primary driver of zero-shot robustness in unconstrained environments, while resilience-enhancing frameworks, such as PureGaze's self-adversarial loss for gaze feature purification, provide a substantial further improvement. Ultimately, this study establishes a rigorous benchmark that provides practical guidelines for practitioners as well as reshaping future research. The dataset and codes are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2605.04770
PDF
-
GMGaze: MoE-Based Context-Aware Gaze Estimation with CLIP and Multiscale Transformer
2026-05-01 17:35:17Xinyuan Zhao, Yihang Wu, Ahmad Chaddad, Sarah A. Alkhodair, Reem KatebAbstract
Gaze estimation methods commonly use facial appearances to predict the direction of a person gaze. However, previous studies show three major challenges with convolutional neural network (CNN)-based, transformer-based, and contrastive language-image pre-training (CLIP)-based methods, including late fusion of image features, lack of factor-aware conditioning, and impractical capacity scaling. To address these challenges, we propose Globally-conditioned Multi-scale Gaze estimation (GMGaze), which leverages a multi-scale transformer architecture. Specifically, the model first introduces semantic prototype conditioning, which modulates the CLIP global image embedding using four learned prototype banks (i.e., illumination, background, head pose and appearance) to generate two complementary context-biased global tokens. These tokens, along with the CLIP patch and CNN tokens, are fused at the first layer. This early unified fusion prevents information loss common in late-stage merging. Finally, each token passes through sparse Mixture-of-Experts modules, providing conditional computational capacity without uniformly increasing dense parameters. For cross-domain adaptation, we incorporate an adversarial domain adaptation technique with a feature separation loss that encourages the two global tokens to remain de-correlated. Experiments using four public benchmarks (MPIIFaceGaze, EYEDIAP, Gaze360, and ETH-XGaze) show that GMGaze achieves mean angular errors of 2.49$^\circ$, 3.22$^\circ$, 10.16$^\circ$, and 1.44$^\circ$, respectively, outperforming previous baselines in all within-domain settings. In cross-domain evaluations, it provides state-of-the-art (SOTA) results on two standard transfer routes.
Abstract (translated)
URL
https://arxiv.org/abs/2605.00799
PDF
-
SGAP-Gaze: Scene Grid Attention Based Point-of-Gaze Estimation Network for Driver Gaze
2026-04-21 18:10:03Pavan Kumar Sharma, Pranamesh ChakrabortyAbstract
Driver gaze estimation is essential for understanding the driver's situational awareness of surrounding traffic. Existing gaze estimation models use driver facial information to predict the Point-of-Gaze (PoG) or the 3D gaze direction vector. We propose a benchmark dataset, Urban Driving-Face Scene Gaze (UD-FSG), comprising synchronized driver-face and traffic-scene images. The scene images provide cues about surrounding traffic, which can help improve the gaze estimation model, along with the face images. We propose SGAP-Gaze, Scene-Grid Attention based Point-of-Gaze estimation network, trained and tested on our UD-FSG dataset, which explicitly incorporates the scene images into the gaze estimation modelling. The gaze estimation network integrates driver face, eye, iris, and scene contextual information. First, the extracted features from facial modalities are fused to form a gaze intent vector. Then, attention scores are computed over the spatial scene grid using a Transformer-based attention mechanism fusing face and scene image features to obtain the PoG. The proposed SGAP-Gaze model achieves a mean pixel error of 104.73 on the UD-FSG dataset and 63.48 on LBW dataset, achieving a 23.5% reduction in mean pixel error compared to state-of-the-art driver gaze estimation models. The spatial pixel distribution analysis shows that SGAP-Gaze consistently achieves lower mean pixel error than existing methods across all spatial ranges, including the outer regions of the scene, which are rare but critical for understanding driver attention. These results highlight the effectiveness of integrating multi-modal gaze cues with scene-aware attention for a robust driver PoG estimation model in real-world driving environments.
Abstract (translated)
URL
https://arxiv.org/abs/2604.19888
PDF
-
See Through the Noise: Improving Domain Generalization in Gaze Estimation
2026-04-17 09:23:35Yanming Peng, Shijing Wang, Yaping Huang, Yi TianAbstract
Generalizable gaze estimation methods have garnered increasing attention due to their critical importance in real-world applications and have achieved significant progress. However, they often overlook the effect of label noise, arising from the inherent difficulty of acquiring precise gaze annotations, on model generalization performance. In this paper, we are the first to comprehensively investigate the negative effects of label noise on generalization in gaze estimation. Further, we propose a novel solution, called See-Through-Noise (SeeTN) framework, which improves generalization from a novel perspective of mitigating label noise. Specifically, we propose to construct a semantic embedding space via a prototype-based transformation to preserve a consistent topological structure between gaze features and continuous labels. We then measure feature-label affinity consistency to distinguish noisy from clean samples, and introduce a novel affinity regularization in the semantic manifold to transfer gaze-related information from clean to noisy samples. Our proposed SeeTN promotes semantic structure alignment and enforces domain-invariant gaze relationships, thereby enhancing robustness against label noise. Extensive experiments demonstrate that our SeeTN effectively mitigates the adverse impact of source-domain noise, leading to superior cross-domain generalization without compromising the source-domain accuracy, and highlight the importance of explicitly handling noise in generalized gaze estimation.
Abstract (translated)
URL
https://arxiv.org/abs/2604.16562
PDF
-
Investigating Bias and Fairness in Appearance-based Gaze Estimation
Abstract
While appearance-based gaze estimation has achieved significant improvements in accuracy and domain adaptation, the fairness of these systems across different demographic groups remains largely unexplored. To date, there is no comprehensive benchmark quantifying algorithmic bias in gaze estimation. This paper presents the first extensive evaluation of fairness in appearance-based gaze estimation, focusing on ethnicity and gender attributes. We establish a fairness baseline by analyzing state-of-the-art models using standard fairness metrics, revealing significant performance disparities. Furthermore, we evaluate the effectiveness of existing bias mitigation strategies when applied to the gaze domain and show that their fairness contributions are limited. We summarize key insights and open issues. Overall, our work calls for research into developing robust, equitable gaze estimators. To support future research and reproducibility, we publicly release our annotations, code, and trained models at: this http URL
Abstract (translated)
URL
https://arxiv.org/abs/2604.10707
PDF
-
Rapidly deploying on-device eye tracking by distilling visual foundation models
2026-04-02 21:07:17Cheng Jiang, Jogendra Kundu, David Colmenares, Fengting Yang, Joseph Robinson, Yatong An, Ali BehroozAbstract
Eye tracking (ET) plays a critical role in augmented and virtual reality applications. However, rapidly deploying high-accuracy, on-device gaze estimation for new products remains challenging because hardware configurations (e.g., camera placement, camera pose, and illumination) often change across device generations. Visual foundation models (VFMs) are a promising direction for rapid training and deployment, and they excel on natural-image benchmarks; yet we find that off-the-shelf VFMs still struggle to achieve high accuracy on specialized near-eye infrared imagery. To address this gap, we introduce DistillGaze, a framework that distills a foundation model by leveraging labeled synthetic data and unlabeled real data for rapid and high-performance on-device gaze estimation. DistillGaze proceeds in two stages. First, we adapt a VFM into a domain-specialized teacher using self-supervised learning on labeled synthetic and unlabeled real images. Synthetic data provides scalable, high-quality gaze supervision, while unlabeled real data helps bridge the synthetic-to-real domain gap. Second, we train an on-device student using both teacher guidance and self-training. Evaluated on a large-scale, crowd-sourced dataset spanning over 2,000 participants, DistillGaze reduces median gaze error by 58.62% relative to synthetic-only baselines while maintaining a lightweight 256K-parameter model suitable for real-time on-device deployment. Overall, DistillGaze provides an efficient pathway for training and deploying ET models that adapt to hardware changes, and offers a recipe for combining synthetic supervision with unlabeled real data in on-device regression tasks.
Abstract (translated)
眼动追踪（ET）在增强现实与虚拟现实应用中发挥着关键作用。然而，由于硬件配置（如摄像头位置、姿态及光照条件）常随设备迭代而变化，快速为新产品部署高精度、端侧眼动估计仍是一项挑战。视觉基础模型（VFMs）为快速训练与部署提供了前景方向，其在自然图像基准测试中表现优异；但我们发现，现成VFMs在专业近眼红外图像上仍难以实现高精度。为弥合这一差距，我们提出了DistillGaze框架，该框架通过利用标注合成数据与无标注真实数据进行知识蒸馏，实现快速且高性能的端侧眼动估计。DistillGaze分两阶段进行：首先，利用标注合成图像与无标注真实图像进行自监督学习，将视觉基础模型适配为领域专用的教师模型——合成数据提供可扩展的高质量眼动监督，而无标注真实数据则有助于缩小合成与真实领域间的差异；其次，结合教师引导与自训练方法训练端侧学生模型。在涵盖2000多名参与者的大规模众包数据集上的评估表明，DistillGaze相较于仅使用合成数据的基线方法，将中位眼动误差降低了58.62%，同时保持了仅25.6万参数的轻量模型，适用于实时端侧部署。总体而言，DistillGaze为适应硬件变化的眼动追踪模型训练与部署提供了一条高效路径，并为端侧回归任务中结合合成监督与无标注真实数据提供了可行方案。
URL
https://arxiv.org/abs/2604.02509
PDF
-
Real-time Appearance-based Gaze Estimation for Open Domains
2026-03-27 19:33:18Zhenhao Li, Zheng Liu, Seunghyun Lee, Amin Fadaeinejad, Yuanhao YuAbstract
Appearance-based gaze estimation (AGE) has achieved remarkable performance in constrained settings, yet we reveal a significant generalization gap where existing AGE models often fail in practical, unconstrained scenarios, particularly those involving facial wearables and poor lighting conditions. We attribute this failure to two core factors: limited image diversity and inconsistent label fidelity across different datasets, especially along the pitch axis. To address these, we propose a robust AGE framework that enhances generalization without requiring additional human-annotated data. First, we expand the image manifold via an ensemble of augmentation techniques, including synthesis of eyeglasses, masks, and varied lighting. Second, to mitigate the impact of anisotropic inter-dataset label deviation, we reformulate gaze regression as a multi-task learning problem, incorporating multi-view supervised contrastive (SupCon) learning, discretized label classification, and eye-region segmentation as auxiliary objectives. To rigorously validate our approach, we curate new benchmark datasets designed to evaluate gaze robustness under challenging conditions, a dimension largely overlooked by existing evaluation protocols. Our MobileNet-based lightweight model achieves generalization performance competitive with the state-of-the-art (SOTA) UniGaze-H, while utilizing less than 1\% of its parameters, enabling high-fidelity, real-time gaze tracking on mobile devices.
Abstract (translated)
基于外观的注视估计（AGE）在受限场景中已取得显著性能，但我们发现其存在明显的泛化差距——现有AGE模型在实际非受限场景中常表现不佳，尤其涉及面部穿戴设备和光照条件恶劣的情况。我们将此归因于两个核心因素：不同数据集间图像多样性有限且标签保真度不一致，尤其在俯仰轴方向。为应对这些问题，我们提出一种无需额外人工标注数据的鲁棒AGE框架以提升泛化能力。首先，通过集成多种增强技术扩展图像流形，包括合成眼镜、口罩及多样化光照。其次，为缓解数据集间各向异性的标签偏差影响，我们将注视回归重构为多任务学习问题，引入多视角监督对比（SupCon）学习、离散化标签分类及眼周区域分割作为辅助目标。为严格验证方法有效性，我们构建了新的基准数据集，专门评估挑战性条件下的注视鲁棒性——这是现有评估协议普遍忽视的维度。基于MobileNet的轻量模型在泛化性能上可与当前最优的UniGaze-H竞争，而参数量不足其1%，实现了移动端高保真实时注视追踪。
URL
https://arxiv.org/abs/2603.26945
PDF
-
Is Geometry Enough? An Evaluation of Landmark-Based Gaze Estimation
2026-03-25 18:51:09Daniele Agostinelli, Thomas Agostinelli, Andrea Generosi, Maura MengoniAbstract
Appearance-based gaze estimation frequently relies on deep Convolutional Neural Networks (CNNs). These models are accurate, but computationally expensive and act as "black boxes", offering little interpretability. Geometric methods based on facial landmarks are a lightweight alternative, but their performance limits and generalization capabilities remain underexplored in modern benchmarks. In this study, we conduct a comprehensive evaluation of landmark-based gaze estimation. We introduce a standardized pipeline to extract and normalize landmarks from three large-scale datasets (Gaze360, ETH-XGaze, and GazeGene) and train lightweight regression models, specifically Extreme Gradient Boosted trees and two neural architectures: a holistic Multi-Layer Perceptron (MLP) and a siamese MLP designed to capture binocular geometry. We find that landmark-based models exhibit lower performance in within-domain evaluation, likely due to noise introduced into the datasets by the landmark detector. Nevertheless, in cross-domain evaluation, the proposed MLP architectures show generalization capabilities comparable to those of ResNet18 baselines. These findings suggest that sparse geometric features encode sufficient information for robust gaze estimation, paving the way for efficient, interpretable, and privacy-friendly edge applications. The source code and generated landmark-based datasets are available at this https URL.
Abstract (translated)
基于外观的注视估计通常依赖于深度卷积神经网络（CNN）。这些模型虽然准确，但计算成本高昂且如同“黑箱模型”，可解释性不足。基于面部标志点的几何方法是一种轻量级替代方案，但其性能极限和泛化能力在现代基准测试中尚未得到充分探索。本研究对基于标志点的注视估计进行了全面评估。我们引入标准化流程，从三个大规模数据集（Gaze360、ETH-XGaze和GazeGene）中提取并归一化标志点，并训练轻量级回归模型，具体包括极端梯度提升树以及两种神经架构：整体式多层感知机（MLP）和用于捕捉双目几何特征的孪生MLP。研究发现，在域内评估中，基于标志点的模型表现较低，这可能源于标志点检测器引入的数据噪声。然而，在跨域评估中，所提出的MLP架构展现出与ResNet18基线相当的泛化能力。这些发现表明，稀疏几何特征编码了足够的信息以实现稳健的注视估计，为高效、可解释且隐私友好型的边缘应用铺平了道路。源代码及生成的基于标志点的数据集可通过此https URL获取。
URL
https://arxiv.org/abs/2603.24724
PDF
-
Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation
2026-03-22 04:18:54Xuening Zhang, Qi Lv, Xiang Deng, Miao Zhang, Xingbo Liu, Liqiang NieAbstract
View transformers process multi-view observations to predict actions and have shown impressive performance in robotic manipulation. Existing methods typically extract static visual representations in a view-specific manner, leading to inadequate 3D spatial reasoning ability and a lack of dynamic adaptation. Taking inspiration from how the human brain integrates static and dynamic views to address these challenges, we propose Cortical Policy, a novel dual-stream view transformer for robotic manipulation that jointly reasons from static-view and dynamic-view streams. The static-view stream enhances spatial understanding by aligning features of geometrically consistent keypoints extracted from a pretrained 3D foundation model. The dynamic-view stream achieves adaptive adjustment through position-aware pretraining of an egocentric gaze estimation model, computationally replicating the human cortical dorsal pathway. Subsequently, the complementary view representations of both streams are integrated to determine the final actions, enabling the model to handle spatially-complex and dynamically-changing tasks under language conditions. Empirical evaluations on RLBench, the challenging COLOSSEUM benchmark, and real-world tasks demonstrate that Cortical Policy outperforms state-of-the-art baselines substantially, validating the superiority of dual-stream design for visuomotor control. Our cortex-inspired framework offers a fresh perspective for robotic manipulation and holds potential for broader application in vision-based robot control.
Abstract (translated)
多视角变换器通过处理多视角观测来预测动作，在机器人操作领域展现出卓越性能。现有方法通常以视角特定的方式提取静态视觉表征，导致三维空间推理能力不足且缺乏动态适应性。受人类大脑整合静态与动态视图解决此类挑战的启发，我们提出Cortical Policy（皮层策略）——一种用于机器人操作的新型双流多视角变换器，能够从静态视图流与动态视图流进行联合推理。静态视图流通过对齐从预训练三维基础模型中提取的几何一致关键点特征来增强空间理解；动态视图流则通过自我中心注视估计模型的位置感知预训练实现自适应调整，在计算层面复现了人类大脑皮层背侧通路的工作机制。随后，双流互补的视图表征被融合以确定最终动作，使模型能够在语言条件下处理空间复杂且动态变化的任务。在RLBench、具有挑战性的COLOSSEUM基准测试及真实世界任务上的实证评估表明，Cortical Policy显著优于现有最先进基线，验证了双流设计在视觉运动控制中的优越性。我们受大脑皮层启发的框架为机器人操作提供了全新视角，并有望在基于视觉的机器人控制领域实现更广泛应用。
URL
https://arxiv.org/abs/2603.21051
PDF
-
Lightweight Gaze Estimation Model Via Fusion Global Information
2026-03-21 03:58:42Zhang Cheng, Yanxia WangAbstract
Deep learning-based appearance gaze estimation methods are gaining popularity due to their high accuracy and fewer constraints from the environment. However, existing high-precision models often rely on deeper networks, leading to problems such as large parameters, long training time, and slow convergence. In terms of this issue, this paper proposes a novel lightweight gaze estimation model FGI-Net(Fusion Global Information). The model fuses global information into the CNN, effectively compensating for the need of multi-layer convolution and pooling to indirectly capture global information, while reducing the complexity of the model, improving the model accuracy and convergence speed. To validate the performance of the model, a large number of experiments are conducted, comparing accuracy with existing classical models and lightweight models, comparing convergence speed with models of different architectures, and conducting ablation experiments. Experimental results show that compared with GazeCaps, the latest gaze estimation model, FGI-Net achieves a smaller angle error with 87.1% and 79.1% reduction in parameters and FLOPs, respectively (MPIIFaceGaze is 3.74°, EyeDiap is 5.15°, Gaze360 is 10.50° and RT-Gene is 6.02°). Moreover, compared with different architectural models such as CNN and Transformer, FGI-Net is able to quickly converge to a higher accuracy range with fewer iterations of training, when achieving optimal accuracy on the Gaze360 and EyeDiap datasets, the FGI-Net model has 25% and 37.5% fewer iterations of training compared to GazeTR, respectively.
Abstract (translated)
基于深度学习的外观注视估计方法因其高精度和环境约束较少而日益流行。然而，现有高精度模型往往依赖更深的网络，导致参数量大、训练时间长及收敛慢等问题。针对此问题，本文提出一种新型轻量级注视估计模型FGI-Net（融合全局信息）。该模型将全局信息融入CNN中，有效弥补了通过多层卷积与池化间接捕获全局信息的需求，同时降低了模型复杂度，提升了模型精度与收敛速度。为验证模型性能，本文进行了大量实验：与现有经典模型及轻量级模型对比精度，与不同架构模型对比收敛速度，并开展消融实验。实验结果表明，相较于最新注视估计模型GazeCaps，FGI-Net在参数量与计算量上分别减少87.1%与79.1%，同时实现更小的角度误差（MPIIFaceGaze数据集为3.74°，EyeDiap为5.15°，Gaze360为10.50°，RT-Gene为6.02°）。此外，在与CNN、Transformer等不同架构模型的对比中，FGI-Net能以更少的训练迭代次数快速收敛至更高精度区间——在Gaze360与EyeDiap数据集达到最优精度时，其训练迭代次数较GazeTR分别减少25%与37.5%。
URL
https://arxiv.org/abs/2411.18064
PDF
-
The Nonverbal Gap: Toward Affective Computer Vision for Safer and More Equitable Online Dating
2026-03-20 03:31:07Ratna Kandala, Niva Manchanda, Akshata Kishore MoharirAbstract
Online dating has become the dominant way romantic relationships begin, yet current platforms strip the nonverbal cues: gaze, facial expression, body posture, response timing, that humans rely on to signal comfort, disinterest, and consent, creating a communication gap with disproportionate safety consequences for women. We argue that this gap represents both a technical opportunity and a moral responsibility for the computer vision community, which has developed the affective tools, facial action unit detection, gaze estimation, engagement modeling, and multimodal affect recognition, needed to begin addressing it, yet has largely ignored the dating domain as a research context. We propose a fairness-first research agenda organized around four capability areas: real-time discomfort detection, engagement asymmetry modeling between partners, consent-aware interaction design, and longitudinal interaction summarization, each grounded in established CV methodology and motivated by the social psychology of romantic communication. We argue that responsible pursuit of this agenda requires purpose-built datasets collected under dyadic consent protocols, fairness evaluation disaggregated across race, gender identity, neurotype, and cultural background, and architectural commitments to on-device processing that prevent affective data from becoming platform surveillance infrastructure. This vision paper calls on the WICV community, whose members are uniquely positioned to understand both the technical opportunity and the human stakes, to establish online dating safety as a first-class research domain before commercial deployment outpaces ethical deliberation.
Abstract (translated)
在线约会已成为 romantic relationships 开始的主要方式，但现有平台剥离了人类赖以 signaling comfort, disinterest, and consent 的非语言线索：目光、面部表情、身体姿态、回应时机，从而造成了沟通鸿沟，并对女性产生不成比例的安全后果。我们认为，这一鸿沟既代表了计算机视觉社区的技术机遇，也构成了其道德责任——该社区已开发出解决此问题所需的情感计算工具：面部动作单元检测、视线估计、参与度建模与多模态情感识别，却 largely ignored 将约会领域作为研究背景。我们提出一个以公平优先的研究议程，围绕四个能力领域组织：实时不适检测、伴侣间参与度不对称建模、注重同意的交互设计、以及纵向交互摘要，每个领域都基于成熟的 CV 方法论，并受浪漫沟通的社会心理学驱动。我们主张，负责任地推进此议程需要：在双主体同意协议下收集的专用数据集；按种族、性别认同、神经类型和文化背景 disaggregated 的公平性评估；以及坚持设备端处理的架构设计，以防止情感数据沦为平台监控基础设施。这篇愿景论文呼吁 WICV 社区——其成员既理解技术机遇又深知人文关切的独特位置——在商业部署超越伦理审议前，将在线约会安全确立为一级研究领域。
URL
https://arxiv.org/abs/2603.26727
PDF
