Abstract
The emergence of advanced multimodal large language models (MLLMs) has significantly enhanced AI assistants' ability to process complex information across modalities. Recently, egocentric videos, by directly capturing user focus, actions, and context in an unified coordinate, offer an exciting opportunity to enable proactive and personalized AI user experiences with MLLMs. However, existing benchmarks overlook the crucial role of gaze as an indicator of user intent. To address this gap, we introduce EgoGazeVQA, an egocentric gaze-guided video question answering benchmark that leverages gaze information to improve the understanding of longer daily-life videos. EgoGazeVQA consists of gaze-based QA pairs generated by MLLMs and refined by human annotators. Our experiments reveal that existing MLLMs struggle to accurately interpret user intentions. In contrast, our gaze-guided intent prompting methods significantly enhance performance by integrating spatial, temporal, and intent-related cues. We further conduct experiments on gaze-related fine-tuning and analyze how gaze estimation accuracy impacts prompting effectiveness. These results underscore the value of gaze for more personalized and effective AI assistants in egocentric settings.
Abstract (translated)
高级多模态大型语言模型（MLLM）的出现显著增强了AI助手处理跨模式复杂信息的能力。最近，以第一人称视角拍摄的视频直接捕捉用户的关注点、行动和统一坐标系中的上下文，为利用MLLM实现主动且个性化的用户体验提供了令人兴奋的机会。然而，现有的基准测试忽略了注视作为指示用户意图的重要指标。为了弥补这一不足，我们引入了EgoGazeVQA——一个基于第一人称视角注视引导的视频问答基准测试，该测试通过利用注视信息来改善对日常生活中较长视频的理解。EgoGazeVQA包含由MLLM生成并经人类标注员完善后的基于注视的问题和答案对。 我们的实验表明，现有的MLLM难以准确解读用户意图。相比之下，我们提出的基于注视引导的意图提示方法，通过整合空间、时间以及与意图相关的线索，显著提高了性能。此外，我们还针对注视相关微调进行了实验，并分析了注视估计精度如何影响提示效果的有效性。这些结果强调了在第一人称视角场景中，利用注视信息对于创建更加个性化且有效的AI助手的重要性。
URL
https://arxiv.org/abs/2509.07447
