Abstract
Current multimodal large language models (MLLMs) cannot effectively utilize eye-gaze information for video understanding, even when gaze cues are supplied via visual overlays or text descriptions. We introduce GazeQwen, a parameter efficient approach that equips an open-source MLLM with gaze awareness through hidden-state modulation. At its core is a compact gaze resampler (~1-5 M trainable parameters) that encodes V-JEPA 2.1 video features together with fixation-derived positional encodings and produces additive residuals injected into selected LLM decoder layers via forward hooks. An optional second training stage adds low-rank adapters (LoRA) to the LLM for tighter integration. Evaluated on all 10 tasks of the StreamGaze benchmark, GazeQwen reaches 63.9% accuracy, a +16.1 point gain over the same Qwen2.5-VL-7B backbone with gaze as visual prompts and +10.5 points over GPT-4o, the highest score among all open-source and proprietary models tested. These results suggest that learning where to inject gaze within an LLM is more effective than scaling model size or engineering better prompts. All code and checkpoints are available at this https URL .
Abstract (translated)
当前多模态大语言模型（MLLMs）无法有效利用眼动信息进行视频理解，即使通过视觉叠加或文本描述提供眼动线索。我们提出GazeQwen，一种参数高效的方法，通过隐状态调制为开源MLLM赋予眼动感知能力。其核心是一个紧凑的眼动重采样器（约1-5M可训练参数），该模块编码V-JEPA 2.1视频特征，结合注视点衍生的位置编码，并通过前向钩子将生成的可加残差注入选定的LLM解码器层。可选的第二阶段训练会为LLM添加低秩适配器（LoRA）以实现更紧密的集成。在StreamGaze基准的全部10项任务评估中，GazeQwen达到63.9%的准确率，相比使用眼动作为视觉提示的相同Qwen2.5-VL-7B主干网络提升16.1个百分点，并超越GPT-4o（所有测试的开源及闭源模型中最高分）10.5个百分点。这些结果表明，学习在LLM中注入眼动信息的位置比扩大模型规模或设计更优提示更有效。所有代码与检查点可通过此https URL获取。
URL
https://arxiv.org/abs/2603.25841
