Abstract
Large vision-language models have endowed GUI agents with strong general capabilities for interface understanding and interaction. However, due to insufficient exposure to domain-specific software operation data during training, these agents exhibit significant domain bias - they lack familiarity with the specific operation workflows (planning) and UI element layouts (grounding) of particular applications, limiting their real-world task performance. In this paper, we present GUIDE (GUI Unbiasing via Instructional-Video Driven Expertise), a training-free, plug-and-play framework that resolves GUI agent domain bias by autonomously acquiring domain-specific expertise from web tutorial videos through a retrieval-augmented automated annotation pipeline. GUIDE introduces two key innovations. First, a subtitle-driven Video-RAG pipeline unlocks video semantics through subtitle analysis, performing progressive three-stage retrieval - domain classification, topic extraction, and relevance matching - to identify task-relevant tutorial videos. Second, a fully automated annotation pipeline built on an inverse dynamics paradigm feeds consecutive keyframes enhanced with UI element detection into VLMs, inferring the required planning and grounding knowledge that are injected into the agent's corresponding modules to address both manifestations of domain bias. Extensive experiments on OSWorld demonstrate GUIDE's generality as a plug-and-play component for both multi-agent systems and single-model agents. It consistently yields over 5% improvements and reduces execution steps - without modifying any model parameters or architecture - validating GUIDE as an architecture-agnostic enhancement to bridge GUI agent domain bias.
Abstract (translated)
大型视觉语言模型赋予了GUI智能体强大的界面理解与交互通用能力。然而，由于训练过程中缺乏领域特定软件操作数据的充分暴露，这些智能体表现出显著的领域偏见——它们不熟悉特定应用的具体操作流程（规划）和UI元素布局（定位），从而限制了其在真实任务中的性能。本文提出GUIDE（基于教学视频驱动专业知识的GUI去偏），一个无需训练、即插即用的框架，通过检索增强的自动标注流水线从网络教学视频中自主获取领域专业知识，从而解决GUI智能体的领域偏见问题。GUIDE引入了两项关键创新：首先，基于字幕的视频检索增强生成（Video-RAG）流水线通过字幕分析解锁视频语义，执行渐进式三阶段检索——领域分类、主题提取和相关性匹配——以识别任务相关的教学视频；其次，基于逆动力学范式的全自动标注流水线将增强UI元素检测的连续关键帧输入视觉语言模型，推断所需的规划和定位知识，并将其注入智能体的相应模块，以解决领域偏见的两种表现形式。在OSWorld上的大量实验表明，GUIDE作为即插即用组件对多智能体系统和单模型智能体均具有通用性。它在不修改任何模型参数或架构的情况下， consistently带来超过5%的性能提升并减少执行步骤，验证了GUIDE作为架构无关增强方案在弥合GUI智能体领域偏见方面的有效性。
URL
https://arxiv.org/abs/2603.26266
