Abstract
Multimodal AI agents are increasingly automating complex real-world workflows that involve online web execution. However, current web-agent benchmarks suffer from a critical limitation: they focus entirely on web-based interaction and perception, lacking grounding in the user's real-world physical surroundings. This limitation prevents evaluation in crucial scenarios, such as when an agent must use egocentric visual perception (e.g., via AR glasses) to recognize an object in the user's surroundings and then complete a related task online. To address this gap, we introduce Ego2Web, the first benchmark designed to bridge egocentric video perception and web agent execution. Ego2Web pairs real-world first-person video recordings with web tasks that require visual understanding, web task planning, and interaction in an online environment for successful completion. We utilize an automatic data-generation pipeline combined with human verification and refinement to curate well-constructed, high-quality video-task pairs across diverse web task types, including e-commerce, media retrieval, knowledge lookup, etc. To facilitate accurate and scalable evaluation for our benchmark, we also develop a novel LLM-as-a-Judge automatic evaluation method, Ego2WebJudge, which achieves approximately 84% agreement with human judgment, substantially higher than existing evaluation methods. Experiments with diverse SoTA agents on our Ego2Web show that their performance is weak, with substantial headroom across all task categories. We also conduct a comprehensive ablation study on task design, highlighting the necessity of accurate video understanding in the proposed task and the limitations of current agents. We hope Ego2Web can be a critical new resource for developing truly capable AI assistants that can seamlessly see, understand, and act across the physical and digital worlds.
Abstract (translated)
多模态AI智能体正日益自动化涉及在线网络执行的复杂现实工作流。然而，当前网络智能体基准存在一个关键局限：它们完全聚焦于基于网络的交互与感知，缺乏对用户真实物理环境的锚定。这一局限导致在关键场景中无法进行评估，例如当智能体必须通过以自我为中心的视觉感知（如通过AR眼镜）识别用户周围环境中的物体，然后在线完成相关任务。为弥补这一空白，我们推出了Ego2Web——首个旨在连接第一人称视频感知与网络智能体执行的基准测试。Ego2Web将真实世界的第一人称视频记录与需要视觉理解、网络任务规划及在线环境交互才能成功完成的任务进行配对。我们利用自动数据生成管道，结合人工验证与精炼， curation了涵盖电商、媒体检索、知识查询等多种网络任务类型的构造精良、高质量视频-任务对。为支持该基准的准确且可扩展评估，我们还开发了一种新颖的LLM-as-a-Judge自动评估方法Ego2WebJudge，其与人类判断的吻合度约84%，显著高于现有评估方法。在Ego2Web上使用多样化的SOTA智能体进行实验表明，其性能普遍较弱，所有任务类别均有 substantial 提升空间。我们还对任务设计进行了全面的消融研究，强调了所提任务中精确视频理解的必要性以及当前智能体的局限性。我们希望Ego2Web能成为开发真正强大的AI助手的关键新资源，这些助手能够无缝地在物理世界与数字世界之间观察、理解与行动。
URL
https://arxiv.org/abs/2603.22529
