Abstract
Computer use agents (CUAs) need to plan task workflows grounded in diverse, ever-changing applications and environments, but learning is hindered by the scarcity of large-scale, high-quality training data in the target application. Existing datasets are domain-specific, static, and costly to annotate, while current synthetic data generation methods often yield simplistic or misaligned task demonstrations. To address these limitations, we introduce Watch & Learn (W&L), a framework that converts human demonstration videos readily available on the Internet into executable UI trajectories at scale. Instead of directly generating trajectories or relying on ad hoc reasoning heuristics, we cast the problem as an inverse dynamics objective: predicting the user's action from consecutive screen states. This formulation reduces manual engineering, is easier to learn, and generalizes more robustly across applications. Concretely, we develop an inverse dynamics labeling pipeline with task-aware video retrieval, generate over 53k high-quality trajectories from raw web videos, and demonstrate that these trajectories improve CUAs both as in-context demonstrations and as supervised training data. On the challenging OSWorld benchmark, UI trajectories extracted with W&L consistently enhance both general-purpose and state-of-the-art frameworks in-context, and deliver stronger gains for open-source models under supervised training. These results highlight web-scale human demonstration videos as a practical and scalable foundation for advancing CUAs towards real-world deployment.
Abstract (translated)
计算机使用代理（CUA）需要在多样化且不断变化的应用程序和环境中规划任务工作流，但学习过程受限于目标应用程序中大规模高质量训练数据的稀缺。现有的数据集通常是特定领域的、静态的，并且标注成本高昂；而当前的人工合成数据生成方法通常会产生过于简单或不相关的任务演示。为了解决这些问题，我们引入了“Watch & Learn”（W&L）框架，该框架可以将互联网上可用的人类示范视频大规模地转换为可执行的UI轨迹。与直接生成轨迹或将推理启发式技术应用于特定场景不同，我们将问题定义为逆向动力学目标：从连续的屏幕状态中预测用户的动作。这种形式化方法减少了手动工程设计的需求，并且更容易学习，同时在跨应用程序方面也具有更好的泛化能力。 具体而言，我们开发了一个基于任务意识的视频检索的逆向动力学标注流水线，从原始网络视频生成了超过53,000条高质量轨迹，并展示了这些轨迹如何作为上下文演示和监督训练数据来改进CUA。在具有挑战性的OSWorld基准测试中，使用W&L提取的UI轨迹在上下文中一致地提高了通用框架和最新技术水平的表现，而且在监督训练下对开源模型提供了更强的优势。 这一结果表明，网络规模的人类示范视频可以作为推进CUA向现实世界部署的实际且可扩展的基础。
URL
https://arxiv.org/abs/2510.04673
