Abstract
Developing autonomous agents for mobile devices can significantly enhance user interactions by offering increased efficiency and accessibility. However, despite the growing interest in mobile device control agents, the absence of a commonly adopted benchmark makes it challenging to quantify scientific progress in this area. In this work, we introduce B-MoCA: a novel benchmark designed specifically for evaluating mobile device control agents. To create a realistic benchmark, we develop B-MoCA based on the Android operating system and define 60 common daily tasks. Importantly, we incorporate a randomization feature that changes various aspects of mobile devices, including user interface layouts and language settings, to assess generalization performance. We benchmark diverse agents, including agents employing large language models (LLMs) or multi-modal LLMs as well as agents trained from scratch using human expert demonstrations. While these agents demonstrate proficiency in executing straightforward tasks, their poor performance on complex tasks highlights significant opportunities for future research to enhance their effectiveness. Our source code is publicly available at this https URL.
Abstract (translated)
开发移动设备上的自主代理可以显著增强用户交互，通过提供更高的效率和可访问性。然而，尽管移动设备控制代理越来越受到关注，但缺乏一个普遍适用的基准使得衡量这一领域科学进展具有挑战性。在这项工作中，我们介绍了B-MoCA：一个专门为评估移动设备控制代理而设计的新的基准。为了创建一个真实的基准，我们基于Android操作系统开发B-MoCA，并定义了60个常见的日常任务。重要的是，我们引入了随机化功能，随机改变移动设备的各个方面，包括用户界面布局和语言设置，以评估泛化性能。我们基准了各种代理，包括使用大型语言模型（LLMs）或多模态LLM训练的代理以及使用人类专家演示训练的代理。虽然这些代理在执行简单任务时表现出熟练，但他们在复杂任务上的表现却显露出未来研究可以改进其有效性的巨大潜力。我们的源代码可在此链接公开使用。
URL
https://arxiv.org/abs/2404.16660
