Abstract
Humans can flexibly switch between different modes of thinking based on task complexity: from rapid intuitive judgments to in-depth analytical understanding. However, current Graphical User Interface (GUI) grounding systems which locate interface elements based on natural language instructions rely solely on immediate prediction without reasoning, struggling to understand complex interface layouts with nested structures and hierarchical relationships, limiting their effectiveness on complex interfaces. Inspired by human dual-system cognition, we present Focus, a novel GUI grounding framework that combines fast prediction with systematic analysis. The framework dynamically switches between rapid and deliberate processing through an adaptive system switching based on task complexity, optimizing both efficiency and accuracy. Focus decomposes grounding into progressive stages: interface summarization, visual focused analysis, and precise coordinate prediction. This structured decomposition enables systematic understanding of both interface layouts and visual relationships. Extensive experiments show that Focus achieves state-of-the-art performance using only 300K of the training data with a 2B parameter model compared to existing approaches. Focus demonstrates superior performance particularly in complex GUI scenarios, achieving 77.4% average accuracy on ScreenSpot and 13.3% on the more challenging ScreenSpot-Pro. Our analysis reveals the effectiveness of this dual-system approach while demonstrating its potential for improving complex GUI interaction scenarios.
Abstract (translated)
人类可以根据任务复杂度灵活地在不同的思维方式之间切换：从快速直观判断到深入的分析理解。然而，目前基于自然语言指令定位界面元素的图形用户界面（GUI）接地系统仅依赖于即时预测而缺乏推理能力，难以理解和处理包含嵌套结构和层级关系的复杂界面布局，从而限制了其在复杂接口中的有效性。 受人类双系统认知启发，我们提出了Focus框架，这是一种结合快速预测与系统分析的新颖GUI接地方法。通过根据任务复杂度动态切换快慢两种处理方式，该框架能够在提高效率的同时优化准确性。Focus将接地过程分解为逐步阶段：界面概要、视觉集中分析和精确坐标预测。这种结构化分解使全面理解和解析界面布局及视觉关系成为可能。 广泛的实验表明，使用仅30万条训练数据的2B参数模型，Focus在性能上达到了最先进的水平，并且在复杂GUI场景中表现尤为突出，在ScreenSpot任务中的平均准确率为77.4%，而在更具挑战性的ScreenSpot-Pro任务中则为13.3%。我们的分析揭示了双系统方法的有效性，并展示了其改进复杂GUI交互场景的潜力。
URL
https://arxiv.org/abs/2503.06470
