Abstract
Humans can learn to operate the user interface (UI) of an application by reading an instruction manual or how-to guide. Along with text, these resources include visual content such as UI screenshots and images of application icons referenced in the text. We explore how to leverage this data to learn generic visio-linguistic representations of UI screens and their components. These representations are useful in many real applications, such as accessibility, voice navigation, and task automation. Prior UI representation models rely on UI metadata (UI trees and accessibility labels), which is often missing, incompletely defined, or not accessible. We avoid such a dependency, and propose Lexi, a pre-trained vision and language model designed to handle the unique features of UI screens, including their text richness and context sensitivity. To train Lexi we curate the UICaption dataset consisting of 114k UI images paired with descriptions of their functionality. We evaluate Lexi on four tasks: UI action entailment, instruction-based UI image retrieval, grounding referring expressions, and UI entity recognition.
Abstract (translated)
人类可以通过阅读手册或指南来学习如何使用应用程序的用户界面(UI)。在文本旁边,这些资源包括可视化内容,例如UI截图和在文本中引用的应用图标图像。我们探索如何利用这些数据来学习通用的视觉语言学表示UI屏幕及其组件。这些表示在许多实际应用程序中非常有用,例如无障碍、语音导航和任务自动化。以前的UI表示模型依赖于UI元数据(UI树和无障碍标签),这往往缺失、不完整或无法访问。我们避免这种依赖关系,并提出Lexi,一个预训练的视觉和语言模型,以处理UI屏幕的独特特征,包括文本丰富性和上下文敏感性。为了训练Lexi,我们整理UICaption数据集,其中包括114,000个UI图像配对它们的功能描述。我们评估Lexi完成了四个任务:UI行动确认、基于指令的UI图像检索、基座参考表达式和UI实体识别。
URL
https://arxiv.org/abs/2301.10165
