Abstract
Graphical User Interface (GUI) datasets are crucial for various downstream tasks. However, GUI datasets often generate annotation information through automatic labeling, which commonly results in inaccurate GUI element BBox annotations, including missing, duplicate, or meaningless BBoxes. These issues can degrade the performance of models trained on these datasets, limiting their effectiveness in real-world applications. Additionally, existing GUI datasets only provide BBox annotations visually, which restricts the development of visually related GUI downstream tasks. To address these issues, we introduce PixelWeb, a large-scale GUI dataset containing over 100,000 annotated web pages. PixelWeb is constructed using a novel automatic annotation approach that integrates visual feature extraction and Document Object Model (DOM) structure analysis through two core modules: channel derivation and layer analysis. Channel derivation ensures accurate localization of GUI elements in cases of occlusion and overlapping elements by extracting BGRA four-channel bitmap annotations. Layer analysis uses the DOM to determine the visibility and stacking order of elements, providing precise BBox annotations. Additionally, PixelWeb includes comprehensive metadata such as element images, contours, and mask annotations. Manual verification by three independent annotators confirms the high quality and accuracy of PixelWeb annotations. Experimental results on GUI element detection tasks show that PixelWeb achieves performance on the mAP95 metric that is 3-7 times better than existing datasets. We believe that PixelWeb has great potential for performance improvement in downstream tasks such as GUI generation and automated user interaction.
Abstract (translated)
图形用户界面（GUI）数据集对于各种下游任务至关重要。然而，这些数据集中生成注释信息通常通过自动标注完成，这常常导致不准确的GUI元素边界框（BBox）注释问题，包括遗漏、重复或无意义的BBoxes。这些问题会降低在这些数据集上训练的模型性能，在现实世界的应用中限制了它们的有效性。此外，现有的GUI数据集仅提供可视化的边界框标注信息，这限制了与视觉相关GUI下游任务的发展。 为了解决上述问题，我们引入了一个大型的GUI数据集PixelWeb，它包含了超过10万张注释网页。PixelWeb使用了一种新颖的自动标注方法来构建，该方法结合了视觉特征提取和文档对象模型（DOM）结构分析，并通过两个核心模块实现：通道导出和层级分析。通道导出通过提取BGRA四通道位图注释确保在遮挡和元素重叠的情况下准确定位GUI元素。层级分析利用DOM确定元素的可见性和堆叠顺序，从而提供精确的BBox注释。此外，PixelWeb还包含全面的元数据信息，如元素图像、轮廓及掩码标注。 通过三名独立标注者的手动验证确认了PixelWeb注释具有高质量和准确性。在GUI元素检测任务上的实验结果表明，在mAP95指标上，PixelWeb的表现比现有的数据集高出3-7倍。我们相信PixelWeb对于下游任务如GUI生成和自动化用户交互等领域的性能改进具有巨大的潜力。
URL
https://arxiv.org/abs/2504.16419
