Abstract
Detecting glass regions is a challenging task due to the ambiguity of their transparency and reflection properties. These transparent glasses share the visual appearance of both transmitted arbitrary background scenes and reflected objects, thus having no fixed patterns.Recent visual foundation models, which are trained on vast amounts of data, have manifested stunning performance in terms of image perception and image generation. To segment glass surfaces with higher accuracy, we make full use of two visual foundation models: Segment Anything (SAM) and Stable Diffusion.Specifically, we devise a simple glass surface segmentor named GEM, which only consists of a SAM backbone, a simple feature pyramid, a discerning query selection module, and a mask decoder. The discerning query selection can adaptively identify glass surface features, assigning them as initialized queries in the mask decoder. We also propose a Synthetic but photorealistic large-scale Glass Surface Detection dataset dubbed S-GSD via diffusion model with four different scales, which contain 1x, 5x, 10x, and 20x of the original real data size. This dataset is a feasible source for transfer learning. The scale of synthetic data has positive impacts on transfer learning, while the improvement will gradually saturate as the amount of data increases. Extensive experiments demonstrate that GEM achieves a new state-of-the-art on the GSD-S validation set (IoU +2.1%). Codes and datasets are available at: this https URL.
Abstract (translated)
检测玻璃区域是一个具有挑战性的任务，因为其透明度和反射特性具有不确定性。这些透明的玻璃分享传感和物体所具有的视觉外观，因此没有固定的模式。 训练大量数据的最近视觉基础模型在图像感知和图像生成方面表现出惊人的性能。为了更准确地分割玻璃表面，我们充分利用两个视觉基础模型：Segment Anything（SAM）和Stable Diffusion。具体来说，我们设计了一个简单的玻璃表面分割器GEM，它仅包含一个SAM骨架、一个简单的特征金字塔、一个精明的查询选择模块和一个掩码解码器。精明的查询选择可以动态地识别玻璃表面特征，并将它们作为初始化查询传递给掩码解码器。我们还通过扩散模型提出了一个合成但更真实的大规模玻璃表面检测数据集，其包含原始数据大小的1x、5x、10x和20x。这个数据集是一个可迁移学习的可行来源。合成数据的规模对迁移学习有积极影响，而随着数据量的增加，提高将逐渐趋于饱和。大量实验证明，GEM在GSD-S验证集上达到了最先进水平（IoU +2.1%）。代码和数据集可在此处下载：https://this URL。
URL
https://arxiv.org/abs/2401.15282
