Abstract
The Segment Anything Model (SAM) is a deep neural network foundational model designed to perform instance segmentation which has gained significant popularity given its zero-shot segmentation ability. SAM operates by generating masks based on various input prompts such as text, bounding boxes, points, or masks, introducing a novel methodology to overcome the constraints posed by dataset-specific scarcity. While SAM is trained on an extensive dataset, comprising ~11M images, it mostly consists of natural photographic images with only very limited images from other modalities. Whilst the rapid progress in visual infrared surveillance and X-ray security screening imaging technologies, driven forward by advances in deep learning, has significantly enhanced the ability to detect, classify and segment objects with high accuracy, it is not evident if the SAM zero-shot capabilities can be transferred to such modalities. This work assesses SAM capabilities in segmenting objects of interest in the X-ray/infrared modalities. Our approach reuses the pre-trained SAM with three different prompts: bounding box, centroid and random points. We present quantitative/qualitative results to showcase the performance on selected datasets. Our results show that SAM can segment objects in the X-ray modality when given a box prompt, but its performance varies for point prompts. Specifically, SAM performs poorly in segmenting slender objects and organic materials, such as plastic bottles. We find that infrared objects are also challenging to segment with point prompts given the low-contrast nature of this modality. This study shows that while SAM demonstrates outstanding zero-shot capabilities with box prompts, its performance ranges from moderate to poor for point prompts, indicating that special consideration on the cross-modal generalisation of SAM is needed when considering use on X-ray/infrared imagery.
Abstract (translated)
Segment Anything Model (SAM)是一种深度神经网络基础模型，旨在执行实例分割，由于其零 shot分割能力而获得了显著的流行。SAM通过根据各种输入提示生成掩码来操作，引入了一种新的方法来克服数据集特异性稀疏性的限制。尽管SAM在广泛的训练数据集上进行训练，包括~11M张图像，但它主要由仅包含非常有限其他模态图像的自然摄影图像组成。尽管随着深度学习技术的进步，视觉红外监视和X射线安全筛选成像技术的发展，大大提高了检测、分类和分割物体的准确性，但目前尚不清楚SAM的零 shot分割能力是否可以应用到这种模态。 本文评估了SAM在X-ray/红外模态中分割物体的能力。我们的方法重用了预训练的SAM，并使用三种不同的提示：边界框、中心点和随机点。我们提供了定量/定性结果，以展示SAM在这些选定数据集上的性能。我们的结果表明，当给定边界框提示时，SAM可以在X-ray模态上分割物体，但性能因点提示而异。具体来说，SAM在分割细长物体和有机材料（如塑料瓶）方面表现不佳。我们发现，由于这种模态的低对比度性质，红外物体也难以通过点提示进行分割。 本研究显示，尽管SAM在边界框提示下表现出出色的零 shot能力，但其在点提示下的性能从中等到差，表明在考虑在X-ray/红外图像上使用SAM时，需要特别注意跨模态通用性。
URL
https://arxiv.org/abs/2404.12285
