Abstract
In this paper, we propose a weakly supervised semantic segmentation approach for food images which takes advantage of the zero-shot capabilities and promptability of the Segment Anything Model (SAM) along with the attention mechanisms of Vision Transformers (ViTs). Specifically, we use class activation maps (CAMs) from ViTs to generate prompts for SAM, resulting in masks suitable for food image segmentation. The ViT model, a Swin Transformer, is trained exclusively using image-level annotations, eliminating the need for pixel-level annotations during training. Additionally, to enhance the quality of the SAM-generated masks, we examine the use of image preprocessing techniques in combination with single-mask and multi-mask SAM generation strategies. The methodology is evaluated on the FoodSeg103 dataset, generating an average of 2.4 masks per image (excluding background), and achieving an mIoU of 0.54 for the multi-mask scenario. We envision the proposed approach as a tool to accelerate food image annotation tasks or as an integrated component in food and nutrition tracking applications.
Abstract (translated)
在这篇论文中，我们提出了一种针对食品图像的弱监督语义分割方法，该方法利用了Segment Anything Model (SAM) 的零样本能力和可提示性以及Vision Transformers (ViTs) 的注意力机制。具体来说，我们使用来自ViT的类激活图(CAMs)为SAM生成提示，从而产生适合于食品图像分割的掩码。作为ViT模型的Swin Transformer仅通过图像级别的标注进行训练，无需在训练过程中使用像素级别的注释。 为了提高由SAM生成的掩码的质量，我们还研究了结合单个掩码和多个掩码的SAM生成策略与图像预处理技术相结合的方法。该方法在FoodSeg103数据集上进行了评估，在不包括背景的情况下平均每张图片产生2.4个掩码，并且在多掩码场景下达到了0.54的mIoU。 我们设想提出的这种方法可以作为一种工具来加速食品图像标注任务，或者作为食物和营养追踪应用中的集成组件。
URL
https://arxiv.org/abs/2509.19028
