Abstract
Weakly Supervised Object Localization (WSOL), which aims to localize objects by only using image-level labels, has attracted much attention because of its low annotation cost in real applications. Current studies focus on the Class Activation Map (CAM) of CNN and the self-attention map of transformer to identify the region of objects. However, both CAM and self-attention maps can not learn pixel-level fine-grained information on the foreground objects, which hinders the further advance of WSOL. To address this problem, we initiatively leverage the capability of zero-shot generalization and fine-grained segmentation in Segment Anything Model (SAM) to boost the activation of integral object regions. Further, to alleviate the semantic ambiguity issue accrued in single point prompt-based SAM, we propose an innovative mask prompt to SAM (Pro2SAM) network with grid points for WSOL task. First, we devise a Global Token Transformer (GTFormer) to generate a coarse-grained foreground map as a flexible mask prompt, where the GTFormer jointly embeds patch tokens and novel global tokens to learn foreground semantics. Secondly, we deliver grid points as dense prompts into SAM to maximize the probability of foreground mask, which avoids the lack of objects caused by a single point/box prompt. Finally, we propose a pixel-level similarity metric to come true the mask matching from mask prompt to SAM, where the mask with the highest score is viewed as the final localization map. Experiments show that the proposed Pro2SAM achieves state-of-the-art performance on both CUB-200-2011 and ILSVRC, with 84.03\% and 66.85\% Top-1 Loc, respectively.
Abstract (translated)
弱监督目标定位（WSOL）旨在仅通过图像级别的标签来定位对象，由于其在实际应用中的低标注成本而备受关注。当前的研究主要集中在卷积神经网络（CNN）的类激活图（CAM）和变压器的自注意力图上以识别物体区域。然而，无论是CAM还是自注意力映射都无法学习前景对象上的像素级细粒度信息，这限制了WSOL技术的进步。为了解决这个问题，我们首次利用Segment Anything Model (SAM) 的零样本泛化能力和细粒度分割能力来增强整体目标区域的激活程度。 为了缓解基于单点提示的SAM在语义模糊性方面的问题，我们提出了一种创新性的mask prompt到SAM（Pro2SAM）网络，并使用网格点来进行WSOL任务。首先，我们设计了一个全局令牌变压器（GTFormer），以生成一个粗粒度的前景图作为灵活的掩码提示，在该过程中，GTFormer同时嵌入了补丁令牌和新的全球令牌来学习前景语义。 其次，我们将密集的网格点作为提示输入到SAM中，以此最大化前景掩码的概率，从而避免由单个点/框提示所引起的对象缺失问题。最后，我们提出了一种像素级相似度度量方法来实现从mask prompt到SAM的掩码匹配过程，其中得分最高的掩码被视为最终的目标定位图。 实验表明，在CUB-200-2011和ILSVRC数据集上，所提出的Pro2SAM网络实现了最先进的性能表现，其Top-1精度分别达到了84.03% 和 66.85%。
URL
https://arxiv.org/abs/2505.04905
