Abstract
Facilitating an entity's interaction with objects requires accurately identifying parts that afford specific actions. Weakly supervised affordance grounding (WSAG) seeks to imitate human learning from third-person demonstrations, where humans intuitively grasp functional parts without needing pixel-level annotations. To achieve this, grounding is typically learned using a shared classifier across images from different perspectives, along with distillation strategies incorporating part discovery process. However, since affordance-relevant parts are not always easily distinguishable, models primarily rely on classification, often focusing on common class-specific patterns that are unrelated to affordance. To address this limitation, we move beyond isolated part-level learning by introducing selective prototypical and pixel contrastive objectives that adaptively learn affordance-relevant cues at both the part and object levels, depending on the granularity of the available information. Initially, we find the action-associated objects in both egocentric (object-focused) and exocentric (third-person example) images by leveraging CLIP. Then, by cross-referencing the discovered objects of complementary views, we excavate the precise part-level affordance clues in each perspective. By consistently learning to distinguish affordance-relevant regions from affordance-irrelevant background context, our approach effectively shifts activation from irrelevant areas toward meaningful affordance cues. Experimental results demonstrate the effectiveness of our method. Codes are available at this http URL.
Abstract (translated)
促进实体与物体互动需要准确识别能够执行特定操作的部分。弱监督功能定位（WSAG）旨在模仿人类从第三人称演示中学习的方式，即人类可以直观地理解功能性部分，而无需像素级别的标注信息。为了实现这一目标，通常使用跨不同视角图像的共享分类器来学习定位，并结合发现部位的过程中的蒸馏策略。然而，由于与功能相关的部分并不总是易于区分，模型主要依赖于分类，往往集中于无关紧要的类别特定模式而非功能性相关模式。 为了解决这个局限性，我们通过引入选择性的原型和像素对比目标超越了孤立的部分级学习方法，这些目标可以根据可用信息的粒度自适应地在部件级别和物体级别上学习功能相关的提示。首先，我们利用CLIP（一种视觉语言模型）来确定第一人称视角（以对象为中心）和第三人称视角图像中的与动作关联的对象。然后，通过将不同视图中发现的对象相互参照，我们在每个视角中挖掘精确的功能部分线索。通过持续学习区分功能相关的区域和非相关背景上下文，我们的方法有效地将激活从无关区域转移到有意义的功能提示上。 实验结果证明了我们方法的有效性。代码可在上述提供的链接处获取。
URL
https://arxiv.org/abs/2508.07877
