Abstract
The 3D weakly-supervised visual grounding task aims to localize oriented 3D boxes in point clouds based on natural language descriptions without requiring annotations to guide model learning. This setting presents two primary challenges: category-level ambiguity and instance-level complexity. Category-level ambiguity arises from representing objects of fine-grained categories in a highly sparse point cloud format, making category distinction challenging. Instance-level complexity stems from multiple instances of the same category coexisting in a scene, leading to distractions during grounding. To address these challenges, we propose a novel weakly-supervised grounding approach that explicitly differentiates between categories and instances. In the category-level branch, we utilize extensive category knowledge from a pre-trained external detector to align object proposal features with sentence-level category features, thereby enhancing category awareness. In the instance-level branch, we utilize spatial relationship descriptions from language queries to refine object proposal features, ensuring clear differentiation among objects. These designs enable our model to accurately identify target-category objects while distinguishing instances within the same category. Compared to previous methods, our approach achieves state-of-the-art performance on three widely used benchmarks: Nr3D, Sr3D, and ScanRef.
Abstract (translated)
三维弱监督视觉定位任务的目标是在没有注释引导模型学习的情况下，根据自然语言描述在点云中定位定向的3D边界框。这种设置提出了两个主要挑战：类别级模糊性和实例级复杂性。类别级模糊性源于以高度稀疏的点云格式表示细粒度类别的对象，这使得分类变得困难。实例级复杂性则来源于同一场景中存在相同类别的多个实例，导致定位时产生干扰。 为了应对这些挑战，我们提出了一种新颖的弱监督定位方法，该方法明确地区分了类别和实例。在类别级分支中，我们利用预训练外部检测器提供的广泛类别知识来对齐对象建议特征与句子级别的类别特征，从而增强模型对类别的识别能力。在实例级分支中，我们利用语言查询中的空间关系描述来细化对象提议的特征，确保不同对象之间的清晰区分。 这些设计使得我们的模型能够准确地识别目标类别的对象，并在同一类别内区分不同的实例。相较于先前的方法，在三个广泛使用的基准测试（Nr3D、Sr3D和ScanRef）上，我们的方法达到了最先进的性能水平。
URL
https://arxiv.org/abs/2505.01809
