Abstract
Weakly supervised visual grounding (VG) aims to locate objects in images based on text descriptions. Despite significant progress, existing methods lack strong cross-modal reasoning to distinguish subtle semantic differences in text expressions due to category-based and attribute-based ambiguity. To address these challenges, we introduce AlignCAT, a novel query-based semantic matching framework for weakly supervised VG. To enhance visual-linguistic alignment, we propose a coarse-grained alignment module that utilizes category information and global context, effectively mitigating interference from category-inconsistent objects. Subsequently, a fine-grained alignment module leverages descriptive information and captures word-level text features to achieve attribute consistency. By exploiting linguistic cues to their fullest extent, our proposed AlignCAT progressively filters out misaligned visual queries and enhances contrastive learning efficiency. Extensive experiments on three VG benchmarks, namely RefCOCO, RefCOCO+, and RefCOCOg, verify the superiority of AlignCAT against existing weakly supervised methods on two VG tasks. Our code is available at: this https URL.
Abstract (translated)
弱监督视觉接地（VG）旨在根据文本描述在图像中定位对象。尽管已经取得了显著进展，现有的方法仍缺乏强大的跨模态推理能力，难以区分基于类别和属性的文本表达中的细微语义差异。为了解决这些挑战，我们提出了AlignCAT，这是一种新颖的查询基语义匹配框架，用于弱监督视觉接地（VG）。 为了增强视觉-语言对齐，我们提出了一种粗粒度对齐模块，该模块利用类别信息和全局上下文，有效减轻了来自类不一致对象的干扰。随后，细粒度对齐模块通过利用描述性信息并捕捉单词级别的文本特征来实现属性一致性。通过充分利用语言线索，我们的AlignCAT逐步过滤掉不对齐的视觉查询，并提高对比学习效率。 在三个VG基准数据集（RefCOCO、RefCOCO+ 和 RefCOCOg）上进行的大量实验验证了AlignCAT在两项VG任务中优于现有的弱监督方法。我们的代码可在以下链接获取：this https URL.
URL
https://arxiv.org/abs/2508.03201
