Abstract
Benefiting from the inductive biases learned from large-scale datasets, open-vocabulary semantic segmentation (OVSS) leverages the power of vision-language models, such as CLIP, to achieve remarkable progress without requiring task-specific training. However, due to CLIP's pre-training nature on image-text pairs, it tends to focus on global semantic alignment, resulting in suboptimal performance when associating fine-grained visual regions with text. This leads to noisy and inconsistent predictions, particularly in local areas. We attribute this to a dispersed bias stemming from its contrastive training paradigm, which is difficult to alleviate using CLIP features alone. To address this, we propose a structure-aware feature rectification approach that incorporates instance-specific priors derived directly from the image. Specifically, we construct a region adjacency graph (RAG) based on low-level features (e.g., colour and texture) to capture local structural relationships and use it to refine CLIP features by enhancing local discrimination. Extensive experiments show that our method effectively suppresses segmentation noise, improves region-level consistency, and achieves strong performance on multiple open-vocabulary segmentation benchmarks.
Abstract (translated)
得益于从大规模数据集中学习到的归纳偏差，开放词汇语义分割（OVSS）利用了视觉-语言模型（如CLIP）的强大功能，在无需特定任务训练的情况下取得了显著进展。然而，由于CLIP在图像文本对上的预训练特性，它倾向于关注全局语义一致性，这导致在将细粒度的视觉区域与文本关联时表现不佳。这进而造成了局部区域预测中的噪声和不一致问题。我们将其归因于其对比学习范式带来的分散偏差，仅使用CLIP特征难以解决这一问题。 为了应对这一挑战，我们提出了一种基于结构感知的特征修正方法，该方法结合了直接从图像中提取的实例特定先验知识。具体而言，我们基于低级视觉特征（如颜色和纹理）构建了一个区域邻接图（RAG），以捕捉局部结构关系，并利用它来通过增强局部辨别能力来优化CLIP特征。 广泛的实验表明，我们的方法有效地抑制了分割中的噪声，提高了区域级别的一致性，并在多个开放词汇语义分割基准测试中取得了优异的性能。
URL
https://arxiv.org/abs/2512.07360
