Abstract
Open-Vocabulary Semantic Segmentation (OVSS) assigns pixel-level labels from an open set of text-defined categories, demanding reliable generalization to unseen classes at inference. Although modern vision-language models (VLMs) support strong open-vocabulary recognition, their representations learned through global contrastive objectives remain suboptimal for dense prediction, prompting many OVSS methods to depend on limited adaptation or refinement of image-text similarity maps. This, in turn, restricts spatial precision and robustness in complex, cluttered scenes. We introduce this http URL, extending this http URL into a dedicated framework for OVSS. Our contributions are four-fold. First, we design a task-specific architecture tailored to this backbone, systematically adapting established design principles from prior open-vocabulary segmentation work. Second, we jointly leverage text embeddings aligned with both the global [CLS] token and local patch-level visual features from ViT-based encoder, effectively combining semantic discrimination with fine-grained spatial locality. Third, unlike prior approaches that rely primarily on post hoc similarity refinement, we perform early refinement of visual representations prior to image-text interaction, followed by late refinement of the resulting image-text correlation features, enabling more accurate and robust dense predictions in cluttered scenes. Finally, we propose a high-resolution local-global inference strategy based on sliding-window aggregation, which preserves spatial detail while maintaining global context. We conduct extensive experiments on five widely adopted OVSS benchmarks to evaluate our approach. The results demonstrate its effectiveness and robustness, consistently outperforming current state-of-the-art methods.
Abstract (translated)
开放词汇语义分割（OVSS）从文本定义的开放类别集合中分配像素级标签，要求在推理时对未见类别具有可靠的泛化能力。尽管现代视觉-语言模型（VLMs）支持强大的开放词汇识别，但其通过全局对比目标学习到的表征对于密集预测仍不够优化，促使许多OVSS方法依赖于对图像-文本相似度图的有限适应或细化。这反过来限制了复杂、杂乱场景中的空间精确性和鲁棒性。我们提出此http URL，将此http URL扩展为OVSS的专用框架。我们的贡献有四点。首先，我们设计了针对该主干网络的任务特定架构，系统性地调整了先前开放词汇分割工作中已确立的设计原则。其次，我们联合利用与全局[CLS]令牌和ViT编码器局部图像块级视觉特征均对齐的文本嵌入，有效结合语义判别性与细粒度空间局部性。第三，不同于主要依赖事后相似度细化的先前方法，我们在图像-文本交互之前对视觉表征进行早期细化，随后对生成的图像-文本相关特征进行后期细化，从而在杂乱场景中实现更准确、鲁棒的密集预测。最后，我们提出了一种基于滑动窗口聚合的高分辨率局部-全局推理策略，在保持全局上下文的同时保留空间细节。我们在五个广泛采用的OVSS基准上进行了广泛实验以评估我们的方法。结果证明了其有效性和鲁棒性，持续优于当前最先进的方法。
URL
https://arxiv.org/abs/2603.19531
