Abstract
A sliding-window inference strategy is commonly adopted in recent training-free open-vocabulary semantic segmentation methods to overcome limitation of the CLIP in processing high-resolution images. However, this approach introduces a new challenge: each window is processed independently, leading to semantic discrepancy across windows. To address this issue, we propose Global-Local Aligned CLIP~(GLA-CLIP), a framework that facilitates comprehensive information exchange across windows. Rather than limiting attention to tokens within individual windows, GLA-CLIP extends key-value tokens to incorporate contextual cues from all windows. Nevertheless, we observe a window bias: outer-window tokens are less likely to be attended, since query features are produced through interactions within the inner window patches, thereby lacking semantic grounding beyond their local context. To mitigate this, we introduce a proxy anchor, constructed by aggregating tokens highly similar to the given query from all windows, which provides a unified semantic reference for measuring similarity across both inner- and outer-window patches. Furthermore, we propose a dynamic normalization scheme that adjusts attention strength according to object scale by dynamically scaling and thresholding the attention map to cope with small-object scenarios. Moreover, GLA-CLIP can be equipped on existing methods and broad their receptive field. Extensive experiments validate the effectiveness of GLA-CLIP in enhancing training-free open-vocabulary semantic segmentation performance. Code is available at this https URL.
Abstract (translated)
近期无需训练的开放词汇语义分割方法普遍采用滑动窗口推理策略，以克服CLIP模型处理高分辨率图像的局限。然而该方法引入新挑战：各窗口独立处理导致窗口间出现语义不一致。针对此问题，我们提出全局-局部对齐CLIP（GLA-CLIP）框架，促进窗口间全面信息交互。GLA-CLIP将键值令牌扩展至整合所有窗口的上下文线索，而非局限于单个窗口内的令牌。进一步观察发现窗口偏差现象：由于查询特征通过内窗口块交互生成，缺乏本地上下文之外的语义依据，导致外窗口令牌更少被关注。为缓解此问题，我们引入代理锚点——通过聚合所有窗口中与给定查询高度相似的令牌构建，为测量内/外窗口块相似性提供统一语义参考。此外提出动态归一化方案：通过动态缩放和阈值化注意力图，依据物体尺度调整注意力强度，以应对小目标场景。GLA-CLIP还可赋能现有方法并扩大其感受野。大量实验验证GLA-CLIP能有效提升无需训练的开放词汇语义分割性能。代码见此https链接。
URL
https://arxiv.org/abs/2603.23030
