Abstract
Fine-tuning Vision Foundation Models (VFMs) with a small number of parameters has shown remarkable performance in Domain Generalized Semantic Segmentation (DGSS). Most existing works either train lightweight adapters or refine intermediate features to achieve better generalization on unseen domains. However, they both overlook the fact that long-term pre-trained VFMs often exhibit artifacts, which hinder the utilization of valuable representations and ultimately degrade DGSS performance. Inspired by causal mechanisms, we observe that these artifacts are associated with non-causal factors, which usually reside in the low- and high-frequency components of the VFM spectrum. In this paper, we explicitly examine the causal and non-causal factors of features within VFMs for DGSS, and propose a simple yet effective method to identify and disentangle them, enabling more robust domain generalization. Specifically, we propose Causal-Tune, a novel fine-tuning strategy designed to extract causal factors and suppress non-causal ones from the features of VFMs. First, we extract the frequency spectrum of features from each layer using the Discrete Cosine Transform (DCT). A Gaussian band-pass filter is then applied to separate the spectrum into causal and non-causal components. To further refine the causal components, we introduce a set of causal-aware learnable tokens that operate in the frequency domain, while the non-causal components are discarded. Finally, refined features are transformed back into the spatial domain via inverse DCT and passed to the next layer. Extensive experiments conducted on various cross-domain tasks demonstrate the effectiveness of Causal-Tune. In particular, our method achieves superior performance under adverse weather conditions, improving +4.8% mIoU over the baseline in snow conditions.
Abstract (translated)
使用少量参数对视觉基础模型（VFMs）进行微调，在领域泛化语义分割（DGSS）中表现出卓越的性能。现有的大多数工作要么训练轻量级适配器，要么改进中间特征以实现在未见领域的更好泛化能力。然而，它们都忽视了这样一个事实：长期预训练的VFMs往往展现出一些阻碍有价值表征利用的艺术瑕疵，最终导致DGSS性能下降。受因果机制启发，我们观察到这些艺术瑕疵通常与非因果因素相关联，而后者通常存在于VFM频谱中的低频和高频部分中。 在本文中，我们明确地考察了用于DGSS的VFMs特征中的因果和非因果因素，并提出了一种简单但有效的方法来识别并分离它们，从而实现更稳健的领域泛化。具体来说，我们提出了Causal-Tune，这是一种新的微调策略，旨在从VFMs特征中提取因果因素并抑制非因果因素。首先，我们使用离散余弦变换（DCT）从每一层中提取频谱特性。然后应用高斯带通滤波器将频谱分为因果和非因果部分。为了进一步精炼因果部分，我们在频域中引入了一组具有因果感知的可学习令牌，而忽略非因果部分。最后，通过逆DCT将细化后的特征转换回空间域，并传递给下一层。 在各种跨领域任务上的广泛实验表明了Causal-Tune的有效性。特别是，在恶劣天气条件下，我们的方法比基线改进了+4.8%的mIoU（雪条件下的表现）。
URL
https://arxiv.org/abs/2512.16567
