Abstract
Change Detection (CD) aims to identify pixels with semantic changes between images. However, annotating massive numbers of pixel-level images is labor-intensive and costly, especially for multi-temporal images, which require pixel-wise comparisons by human experts. Considering the excellent performance of visual language models (VLMs) for zero-shot, open-vocabulary, etc. with prompt-based reasoning, it is promising to utilize VLMs to make better CD under limited labeled data. In this paper, we propose a VLM guidance-based semi-supervised CD method, namely DiffMatch. The insight of DiffMatch is to synthesize free change labels using VLMs to provide additional supervision signals for unlabeled data. However, almost all current VLMs are designed for single-temporal images and cannot be directly applied to bi- or multi-temporal images. Motivated by this, we first propose a VLM-based mixed change event generation (CEG) strategy to yield pseudo labels for unlabeled CD data. Since the additional supervised signals provided by these VLM-driven pseudo labels may conflict with the pseudo labels from the consistency regularization paradigm (e.g. FixMatch), we propose the dual projection head for de-entangling different signal sources. Further, we explicitly decouple the bi-temporal images semantic representation through two auxiliary segmentation decoders, which are also guided by VLM. Finally, to make the model more adequately capture change representations, we introduce metric-aware supervision by feature-level contrastive loss in auxiliary branches. Extensive experiments show the advantage of DiffMatch. For instance, DiffMatch improves the FixMatch baseline by +5.3 IoU on WHU-CD and by +2.4 IoU on LEVIR-CD with 5% labels. In addition, our CEG strategy, in an un-supervised manner, can achieve performance far superior to state-of-the-art un-supervised CD methods.
Abstract (translated)
变化检测（CD）旨在识别图像之间语义变化的像素。然而，标注大量像素级别图像劳动密集且代价昂贵，尤其是在需要通过人类专家进行逐像素比较的多时间尺度图像上。考虑到视觉语言模型（VLMs）在零散、开词等提示下推理的优秀表现，我们有望在有限的标注数据下使用VLMs实现更好的CD。在本文中，我们提出了一种基于VLM指导的半监督CD方法，即DiffMatch。DiffMatch的洞察力在于利用VLMs合成自由变化标签，为未标注数据提供额外的监督信号。然而，几乎所有当前的VLMs都是为单时间尺度图像设计的，不能直接应用于双或多时间尺度图像。因此，我们首先提出了一种基于VLM的混合变化事件生成（CEG）策略，为未标注的CD数据生成伪标签。由于这些VLM驱动的伪标签可能与一致性正则化范式（例如FixMatch）中的伪标签发生冲突，我们提出了双投影头以解开不同信号源。此外，我们通过两个辅助分割解码器明确地解耦双时间尺度图像的语义表示。最后，为了使模型更好地捕捉变化表示，我们在辅助分支上引入基于特征级的对比损失的度量指导。大量实验证明，DiffMatch具有优势。例如，DiffMatch在WHU-CD上的IoU提高了+5.3，而在LEVIR-CD上的IoU提高了+2.4，同时我们的CEG策略在没有监督的情况下可以达到比最先进的无监督CD方法更出色的性能。
URL
https://arxiv.org/abs/2405.04788
