Abstract
This paper proposes a novel self-supervised learning method for semantic segmentation using selective masking image reconstruction as the pretraining task. Our proposed method replaces the random masking augmentation used in most masked image modelling pretraining methods. The proposed selective masking method selectively masks image patches with the highest reconstruction loss by breaking the image reconstruction pretraining into iterative steps to leverage the trained model's knowledge. We show on two general datasets (Pascal VOC and Cityscapes) and two weed segmentation datasets (Nassar 2020 and Sugarbeets 2016) that our proposed selective masking method outperforms the traditional random masking method and supervised ImageNet pretraining on downstream segmentation accuracy by 2.9% for general datasets and 2.5% for weed segmentation datasets. Furthermore, we found that our selective masking method significantly improves accuracy for the lowest-performing classes. Lastly, we show that using the same pretraining and downstream dataset yields the best result for low-budget self-supervised pretraining. Our proposed Selective Masking Image Reconstruction method provides an effective and practical solution to improve end-to-end semantic segmentation workflows, especially for scenarios that require limited model capacity to meet inference speed and computational resource requirements.
Abstract (translated)
本文提出了一种新颖的自监督学习方法，用于语义分割任务，采用选择性掩码图像重建作为预训练任务。我们的方法用选择性掩码替代了大多数掩码图像建模预训练方法中使用的随机掩码增强技术。所提出的这种方法通过将图像重构预训练分解为迭代步骤来选择性地对具有最高重构损失的图像块进行掩码，从而利用已训练模型的知识。 我们在两个通用数据集（Pascal VOC 和 Cityscapes）以及两个杂草分割数据集（Nassar 2020 和 Sugarbeets 2016）上进行了实验，结果表明我们的选择性掩码方法在下游分割精度方面优于传统的随机掩码方法和监督的ImageNet预训练：对于通用数据集提高了2.9%，对于杂草分割数据集提高了2.5%。 此外，我们发现选择性掩码方法显著提升了表现最差类别（即那些分类准确性最低的数据）的准确率。最后，我们还证明了使用相同的预训练和下游数据集可以为低预算自监督预训练提供最佳结果。 我们的选择性掩码图像重建方法为改进端到端语义分割工作流程提供了有效且实用的解决方案，特别是在需要有限模型容量以满足推理速度和计算资源要求的情况下。
URL
https://arxiv.org/abs/2512.06981
