Abstract
Unsupervised semantic segmentation aims to automatically partition images into semantically meaningful regions by identifying global categories within an image corpus without any form of annotation. Building upon recent advances in self-supervised representation learning, we focus on how to leverage these large pre-trained models for the downstream task of unsupervised segmentation. We present PriMaPs - Principal Mask Proposals - decomposing images into semantically meaningful masks based on their feature representation. This allows us to realize unsupervised semantic segmentation by fitting class prototypes to PriMaPs with a stochastic expectation-maximization algorithm, PriMaPs-EM. Despite its conceptual simplicity, PriMaPs-EM leads to competitive results across various pre-trained backbone models, including DINO and DINOv2, and across datasets, such as Cityscapes, COCO-Stuff, and Potsdam-3. Importantly, PriMaPs-EM is able to boost results when applied orthogonally to current state-of-the-art unsupervised semantic segmentation pipelines.
Abstract (translated)
无监督语义分割旨在通过在图像集合中识别全局类别，自动将图像划分为语义上有意义的区域。无监督语义分割是基于自监督表示学习最近取得的进展，我们关注如何利用这些大型的预训练模型来实现下游任务的未监督分割。我们提出了PrimeMaPs - 主要掩码建议，通过基于它们的特征表示分解图像为语义上有意义的掩码。这使我们能够通过随机期望-最大化算法将类原型拟合到PrimeMaPs-EM，实现无监督语义分割。尽管其概念上很简单，但PrimeMaPs-EM在各种预训练骨干模型（包括DINO和DINOv2）和各种数据集（如Cityscapes、COCO-Stuff和Potsdam-3）上都取得了竞争力的结果。重要的是，当应用与当前最先进的无监督语义分割管道成角度时，PrimeMaPs-EM能够提高结果。
URL
https://arxiv.org/abs/2404.16818
