Abstract
Open-world semantic segmentation presently relies significantly on extensive image-text pair datasets, which often suffer from a lack of fine-grained pixel annotations on sufficient categories. The acquisition of such data is rendered economically prohibitive due to the substantial investments of both human labor and time. In light of the formidable image generation capabilities of diffusion models, we introduce a novel diffusion model-driven pipeline for automatically generating datasets tailored to the needs of open-world semantic segmentation, named "MagicSeg". Our MagicSeg initiates from class labels and proceeds to generate high-fidelity textual descriptions, which in turn serve as guidance for the diffusion model to generate images. Rather than only generating positive samples for each label, our process encompasses the simultaneous generation of corresponding negative images, designed to serve as paired counterfactual samples for contrastive training. Then, to provide a self-supervised signal for open-world segmentation pretraining, our MagicSeg integrates an open-vocabulary detection model and an interactive segmentation model to extract object masks as precise segmentation labels from images based on the provided category labels. By applying our dataset to the contrastive language-image pretraining model with the pseudo mask supervision and the auxiliary counterfactual contrastive training, the downstream model obtains strong performance on open-world semantic segmentation. We evaluate our model on PASCAL VOC, PASCAL Context, and COCO, achieving SOTA with performance of 62.9%, 26.7%, and 40.2%, respectively, demonstrating our dataset's effectiveness in enhancing open-world semantic segmentation capabilities. Project website: this https URL.
Abstract (translated)
开放世界语义分割目前严重依赖大规模图像-文本对数据集，这些数据集通常存在对足够类别的细粒度像素标注不足的问题。由于人力和时间的大量投入，此类数据的获取在经济上往往不可行。鉴于扩散模型强大的图像生成能力，我们提出了一种新颖的、基于扩散模型的自动化数据集生成流程，专为开放世界语义分割需求定制，命名为"MagicSeg"。MagicSeg从类别标签出发，首先生成高保真文本描述，进而指导扩散模型生成图像。我们的流程不仅为每个标签生成正样本，还同时生成对应的负样本图像，旨在作为对比训练的配对反事实样本。随后，为提供开放世界分割预训练的自监督信号，MagicSeg整合了开放词汇检测模型和交互式分割模型，依据给定的类别标签从图像中提取对象掩码作为精确的分割标注。通过将我们生成的数据集应用于对比语言-图像预训练模型，并配合伪掩码监督与辅助反事实对比训练，下游模型在开放世界语义分割上展现出卓越性能。我们在PASCAL VOC、PASCAL Context和COCO数据集上评估模型，分别达到62.9%、26.7%和40.2%的SOTA性能，证明了本数据集在提升开放世界语义分割能力方面的有效性。项目网站：此https链接。
URL
https://arxiv.org/abs/2603.19575
