Abstract
Given the inherently costly and time-intensive nature of pixel-level annotation, the generation of synthetic datasets comprising sufficiently diverse synthetic images paired with ground-truth pixel-level annotations has garnered increasing attention recently for training high-performance semantic segmentation models. However, existing methods necessitate to either predict pseudo annotations after image generation or generate images conditioned on manual annotation masks, which incurs image-annotation semantic inconsistency or scalability problem. To migrate both problems with one stone, we present a novel dataset generative diffusion framework for semantic segmentation, termed JoDiffusion. Firstly, given a standard latent diffusion model, JoDiffusion incorporates an independent annotation variational auto-encoder (VAE) network to map annotation masks into the latent space shared by images. Then, the diffusion model is tailored to capture the joint distribution of each image and its annotation mask conditioned on a text prompt. By doing these, JoDiffusion enables simultaneously generating paired images and semantically consistent annotation masks solely conditioned on text prompts, thereby demonstrating superior scalability. Additionally, a mask optimization strategy is developed to mitigate the annotation noise produced during generation. Experiments on Pascal VOC, COCO, and ADE20K datasets show that the annotated dataset generated by JoDiffusion yields substantial performance improvements in semantic segmentation compared to existing methods.
Abstract (translated)
鉴于像素级标注固有的高成本和耗时特性，近期越来越多的研究关注于生成包含足够多样化合成图像及其真实像素级标注的合成数据集，以训练高性能的语义分割模型。然而，现有方法要么在图像生成后预测伪标签，要么根据手动注释掩码生成图像，这会导致图像-标注语义不一致或可扩展性问题。为了同时解决这两个问题，我们提出了一种新的针对语义分割的数据集生成扩散框架，名为JoDiffusion。 首先，给定标准的潜在扩散模型，JoDiffusion整合了一个独立的注释变分自动编码器（VAE）网络，将注释掩码映射到与图像共享的潜在空间中。然后，该扩散模型被定制以捕捉每个图像及其标注掩码在文本提示条件下的联合分布。通过这种方式，JoDiffusion能够仅根据文本提示同时生成配对的图像和语义一致性的标注掩码，从而表现出卓越的可扩展性。 此外，我们开发了一种掩码优化策略来减轻生成过程中的注释噪声。在Pascal VOC、COCO和ADE20K数据集上的实验表明，JoDiffusion生成的标注数据集相比现有方法，在语义分割性能上取得了显著提升。
URL
https://arxiv.org/abs/2512.13014
