Abstract
Weakly Supervised Semantic Segmentation (WSSS) with image level labels aims to produce pixel level predictions without requiring dense annotations. While recent approaches have leveraged generative models to augment existing data, they remain dependent on real world training samples. In this paper, we introduce a novel direction, Zero Shot Weakly Supervised Semantic Segmentation (ZSWSSS), and propose SynthSeg Agents, a multi agent framework driven by Large Language Models (LLMs) to generate synthetic training data entirely without real images. SynthSeg Agents comprises two key modules, a Self Refine Prompt Agent and an Image Generation Agent. The Self Refine Prompt Agent autonomously crafts diverse and semantically rich image prompts via iterative refinement, memory mechanisms, and prompt space exploration, guided by CLIP based similarity and nearest neighbor diversity filtering. These prompts are then passed to the Image Generation Agent, which leverages Vision Language Models (VLMs) to synthesize candidate images. A frozen CLIP scoring model is employed to select high quality samples, and a ViT based classifier is further trained to relabel the entire synthetic dataset with improved semantic precision. Our framework produces high quality training data without any real image supervision. Experiments on PASCAL VOC 2012 and COCO 2014 show that SynthSeg Agents achieves competitive performance without using real training images. This highlights the potential of LLM driven agents in enabling cost efficient and scalable semantic segmentation.
Abstract (translated)
弱监督语义分割（WSSS）使用图像级标签的目标是无需密集注释就能生成像素级别的预测。尽管最近的研究已经利用了生成模型来增强现有数据，但它们仍然依赖于真实世界的训练样本。在本文中，我们提出了一种新颖的方向——零样本弱监督语义分割（ZSWSSS），并提出了SynthSeg Agents，这是一个由大型语言模型（LLMs）驱动的多代理框架，用于完全不使用真实图像生成合成训练数据。SynthSeg Agents 包括两个关键模块：自我优化提示代理和图像生成代理。 **自我优化提示代理**自主地通过迭代细化、记忆机制以及基于CLIP相似性和最近邻多样性过滤的提示空间探索来创建多样且语义丰富的图像提示。 这些提示随后传递给**图像生成代理**，该代理利用视觉语言模型（VLMs）合成候选图像。一个冻结的CLIP评分模型被用来选择高质量样本，并通过训练一个基于ViT的分类器进一步为整个合成数据集重新标记，以提高语义精度。 我们的框架能够在没有任何真实图像监督的情况下产生高质量的训练数据。在PASCAL VOC 2012和COCO 2014上的实验表明，SynthSeg Agents 在不使用任何实际训练图像的情况下也达到了竞争性的性能水平。这突显了LLM驱动代理在实现低成本且可扩展语义分割中的潜力。
URL
https://arxiv.org/abs/2512.15310
