Abstract
We present LlamaSeg, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions. We reformulate image segmentation as a visual generation problem, representing masks as "visual" tokens and employing a LLaMA-style Transformer to predict them directly from image inputs. By adhering to the next-token prediction paradigm, our approach naturally integrates segmentation tasks into autoregressive architectures. To support large-scale training, we introduce a data annotation pipeline and construct the SA-OVRS dataset, which contains 2M segmentation masks annotated with over 5,800 open-vocabulary labels or diverse textual descriptions, covering a wide spectrum of real-world scenarios. This enables our model to localize objects in images based on text prompts and to generate fine-grained masks. To more accurately evaluate the quality of masks produced by visual generative models, we further propose a composite metric that combines Intersection over Union (IoU) with Average Hausdorff Distance (AHD), offering a more precise assessment of contour fidelity. Experimental results demonstrate that our method surpasses existing generative models across multiple datasets and yields more detailed segmentation masks.
Abstract (translated)
我们介绍了LlamaSeg，这是一种视觉自回归框架，通过自然语言指令统一了多种图像分割任务。我们将图像分割重新表述为一个视觉生成问题，将掩码表示为“视觉”令牌，并采用LLaMA风格的Transformer模型直接从图像输入预测它们。通过遵循下一个标记预测范式，我们的方法自然地将分割任务整合到自回归架构中。 为了支持大规模训练，我们引入了一个数据标注流水线并构建了SA-OVRS数据集，该数据集中包含200万张带有超过5800个开放词汇标签或多样化文本描述的分割掩码，涵盖了广泛的现实世界场景。这使得我们的模型能够在基于文本提示的情况下在图像中定位对象，并生成精细的掩码。 为了更准确地评估视觉生成模型产生的掩码质量，我们进一步提出了一种复合度量标准，该标准结合了交并比（IoU）和平均豪斯多夫距离（AHD），为轮廓保真度提供了更为精确的评价。实验结果表明，我们的方法在多个数据集上超越了现有的生成模型，并产生了更详细的分割掩码。
URL
https://arxiv.org/abs/2505.19422
