Abstract
Multimodal Large Language Models (MLLMs) have shown exceptional capabilities in vision-language tasks. However, effectively integrating image segmentation into these models remains a significant challenge. In this work, we propose a novel text-as-mask paradigm that casts image segmentation as a text generation problem, eliminating the need for additional decoders and significantly simplifying the segmentation process. Our key innovation is semantic descriptors, a new textual representation of segmentation masks where each image patch is mapped to its corresponding text label. We first introduce image-wise semantic descriptors, a patch-aligned textual representation of segmentation masks that integrates naturally into the language modeling pipeline. To enhance efficiency, we introduce the Row-wise Run-Length Encoding (R-RLE), which compresses redundant text sequences, reducing the length of semantic descriptors by 74% and accelerating inference by $3\times$, without compromising performance. Building upon this, our initial framework Text4Seg achieves strong segmentation performance across a wide range of vision tasks. To further improve granularity and compactness, we propose box-wise semantic descriptors, which localizes regions of interest using bounding boxes and represents region masks via structured mask tokens called semantic bricks. This leads to our refined model, Text4Seg++, which formulates segmentation as a next-brick prediction task, combining precision, scalability, and generative efficiency. Comprehensive experiments on natural and remote sensing datasets show that Text4Seg++ consistently outperforms state-of-the-art models across diverse benchmarks without any task-specific fine-tuning, while remaining compatible with existing MLLM backbones. Our work highlights the effectiveness, scalability, and generalizability of text-driven image segmentation within the MLLM framework.
Abstract (translated)
多模态大型语言模型（MLLM）在视觉-语言任务中展现了卓越的能力。然而，有效地将图像分割整合到这些模型中仍然是一个重大挑战。在这项工作中，我们提出了一种新的文本作为掩码的范式，它将图像分割视为文本生成问题，消除了对额外解码器的需求，并大大简化了分割过程。我们的关键创新是语义描述符，这是一种全新的、用于表示分割掩码的文本形式，在这种形式中，每个图像块都映射到相应的文本标签上。我们首先介绍了基于图象的语义描述符，这是与语言建模管道自然整合的一种补丁对齐的文本表示方式。为了提高效率，我们引入了行级行程长度编码（R-RLE），它压缩冗余的文本序列，在不牺牲性能的前提下将语义描述符长度减少74%，并将推理速度提高了3倍。在此基础上，我们的初始框架Text4Seg在各种视觉任务中实现了强大的分割效果。为了进一步提高精确度和紧凑性，我们提出了基于边界框的语义描述符，该方法使用结构化的掩码标记（称为语义砖）来定位感兴趣区域并表示区域掩码。这导致了我们的改进模型Text4Seg++，它将分割定义为下一个砖块预测任务，并结合了精确度、可扩展性和生成效率。在自然和遥感数据集上的全面实验表明，在没有任何特定任务的微调的情况下，Text4Seg++始终优于各种基准测试中的现有最先进的模型，同时保持与现有的MLLM骨干结构兼容。我们的工作突显了基于文本驱动图像分割的有效性、可扩展性和泛化能力。
URL
https://arxiv.org/abs/2509.06321
