Abstract
This paper presents a new unified approach to semantic segmentation in both images and videos by using language modeling to output the masks as sequences of discrete tokens. We use run length encoding (RLE) to discretize the segmentation masks and then train a modified version of Pix2Seq \cite{p2s} to output these RLE tokens through autoregression. We propose novel tokenization strategies to compress the length of the token sequence to make it practicable to extend this approach to videos. We also show how instance information can be incorporated into the tokenization process to perform panoptic segmentation. We evaluate our proposed models on two datasets to show that they are competitive with the state of the art in spite of being bottlenecked by our limited computational resources.
Abstract (translated)
本文提出了一种新的统一方法，用于图像和视频的语义分割，该方法通过使用语言模型以离散令牌序列的形式输出掩码来实现。我们采用游程编码（RLE）将分割掩码进行离散化处理，并训练了Pix2Seq的一个修改版本\cite{p2s}，使其能够通过自回归方式生成这些RLE令牌。为了使该方法在视频应用中具有实用性，我们提出了一些新颖的标记策略来压缩令牌序列长度。此外，我们展示了如何将实例信息整合到标记化过程中以执行全景分割（panoptic segmentation）。我们在两个数据集上评估了所提出的模型，并表明尽管受到计算资源限制的影响，我们的模型仍能与当前最佳方法相媲美。
URL
https://arxiv.org/abs/2602.21627
