Abstract
The Segmentation Anything Model 2 (SAM2) has proven to be a powerful foundation model for promptable visual object segmentation in both images and videos, capable of storing object-aware memories and transferring them temporally through memory blocks. While SAM2 excels in video object segmentation by providing dense segmentation masks based on prompts, extending it to dense Video Semantic Segmentation (VSS) poses challenges due to the need for spatial accuracy, temporal consistency, and the ability to track multiple objects with complex boundaries and varying scales. This paper explores the extension of SAM2 for VSS, focusing on two primary approaches and highlighting firsthand observations and common challenges faced during this process. The first approach involves using SAM2 to extract unique objects as masks from a given image, with a segmentation network employed in parallel to generate and refine initial predictions. The second approach utilizes the predicted masks to extract unique feature vectors, which are then fed into a simple network for classification. The resulting classifications and masks are subsequently combined to produce the final segmentation. Our experiments suggest that leveraging SAM2 enhances overall performance in VSS, primarily due to its precise predictions of object boundaries.
Abstract (translated)
《分割任何模型2（SAM2）》已被证明是图像和视频中提示式视觉对象分割的强大基础模型，能够存储物体感知记忆并通过内存块进行时间上的传递。虽然SAM2在视频对象分割方面表现出色，可以通过提示提供密集的分割掩码，但将其扩展到密集型视频语义分割（VSS）则面临挑战，因为需要空间精度、时间一致性以及跟踪具有复杂边界和不同尺度的多个物体的能力。本文探讨了将SAM2应用于VSS的方法，并重点介绍了两种主要方法及在此过程中亲身经历的问题与共同面临的挑战。 第一种方法是利用SAM2从给定图像中提取独特的对象作为掩码，同时采用分割网络并行生成和优化初始预测。第二种方法则是使用预测的掩码来抽取独特的特征向量，然后将这些向量输入到一个简单的分类网络进行分类处理。最后，通过合并得到的分类结果与掩码来产生最终的分割。 我们的实验表明，利用SAM2可以显著提高VSS的整体性能，主要原因是它能够精确地预测物体边界。
URL
https://arxiv.org/abs/2512.01774
