Abstract
The goal of video segmentation is to accurately segment and track every pixel in diverse scenarios. In this paper, we present Tube-Link, a versatile framework that addresses multiple core tasks of video segmentation with a unified architecture. Our framework is a near-online approach that takes a short subclip as input and outputs the corresponding spatial-temporal tube masks. To enhance the modeling of cross-tube relationships, we propose an effective way to perform tube-level linking via attention along the queries. In addition, we introduce temporal contrastive learning to instance-wise discriminative features for tube-level association. Our approach offers flexibility and efficiency for both short and long video inputs, as the length of each subclip can be varied according to the needs of datasets or scenarios. Tube-Link outperforms existing specialized architectures by a significant margin on five video segmentation datasets. Specifically, it achieves almost 13% relative improvements on VIPSeg and 4% improvements on KITTI-STEP over the strong baseline Video K-Net. When using a ResNet50 backbone on Youtube-VIS-2019 and 2021, Tube-Link boosts IDOL by 3% and 4%, respectively. Code will be available.
Abstract (translated)
视频分割的目标是在各种不同的场景中准确地分割和跟踪每个像素。在本文中,我们介绍了 Tube-Link,一个多功能框架,以统一架构解决了视频分割多个核心任务。我们的框架是近在线方法,以短片段作为输入并输出相应的空间-时间 tube 掩码。为了提高交叉tube关系建模,我们提出了一种有效的方法,通过注意力在查询沿线进行 tube 级别链接。此外,我们引入了时间对比学习,以实例wise 区分性的 tube 级别特征。我们的方法提供了长短视频输入的灵活和效率,因为每个短片段的长度可以根据数据集或场景的需求进行调整。 Tube-Link 在五个视频分割数据集上比现有专门的架构表现更好,具体来说,它几乎实现了 VIP Segment 的相对改进以及 KITTI-Step 的 4% 改进,在使用ResNet50作为 Youtube-VIS-2019和2021的基线视频K-Net时, Tube-Link分别提高了DOLbyby 3% 和 4%。代码将可用。
URL
https://arxiv.org/abs/2303.12782