Abstract
Previous works on video object segmentation (VOS) are trained on densely annotated videos. Nevertheless, acquiring annotations in pixel level is expensive and time-consuming. In this work, we demonstrate the feasibility of training a satisfactory VOS model on sparsely annotated videos-we merely require two labeled frames per training video while the performance is sustained. We term this novel training paradigm as two-shot video object segmentation, or two-shot VOS for short. The underlying idea is to generate pseudo labels for unlabeled frames during training and to optimize the model on the combination of labeled and pseudo-labeled data. Our approach is extremely simple and can be applied to a majority of existing frameworks. We first pre-train a VOS model on sparsely annotated videos in a semi-supervised manner, with the first frame always being a labeled one. Then, we adopt the pre-trained VOS model to generate pseudo labels for all unlabeled frames, which are subsequently stored in a pseudo-label bank. Finally, we retrain a VOS model on both labeled and pseudo-labeled data without any restrictions on the first frame. For the first time, we present a general way to train VOS models on two-shot VOS datasets. By using 7.3% and 2.9% labeled data of YouTube-VOS and DAVIS benchmarks, our approach achieves comparable results in contrast to the counterparts trained on fully labeled set. Code and models are available at this https URL.
Abstract (translated)
以往的视频对象分割工作都是在稠密标注的视频上进行训练。然而,在这项工作中,我们证明了在稠密标注的视频上训练令人满意的VOS模型的可行性。我们只需要每训练视频标注两个帧,而表现却能持续维持。我们称之为“两帧视频对象分割”或“两帧VOS”,这是一种新的训练范式。其基本思想是在训练期间为未标注帧生成伪标签,并优化基于标注和伪标签数据的模型。我们的方法和思路非常简单,可以应用于大多数现有的框架。我们首先在未标注的视频上采用半监督的方式预训练VOS模型,而第一个帧总是标注的。然后,我们采用预训练的VOS模型为所有未标注帧生成伪标签,并将其存储在伪标签库中。最后,我们重新训练基于标注和伪标签数据的VOS模型,而不受第一个帧的限制。有史以来第一次,我们提出了一种通用的方法来训练两帧VOS数据集上的VOS模型。利用YouTube-VOS和 Davis基准视频的7.3%和2.9%的标注数据,我们的方法和模型与在完全标注数据集上训练的相对应。代码和模型可在该httpsURL上获取。
URL
https://arxiv.org/abs/2303.12078
