Abstract
Open-world video instance segmentation is an important video understanding task. Yet most methods either operate in a closed-world setting, require an additional user-input, or use classic region-based proposals to identify never before seen objects. Further, these methods only assign a one-word label to detected objects, and don't generate rich object-centric descriptions. They also often suffer from highly overlapping predictions. To address these issues, we propose Open-World Video Instance Segmentation and Captioning (OW-VISCap), an approach to jointly segment, track, and caption previously seen or unseen objects in a video. For this, we introduce open-world object queries to discover never before seen objects without additional user-input. We generate rich and descriptive object-centric captions for each detected object via a masked attention augmented LLM input. We introduce an inter-query contrastive loss to ensure that the object queries differ from one another. Our generalized approach matches or surpasses state-of-the-art on three tasks: open-world video instance segmentation on the BURST dataset, dense video object captioning on the VidSTG dataset, and closed-world video instance segmentation on the OVIS dataset.
Abstract (translated)
开放世界视频实例分割是一个重要的视频理解任务。然而，大多数方法要么在关闭的世界设置中操作，需要额外的用户输入，要么使用基于经典区域的提议来识别从未见过的物体。此外，这些方法只给检测到的物体分配一个单词标签，并且不生成丰富的物体中心描述。它们还经常遭受高度重叠预测的问题。为了解决这些问题，我们提出了Open-World Video Instance Segmentation and Captioning (OW-VISCap)，一种在视频中共同分割、跟踪和捕获之前见过的或未见过的物体的方法。为此，我们引入了开放世界物体查询来发现没有额外用户输入的从未见过的物体。我们通过遮罩注意增强LLM输入为每个检测到的物体生成丰富而描述性的物体中心描述。我们引入了跨查询对比损失来确保物体查询彼此不同。我们的通用方法在三个任务上都超越了最先进的水平：在BURST数据集上的开放世界视频实例分割，VidSTG数据集上的密集视频物体注释和OVIS数据集上的关闭世界视频实例分割。
URL
https://arxiv.org/abs/2404.03657
