Abstract
Learning from videos is an emerging research area that enables robots to acquire skills from human demonstrations, such as procedural videos. To do this, video-language models must be able to obtain structured understandings, such as the temporal segmentation of a demonstration into sequences of actions and skills, and to generalize the understandings to novel domains. In pursuit of this goal, we introduce Spacewalk-18, a benchmark containing two tasks: (1) step recognition and (2) intra-video retrieval over a dataset of temporally segmented and labeled tasks in International Space Station spacewalk recordings. In tandem, the two tasks quantify a model's ability to make use of: (1) out-of-domain visual information; (2) a high temporal context window; and (3) multimodal (text + video) domains. This departs from existing benchmarks for procedural video understanding, which typically deal with short context lengths and can be solved with a single modality. Spacewalk-18, with its inherent multimodal and long-form complexity, exposes the high difficulty of task recognition and segmentation. We find that state-of-the-art methods perform poorly on our benchmark, demonstrating that the goal of generalizable procedural video understanding models is far out and underscoring the need to develop new approaches to these tasks. Data, model, and code will be publicly released.
Abstract (translated)
学习视频是一个新兴的研究领域，它使机器人能够从人类演示中获取技能，例如程序视频。为此，视频语言模型必须能够获得结构化的理解，例如将演示的时间分割为一系列动作和技能的时间序列，并将理解泛化到新的领域。为了实现这一目标，我们引入了Spacewalk-18，一个包含两个任务的基准：（1）步骤识别；（2）在国际空间站空间行走录音数据集中的视频内检索。与此同时，这两个任务衡量了模型利用以下能力：1）跨域视觉信息；2）高时间上下文窗口；3）多模态（文本+视频）领域。这不同于现有程序视频理解的基准，通常处理短上下文长度，并且可以用单一模式解决。Spacewalk-18，由于其固有的多模态和长形式复杂性，揭示了任务识别和分割的高难度。我们发现，最先进的方法在我们的基准上表现不佳，这表明泛化程序视频理解模型的目标是远远超出了，并强调了开发新的方法来解决这些任务的需求。数据、模型和代码将公开发布。
URL
https://arxiv.org/abs/2311.18773
