Abstract
We study multi-modal summarization for instructional videos, whose goal is to provide users an efficient way to learn skills in the form of text instructions and key video frames. We observe that existing benchmarks focus on generic semantic-level video summarization, and are not suitable for providing step-by-step executable instructions and illustrations, both of which are crucial for instructional videos. We propose a novel benchmark for user interface (UI) instructional video summarization to fill the gap. We collect a dataset of 2,413 UI instructional videos, which spans over 167 hours. These videos are manually annotated for video segmentation, text summarization, and video summarization, which enable the comprehensive evaluations for concise and executable video summarization. We conduct extensive experiments on our collected MS4UI dataset, which suggest that state-of-the-art multi-modal summarization methods struggle on UI video summarization, and highlight the importance of new methods for UI instructional video summarization.
Abstract (translated)
我们研究了针对教学视频的多模态摘要技术，其目标是为用户提供一种通过文本说明和关键视频帧的形式来高效学习技能的方法。我们注意到现有的基准测试侧重于通用语义层面的视频摘要，并且不适合提供逐步骤的操作性指令和插图，而这对于教学视频来说至关重要。为此，我们提出了一种新的用户界面（UI）教学视频摘要的基准测试，以填补这一空白。 我们收集了一个包含2,413个UI教学视频的数据集，这些视频时长超过167小时。所有视频均经过人工标注，包括视频分割、文本总结和视频总结，这使得对简洁且可执行的视频摘要进行全面评估成为可能。 我们在所采集的MS4UI数据集上进行了广泛的实验，结果表明现有的最先进的多模态摘要方法在UI视频摘要方面存在困难，并强调了开发新的UI教学视频摘要方法的重要性。
URL
https://arxiv.org/abs/2506.12623
