Abstract
Publishing open-source academic video recordings is an emergent and prevalent approach to sharing knowledge online. Such videos carry rich multimodal information including speech, the facial and body movements of the speakers, as well as the texts and pictures in the slides and possibly even the papers. Although multiple academic video datasets have been constructed and released, few of them support both multimodal content recognition and understanding tasks, which is partially due to the lack of high-quality human annotations. In this paper, we propose a novel multimodal, multigenre, and multipurpose audio-visual academic lecture dataset (M$^3$AV), which has almost 367 hours of videos from five sources covering computer science, mathematics, and medical and biology topics. With high-quality human annotations of the spoken and written words, in particular high-valued name entities, the dataset can be used for multiple audio-visual recognition and understanding tasks. Evaluations performed on contextual speech recognition, speech synthesis, and slide and script generation tasks demonstrate that the diversity of M$^3$AV makes it a challenging dataset.
Abstract (translated)
发布开源学术视频是一种新兴且普遍的知识共享在线方法。这些视频携带丰富的多模态信息，包括演讲者的面部和身体动作，以及幻灯片上的文本和图片，甚至可能还有论文。尽管已经构建和发布了许多学术视频数据集，但大多数都不支持多模态内容识别和理解任务，这部分原因是缺乏高质量的人类注释。在本文中，我们提出了一个名为M$^3$AV的多模态、多流派和多功能音频-视觉学术讲座数据集（M$^3$AV），它包含了来自五个来源的超过367小时的视频，涵盖了计算机科学、数学和医学生物主题。由于高质量的人类注释，特别是高价值的名词，这个数据集可以用于多个音频-视觉识别和理解任务。在上下文语音识别、语音合成和幻灯和脚本生成任务的评估表明，M$^3$AV的多样性使得它成为一个具有挑战性的数据集。
URL
https://arxiv.org/abs/2403.14168
