Abstract
Video is transforming education with online courses and recorded lectures supplementing and replacing classroom teaching. Recent research has focused on enhancing information retrieval for video lectures with advanced navigation, searchability, summarization, as well as question answering chatbots. Visual elements like tables, charts, and illustrations are central to comprehension, retention, and data presentation in lecture videos, yet their full potential for improving access to video content remains underutilized. A major factor is that accurate automatic detection of visual elements in a lecture video is challenging; reasons include i) most visual elements, such as charts, graphs, tables, and illustrations, are artificially created and lack any standard structure, and ii) coherent visual objects may lack clear boundaries and may be composed of connected text and visual components. Despite advancements in deep learning based object detection, current models do not yield satisfactory performance due to the unique nature of visual content in lectures and scarcity of annotated datasets. This paper reports on a transfer learning approach for detecting visual elements in lecture video frames. A suite of state of the art object detection models were evaluated for their performance on lecture video datasets. YOLO emerged as the most promising model for this task. Subsequently YOLO was optimized for lecture video object detection with training on multiple benchmark datasets and deploying a semi-supervised auto labeling strategy. Results evaluate the success of this approach, also in developing a general solution to the problem of object detection in lecture videos. Paper contributions include a publicly released benchmark of annotated lecture video frames, along with the source code to facilitate future research.
Abstract (translated)
视频正在通过在线课程和录制的讲座改变教育，这些内容正补充甚至取代传统的课堂教学。近期的研究重点在于利用高级导航、搜索能力、摘要生成以及问答聊天机器人来增强对视频讲座的信息检索功能。视觉元素如表格、图表和插图在讲义视频的理解、记忆以及数据展示中至关重要，但它们在整个视频内容的访问改善方面的潜力尚未得到充分利用。一个主要因素是准确自动检测讲义视频中的视觉元素具有挑战性；原因包括：一）大多数视觉元素（例如图表、图形、表格和插图）都是人工创建且缺乏任何标准结构；二）连贯的视觉对象可能没有明确边界，并由连接的文本和视觉组件组成。尽管基于深度学习的对象检测技术已经取得了进展，但由于讲义视频中视觉内容的独特性和标注数据集的稀缺性，现有模型的表现并不令人满意。 本文报道了一种用于识别讲座视频帧中的视觉元素的迁移学习方法。评估了一系列最先进的对象检测模型在讲座视频数据集上的表现，YOLO（You Only Look Once）被认为是最有前途的模型。随后，通过对多个基准数据集进行训练并部署半监督自动标注策略对YOLO进行了优化以适应讲座视频的对象检测需求。实验结果评估了这种方法的成功，并为解决讲义视频中的对象检测问题开发了一种通用解决方案提供了支持。 本文贡献包括公开发布了一个包含注释的讲义视频帧的基准测试，以及源代码，以促进未来的相关研究。
URL
https://arxiv.org/abs/2506.21903
