Abstract
Multimodal Large Language Models (MLLMs) have achieved considerable accuracy in Optical Character Recognition (OCR) from static images. However, their efficacy in video OCR is significantly diminished due to factors such as motion blur, temporal variations, and visual effects inherent in video content. To provide clearer guidance for training practical MLLMs, we introduce the MME-VideoOCR benchmark, which encompasses a comprehensive range of video OCR application scenarios. MME-VideoOCR features 10 task categories comprising 25 individual tasks and spans 44 diverse scenarios. These tasks extend beyond text recognition to incorporate deeper comprehension and reasoning of textual content within videos. The benchmark consists of 1,464 videos with varying resolutions, aspect ratios, and durations, along with 2,000 meticulously curated, manually annotated question-answer pairs. We evaluate 18 state-of-the-art MLLMs on MME-VideoOCR, revealing that even the best-performing model (Gemini-2.5 Pro) achieves an accuracy of only 73.7%. Fine-grained analysis indicates that while existing MLLMs demonstrate strong performance on tasks where relevant texts are contained within a single or few frames, they exhibit limited capability in effectively handling tasks that demand holistic video comprehension. These limitations are especially evident in scenarios that require spatio-temporal reasoning, cross-frame information integration, or resistance to language prior bias. Our findings also highlight the importance of high-resolution visual input and sufficient temporal coverage for reliable OCR in dynamic video scenarios.
Abstract (translated)
多模态大型语言模型（MLLM）在静态图像的光学字符识别（OCR）方面已经取得了相当高的准确度。然而，由于视频内容中存在的运动模糊、时间变化和视觉效果等因素，它们在视频OCR中的有效性显著降低。为了为训练实用的MLLM提供更清晰的指导，我们引入了MME-VideoOCR基准测试，它涵盖了视频OCR应用的各种场景。MME-VideoOCR包含10个任务类别，共计25项独立任务，并涵盖44种多样化的应用场景。这些任务不仅限于文本识别，还涉及对视频中文字内容的深入理解和推理。该基准测试包括具有不同分辨率、宽高比和时长的1,464段视频，以及精心策划并人工标注了问答配对的2,000个问题-答案对。 我们在MME-VideoOCR上评估了18种最先进的MLLM模型，结果表明即使是表现最好的模型（Gemini-2.5 Pro）也只能达到73.7%的准确率。精细分析显示，现有的MLLM在处理包含相关文本仅限于单帧或少数几帧的任务时表现出色，但在需要全面视频理解的任务上能力有限。这种局限性尤其体现在要求时空推理、跨帧信息整合或是抵抗语言先入为主偏差的应用场景中。 我们的研究结果还强调了在动态视频环境中进行可靠OCR所需高分辨率视觉输入和充足的时间覆盖的重要性。
URL
https://arxiv.org/abs/2505.21333
