Abstract
Automatic speech recognition (ASR) systems often struggle with domain-specific terminology, especially in specialized settings such as academic lectures. To address this, we define the SlideASR task, which leverages the rich visual information from presentation slides to improve transcription accuracy. Existing pipeline methods for this task tend to be complex and underperform. Although omni-modal large language models (OLLMs) provide a promising end-to-end framework, they frequently fail in practice by degenerating into simple optical character recognition (OCR) systems. To overcome this, we propose Visually-Anchored Policy Optimization (VAPO), a novel post-training method designed to control the model's reasoning process. Drawing on the Chain-of-Thought reasoning paradigm, VAPO enforces a structured "Look before Transcription" procedure using a <think><answer> format. Specifically, the model first performs OCR on the slide content within the think step, then generates the transcription by referencing this recognized visual information in the answer step. This reasoning process is optimized via reinforcement learning with four distinct rewards targeting format compliance, OCR accuracy, ASR quality, and visual anchoring consistency. To support further research, we construct SlideASR-Bench, a new entity-rich benchmark consisting of a synthetic dataset for training and testing, and a challenging real-world set for evaluation. Extensive experiments demonstrate that VAPO significantly improves recognition of domain-specific terms, establishing an effective end-to-end paradigm for SlideASR.
Abstract (translated)
自动语音识别（ASR）系统在处理特定领域的术语时常常遇到困难，尤其是在学术讲座等专业环境中。为了解决这个问题，我们定义了SlideASR任务，该任务利用演示幻灯片中的丰富视觉信息来提高转录准确性。现有的管道方法对于此任务通常很复杂且表现不佳。虽然全能型大型语言模型（OLLM）提供了一个有前途的端到端框架，但它们在实践中往往会退化为简单的光学字符识别（OCR）系统而失败。为了克服这个问题，我们提出了视觉锚定策略优化（VAPO），这是一种新颖的训练后方法，旨在控制模型的推理过程。借鉴链式思维推理范式，VAPO通过采用<think><answer>格式强制执行结构化的“先查看再转录”程序。具体来说，在思考步骤中，模型首先对幻灯片内容进行OCR处理，然后在回答步骤中引用已识别的视觉信息生成转录文本。该推理过程通过强化学习优化，并针对格式合规性、OCR准确性、ASR质量和视觉锚定一致性设置了四种不同的奖励。 为了支持进一步的研究，我们构建了SlideASR-Bench，这是一个新的实体丰富的基准测试，包括用于训练和测试的人工合成数据集以及一个具有挑战性的现实世界评估数据集。广泛的实验表明，VAPO显著提高了特定领域术语的识别能力，并为SlideASR建立了有效的端到端范式。
URL
https://arxiv.org/abs/2510.08618
