Abstract
This study proposes a novel approach to using TV subtitles within a weakly supervised (WS) Automatic Speech Recognition (ASR) framework. Although TV subtitles are readily available, their imprecise alignment with corresponding audio limits their applicability as supervised targets for verbatim transcription. Rather than using subtitles as direct supervision signals, our method reimagines them as context-rich prompts. This design enables the model to handle discrepancies between spoken audio and subtitle text. Instead, generated pseudo transcripts become the primary targets, with subtitles acting as guiding cues for iterative refinement. To further enhance the process, we introduce a weighted attention mechanism that emphasizes relevant subtitle tokens during inference. Our experiments demonstrate significant improvements in transcription accuracy, highlighting the effectiveness of the proposed method in refining transcripts. These enhanced pseudo-labeled datasets provide high-quality foundational resources for training robust ASR systems.
Abstract (translated)
这项研究提出了一种在弱监督（WS）自动语音识别（ASR）框架中使用电视字幕的创新方法。尽管电视字幕很容易获取，但由于与相应音频的对齐不够精确，它们作为逐字转录的监督目标的应用性受到限制。我们的方法不将字幕直接用作监督信号，而是将其重新想象为富含上下文信息的提示。这种设计使模型能够处理语音和字幕文本之间的差异。相反，生成的伪转录成为主要的目标，而字幕则作为迭代改进过程中的指导线索发挥作用。 为了进一步增强这一流程，我们引入了一种加权注意力机制，在推断过程中强调相关的字幕标记。我们的实验表明，这种方法在转写准确性方面取得了显著改善，突显了所提出方法的有效性。通过这种方式生成的增强伪标注数据集为训练稳健的ASR系统提供了高质量的基础资源。
URL
https://arxiv.org/abs/2509.04491
