Abstract
Visual Automatic Speech Recognition (V-ASR) is a challenging task that involves interpreting spoken language solely from visual information, such as lip movements and facial expressions. This task is notably challenging due to the absence of auditory cues and the visual ambiguity of phonemes that exhibit similar visemes-distinct sounds that appear identical in lip motions. Existing methods often aim to predict words or characters directly from visual cues, but they commonly suffer from high error rates due to viseme ambiguity and require large amounts of pre-training data. We propose a novel phoneme-based two-stage framework that fuses visual and landmark motion features, followed by an LLM model for word reconstruction to address these challenges. Stage 1 consists of V-ASR, which outputs the predicted phonemes, thereby reducing training complexity. Meanwhile, the facial landmark features address speaker-specific facial characteristics. Stage 2 comprises an encoder-decoder LLM model, NLLB, that reconstructs the output phonemes back to words. Besides using a large visual dataset for deep learning fine-tuning, our PV-ASR method demonstrates superior performance by achieving 17.4% WER on the LRS2 and 21.0% WER on the LRS3 dataset.
Abstract (translated)
视觉自动语音识别（V-ASR）是一项具有挑战性的任务，它需要仅通过视觉信息（如唇部动作和面部表情）来解读口语。由于缺少听觉线索以及表现出类似视觉音素的发音在唇部运动中看起来相同造成的视觉模糊性，这项任务尤为困难。现有的方法通常旨在直接从视觉提示预测单词或字符，但它们往往因视觉音素的歧义而产生较高的错误率，并且需要大量的预训练数据。 我们提出了一种基于语音单元（音位）的两阶段框架，该框架融合了视觉和面部关键点运动特征，并随后使用LLM模型进行词汇重构以应对这些挑战。第一阶段包括V-ASR，输出预测的音素，从而减少训练复杂度。与此同时，面部关键点特征处理特定说话人的面部特性。第二阶段包含一个编码器-解码器LLM模型NLLB，用于将输出的音素重新构建成单词。 除了使用大规模视觉数据集进行深度学习微调外，我们的PV-ASR方法通过在LRS2和LRS3数据集中分别实现了17.4%和21.0%的词错误率（WER），展示了优于现有技术的表现。
URL
https://arxiv.org/abs/2507.18863
