Abstract
Lip-to-speech (L2S) synthesis for Mandarin is a significant challenge, hindered by complex viseme-to-phoneme mappings and the critical role of lexical tones in intelligibility. To address this issue, we propose Lexical Tone-Aware Lip-to-Speech (LTA-L2S). To tackle viseme-to-phoneme complexity, our model adapts an English pre-trained audio-visual self-supervised learning (SSL) model via a cross-lingual transfer learning strategy. This strategy not only transfers universal knowledge learned from extensive English data to the Mandarin domain but also circumvents the prohibitive cost of training such a model from scratch. To specifically model lexical tones and enhance intelligibility, we further employ a flow-matching model to generate the F0 contour. This generation process is guided by ASR-fine-tuned SSL speech units, which contain crucial suprasegmental information. The overall speech quality is then elevated through a two-stage training paradigm, where a flow-matching postnet refines the coarse spectrogram from the first stage. Extensive experiments demonstrate that LTA-L2S significantly outperforms existing methods in both speech intelligibility and tonal accuracy.
Abstract (translated)
唇部到语音（Lip-to-Speech，简称L2S）合成对于普通话来说是一个重大的挑战，这主要是由于复杂的唇形单元至音素映射以及声调在可懂度中的关键作用。为了解决这一问题，我们提出了一个考虑到词汇音调的唇部到语音合成方法（Lexical Tone-Aware Lip-to-Speech, LTA-L2S）。为了应对唇形单元至音素复杂性的挑战，我们的模型通过跨语言迁移学习策略来调整一个预先训练好的英语音频-视觉自监督学习(SSL)模型。这一策略不仅将从大量英语数据中习得的通用知识转移到普通话领域，而且还避免了从头开始训练此类模型所需的高昂成本。 为了具体建模词汇音调并增强可懂度，我们进一步使用了一个流匹配（flow-matching）模型来生成基频（F0）轮廓。这一生成过程受经过语音识别细化后的SSL语音单元指导，这些单位包含至关重要的超分段信息。通过一个两阶段的训练范式，整体语音质量得到了提升，在此过程中，流匹配后处理网络在第一阶段生成粗略光谱图之后对其进行精细调整。 广泛的实验表明，LTA-L2S方法在语音可懂度和声调准确性方面显著优于现有技术。
URL
https://arxiv.org/abs/2509.25670
