Abstract
Whispered speech lacks vocal fold vibration and fundamental frequency, resulting in degraded acoustic cues and making whisper-to-normal (W2N) conversion challenging, especially with limited parallel data. We propose WhispEar, a bidirectional framework based on unified semantic representations that capture speaking-mode-invariant information shared by whispered and normal speech. The framework contains both W2N and normal-to-whisper (N2W) models. Notably, the N2W model enables zero-shot pseudo-parallel whisper generation from abundant normal speech, allowing scalable data augmentation for W2N training. Increasing generated data consistently improves performance. We also release the largest bilingual (Chinese-English) whispered-normal parallel corpus to date. Experiments demonstrate that WhispEar outperforms strong baselines and benefits significantly from scalable pseudo-parallel data.
Abstract (translated)
耳语缺乏声带振动和基频，导致声学线索降级，使得从耳语转换为正常语音（W2N）变得具有挑战性，尤其是在平行数据有限的情况下。我们提出了 WhispEar，这是一个基于统一语义表示的双向框架，该框架捕捉了耳语和正常语音之间共有的说话模式不变的信息。该框架包含 W2N 和正常到耳语（N2W）模型。特别地，N2W 模型能够从大量的正常语言中零样本生成伪平行耳语数据，从而为 W2N 训练提供可扩展的数据增强支持。随着生成数据的增加，性能会持续提升。此外，我们还发布了迄今为止最大的双语（中文-英语）耳语与正常语音平行语料库。实验表明 WhispEar 超过了强大的基线模型，并且从可扩展的伪平行数据中受益匪浅。
URL
https://arxiv.org/abs/2603.08046
