Abstract
Expressive speech-to-speech translation (S2ST) aims to transfer prosodic attributes of source speech to target speech while maintaining translation accuracy. Existing research in expressive S2ST is limited, typically focusing on a single expressivity aspect at a time. Likewise, this research area lacks standard evaluation protocols and well-curated benchmark datasets. In this work, we propose a holistic cascade system for expressive S2ST, combining multiple prosody transfer techniques previously considered only in isolation. We curate a benchmark expressivity test set in the TV series domain and explored a second dataset in the audiobook domain. Finally, we present a human evaluation protocol to assess multiple expressive dimensions across speech pairs. Experimental results indicate that bi-lingual annotators can assess the quality of expressive preservation in S2ST systems, and the holistic modeling approach outperforms single-aspect systems. Audio samples can be accessed through our demo webpage: this https URL.
Abstract (translated)
表达性语音到语音翻译(S2ST)的目标是将源语音的音素表达特性转移到目标语音中,同时保持翻译的准确性。目前,在表达性S2ST方面的研究有限,通常只关注一个表达特性方面。类似地,这个研究领域缺乏标准评估 protocols 和精心整理的基准数据集。在本研究中,我们提出了一个整体Cascade系统,将多个以前仅考虑独立的音素表达 Transfer 技术相结合。我们在电视剧领域创建了一个基准表达性能测试集,并在音频故事领域探索了另一个数据集。最后,我们提出了一个人类评估协议,以评估在不同 speech 配对中表达的多个表达维度。实验结果表明,双语标注员可以评估S2ST系统中表达保留的质量,整体建模方法比单一特性方法表现更好。音频样本可以通过我们的演示页面访问: this https URL。
URL
https://arxiv.org/abs/2301.10606