Abstract
We introduce Hibiki, a decoder-only model for simultaneous speech translation. Hibiki leverages a multistream language model to synchronously process source and target speech, and jointly produces text and audio tokens to perform speech-to-text and speech-to-speech translation. We furthermore address the fundamental challenge of simultaneous interpretation, which unlike its consecutive counterpart, where one waits for the end of the source utterance to start translating, adapts its flow to accumulate just enough context to produce a correct translation in real-time, chunk by chunk. To do so, we introduce a weakly-supervised method that leverages the perplexity of an off-the-shelf text translation system to identify optimal delays on a per-word basis and create aligned synthetic data. After supervised training, Hibiki performs adaptive, simultaneous speech translation with vanilla temperature sampling. On a French-English simultaneous speech translation task, Hibiki demonstrates state-of-the-art performance in translation quality, speaker fidelity and naturalness. Moreover, the simplicity of its inference process makes it compatible with batched translation and even real-time on-device deployment. We provide examples as well as models and inference code.
Abstract (translated)
我们介绍了一种名为Hibiki的解码器模型，用于实时语音翻译。Hibiki利用一个多流语言模型同步处理源语音和目标语音，并同时生成文本和音频令牌以执行语音到文本以及语音到语音的翻译。此外，我们还解决了实时口译的基本挑战，不同于连续口译（后者需要等待源语句结束才开始翻译），实时口译必须根据积累足够的上下文来逐块地产生正确的实时翻译进行调整。 为了实现这一目标，我们引入了一种弱监督方法，利用现成的文本翻译系统的困惑度(perplexity)来识别每个单词的最佳延迟，并创建对齐的人工合成数据。经过有监督训练后，Hibiki能够使用简单的温度抽样进行自适应、实时语音翻译。在法语-英语的实时语音翻译任务中，Hibiki展示了业界领先的翻译质量、说话人真实性和自然度。 此外，其推理过程的简洁性使其与批处理翻译甚至设备上的实时部署兼容。我们提供了示例模型以及推理代码。
URL
https://arxiv.org/abs/2502.03382
