Abstract
Streaming multi-talker speech translation is a task that involves not only generating accurate and fluent translations with low latency but also recognizing when a speaker change occurs and what the speaker's gender is. Speaker change information can be used to create audio prompts for a zero-shot text-to-speech system, and gender can help to select speaker profiles in a conventional text-to-speech model. We propose to tackle streaming speaker change detection and gender classification by incorporating speaker embeddings into a transducer-based streaming end-to-end speech translation model. Our experiments demonstrate that the proposed methods can achieve high accuracy for both speaker change detection and gender classification.
Abstract (translated)
流式多说话人语音翻译任务不仅要求生成准确流畅、延迟低的翻译，还需要识别说话人的变更以及判断说话人性别。说话人变更信息可用于为零样本文本转语音系统创建音频提示，而性别则有助于在传统文本转语音模型中选择合适的发音者配置文件。我们提出通过将说话人嵌入整合到基于转换器的流式端到端语音翻译模型中来解决流式说话人变更检测和性别分类问题。实验结果表明，所提出的方案能够实现高准确度的说话人变更检测和性别分类。
URL
https://arxiv.org/abs/2502.02683
