Abstract
Traditional Automatic Video Dubbing (AVD) pipeline consists of three key modules, namely, Automatic Speech Recognition (ASR), Neural Machine Translation (NMT), and Text-to-Speech (TTS). Within AVD pipelines, isometric-NMT algorithms are employed to regulate the length of the synthesized output text. This is done to guarantee synchronization with respect to the alignment of video and audio subsequent to the dubbing process. Previous approaches have focused on aligning the number of characters and words in the source and target language texts of Machine Translation models. However, our approach aims to align the number of phonemes instead, as they are closely associated with speech duration. In this paper, we present the development of an isometric NMT system using Reinforcement Learning (RL), with a focus on optimizing the alignment of phoneme counts in the source and target language sentence pairs. To evaluate our models, we propose the Phoneme Count Compliance (PCC) score, which is a measure of length compliance. Our approach demonstrates a substantial improvement of approximately 36% in the PCC score compared to the state-of-the-art models when applied to English-Hindi language pairs. Moreover, we propose a student-teacher architecture within the framework of our RL approach to maintain a trade-off between the phoneme count and translation quality.
Abstract (translated)
传统自动视频配音（AVD）流程包括三个关键模块，即自动语音识别（ASR）、神经机器翻译（NMT）和文本转语音（TTS）。在AVD流程中，等距-NMT算法用于调节合成输出文本的长度。这是为了确保在配音过程后，视频和音频的对齐。之前的解决方案专注于将机器翻译模型的源语言和目标语言文本中的字符和单词对齐。然而，我们的方法旨在将等距-NMT算法的注意力放在对齐源语言和目标语言句子对中的音素计数上，因为它们与语音持续时间密切相关。在本文中，我们介绍了使用强化学习（RL）开发等距NMT系统的研究，重点优化源语言和目标语言句子对中的音素计数。为了评估我们的模型，我们提出了音素计数符合（PCC）分数，这是一种衡量长度符合的指标。我们的方法在将AHD-English和AHD-Hindi语言对应用到我们的RL方法时，PCC分数比最先进的模型大约提高了36%。此外，我们还提出了一种学生-教师架构，在我们的RL方法中保持音素计数和翻译质量之间的平衡。
URL
https://arxiv.org/abs/2403.15469
