Abstract
A rising interest in the modality extension of foundation language models warrants discussion on the most effective, and efficient, multimodal training approach. This work focuses on neural machine translation (NMT) and proposes a joint multimodal training regime of Speech-LLM to include automatic speech translation (AST). We investigate two different foundation model architectures, decoder-only GPT and encoder-decoder T5, extended with Canary-1B's speech encoder. To handle joint multimodal training, we propose a novel training framework called EMMeTT. EMMeTT improves training efficiency with the following: balanced sampling across languages, datasets, and modalities; efficient sequential data iteration; and a novel 2D bucketing scheme for multimodal data, complemented by a batch size optimizer (OOMptimizer). We show that a multimodal training consistently helps with both architectures. Moreover, SALM-T5 trained with EMMeTT retains the original NMT capability while outperforming AST baselines on four-language subsets of FLORES and FLEURS. The resultant Multimodal Translation Model produces strong text and speech translation results at the same time.
Abstract (translated)
对基础语言模型扩展模式的日益关注需要讨论最有效、最有效的多模态训练方法。本文重点关注神经机器翻译（NMT），并提出了一个结合Speech-LLM的联合多模态训练方案，包括自动语音翻译（AST）。我们研究了两种不同的基础模型架构：仅解码器GPT和编码器-解码器T5，用Canary-1B的语音编码器进行了扩展。为了处理联合多模态训练，我们提出了一个名为EMMeTT的新型训练框架。EMMeTT通过以下方式提高了训练效率：平衡跨语言、数据集和模态的采样；高效的序列数据迭代；以及一种新颖的2D桶位方案，由批量大小优化器（OOMptimizer）补充。我们证明了多模态训练在两种架构中都是有效的。此外，使用EMMeTT训练的SALM-T5保留了原始NMT能力，同时在FLORES和FLEURS的四个语言子集上优于AST基线。所得到的跨模态翻译模型可以在同一时间产生强文本和语音翻译结果。
URL
https://arxiv.org/abs/2409.13523
