Abstract
Recent research shows a big convergence in model architecture, training objectives, and inference methods across various tasks for different modalities. In this paper, we propose VioLA, a single auto-regressive Transformer decoder-only network that unifies various cross-modal tasks involving speech and text, such as speech-to-text, text-to-text, text-to-speech, and speech-to-speech tasks, as a conditional codec language model task via multi-task learning framework. To accomplish this, we first convert all the speech utterances to discrete tokens (similar to the textual data) using an offline neural codec encoder. In such a way, all these tasks are converted to token-based sequence conversion problems, which can be naturally handled with one conditional language model. We further integrate task IDs (TID) and language IDs (LID) into the proposed model to enhance the modeling capability of handling different languages and tasks. Experimental results demonstrate that the proposed VioLA model can support both single-modal and cross-modal tasks well, and the decoder-only model achieves a comparable and even better performance than the strong baselines.
Abstract (translated)
最近的研究表明，在各种任务中，模型架构、训练目标和推断方法在多种modality的任务上都有很大的一致性。在本文中，我们提出了VioLA模型，它是一种单一的自回归Transformer解码器，通过多任务学习框架将涉及语音和文本的各种跨模态任务统一为条件codec语言模型任务。为了实现这一目标，我们首先使用在线神经网络codec编码器将所有的语音 utterances转换为离散代币(类似于文本数据)。这样，所有这些任务都被转换为代币序列转换问题，这种问题可以用一个条件语言模型自然处理。我们还将任务标识符(TID)和语言标识符(LID)集成到 proposed 模型中，以增强处理不同语言和任务的计算能力。实验结果表明，提出的VioLA模型可以同时支持单模态和跨模态任务，而解码器单独模型的表现甚至比强大的基线模型还要好。
URL
https://arxiv.org/abs/2305.16107
