Abstract
Perceiving multi-modal information and fulfilling dialogues with humans is a long-term goal of artificial intelligence. Pre-training is commonly regarded as an effective approach for multi-modal dialogue. However, due to the limited availability of multi-modal dialogue data, there is still scarce research on multi-modal dialogue pre-training. Yet another intriguing challenge emerges from the encompassing nature of multi-modal dialogue, which involves various modalities and tasks. Moreover, new forms of tasks may arise at unpredictable points in the future. Hence, it is essential for designed multi-modal dialogue models to possess sufficient flexibility to adapt to such scenarios. This paper proposes \textbf{PaCE}, a unified, structured, compositional multi-modal dialogue pre-training framework. It utilizes a combination of several fundamental experts to accommodate multiple dialogue-related tasks and can be pre-trained using limited dialogue and extensive non-dialogue multi-modal data. Furthermore, we propose a progressive training method where old experts from the past can assist new experts, facilitating the expansion of their capabilities. Experimental results demonstrate that PaCE achieves state-of-the-art results on eight multi-modal dialog benchmarks.
Abstract (translated)
感知多模态信息并与人类进行对话是人工智能的长期目标。预处理通常被视为多模态对话的有效方法。然而，由于多模态对话数据有限，仍有关于多模态对话预处理的研究稀缺。此外，从多模态对话的全面性特性中涌现的另一个令人感兴趣的挑战，涉及各种模式和任务。此外，未来可能会出现新的任务形式，因此在不可预测的时刻，设计多模态对话模型必须具有足够的灵活性来适应这些场景。本文提出了 \textbf{PaCE}，一个统一、结构良好、合成的多模态对话预处理框架。它利用多个基本专家的组合来适应多个对话相关任务，并且可以使用有限的对话和非对话多模态数据进行预处理。此外，我们提出了一种渐进的训练方法，其中过去的专家可以协助新的专家，促进其能力扩展。实验结果显示，PaCE在八个多模态对话基准上取得了最先进的结果。
URL
https://arxiv.org/abs/2305.14839
