Abstract
Sparse Mixtures of Experts (SMoE) scales model capacity without significant increases in training and inference costs, but exhibits the following two issues: (1) Low expert activation, where only a small subset of experts are activated for optimization. (2) Lacking fine-grained analytical capabilities for multiple semantic concepts within individual tokens. We propose Multi-Head Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each token into multiple sub-tokens. These sub-tokens are then assigned to and processed by a diverse set of experts in parallel, and seamlessly reintegrated into the original token form. The multi-head mechanism enables the model to collectively attend to information from various representation spaces within different experts, while significantly enhances expert activation, thus deepens context understanding and alleviate overfitting. Moreover, our MH-MoE is straightforward to implement and decouples from other SMoE optimization methods, making it easy to integrate with other SMoE models for enhanced performance. Extensive experimental results across three tasks: English-focused language modeling, Multi-lingual language modeling and Masked multi-modality modeling tasks, demonstrate the effectiveness of MH-MoE.
Abstract (translated)
Sparse Mixtures of Experts (SMoE)是一种在不显著增加训练和推理成本的情况下扩展模型容量的方法，但存在以下两个问题：（1）专家激活度低，仅激活一小部分专家进行优化；（2）对多个语义概念的细粒度分析能力不足。我们提出了多头专家混合专家（MH-MoE）方法，它采用一个多头机制将每个词划分为多个子词。这些子词随后被分配给多个专家并并行处理，无缝地重新整合到原始词形式中。多头机制使模型能够集体关注不同专家对各个表示空间的信息，从而显著增强专家激活，加深上下文理解，缓解过拟合。此外，我们的MH-MoE易于实现，与其他SMoE优化方法解耦，容易与其他SMoE模型集成以提高性能。在三个任务（英语关注语言建模、多语言语言建模和遮罩多模态建模）上的实验结果表明，MH-MoE的有效性。
URL
https://arxiv.org/abs/2404.15045
