Abstract
The sparse Mixture-of-Experts (MoE) has achieved significant progress for neural machine translation (NMT). However, there exist two limitations in current MoE solutions which may lead to sub-optimal performance: 1) they directly use the task knowledge of NMT into MoE (\emph{e.g.}, domain/linguistics-specific knowledge), which are generally unavailable at practical application and neglect the naturally grouped domain/linguistic properties; 2) the expert selection only depends on the localized token representation without considering the context, which fully grasps the state of each token in a global view. To address the above limitations, we propose THOR-MoE via arming the MoE with hierarchical task-guided and context-responsive routing policies. Specifically, it 1) firstly predicts the domain/language label and then extracts mixed domain/language representation to allocate task-level experts in a hierarchical manner; 2) injects the context information to enhance the token routing from the pre-selected task-level experts set, which can help each token to be accurately routed to more specialized and suitable experts. Extensive experiments on multi-domain translation and multilingual translation benchmarks with different architectures consistently demonstrate the superior performance of THOR-MoE. Additionally, the THOR-MoE operates as a plug-and-play module compatible with existing Top-$k$~\cite{shazeer2017} and Top-$p$~\cite{huang-etal-2024-harder} routing schemes, ensuring broad applicability across diverse MoE architectures. For instance, compared with vanilla Top-$p$~\cite{huang-etal-2024-harder} routing, the context-aware manner can achieve an average improvement of 0.75 BLEU with less than 22\% activated parameters on multi-domain translation tasks.
Abstract (translated)
稀疏的专家混合（Mixture-of-Experts，MoE）在神经机器翻译（NMT）方面取得了显著进展。然而，当前的MoE解决方案存在两个可能导致性能次优的问题：1) 它们直接将NMT的任务知识（例如，特定领域的或语言学的知识）用于MoE中，这些信息在实际应用中通常不可用，并且忽视了自然分组的领域/语言特性；2) 专家选择仅依赖于局部标记表示，而不考虑上下文，这无法从全局视角全面掌握每个标记的状态。为了解决上述问题，我们提出了通过配备层次化任务引导和上下文响应路由策略来增强MoE能力的THOR-MoE模型。 具体来说，1）该方法首先预测领域/语言标签，然后提取混合领域的表示以分层方式分配任务级别的专家；2）注入上下文信息以增强预选的任务级专家集合中的标记路由，这可以帮助每个标记更准确地路由到更加专业化且合适的专家。在不同架构上的多域翻译和跨语言翻译基准测试中进行了广泛的实验，结果一致表明THOR-MoE具有卓越的性能。 此外，THOR-MoE作为即插即用模块与现有的Top-$k$~\cite{shazeer2017} 和 Top-$p$~\cite{huang-etal-2024-harder} 路由方案兼容，在广泛的MoE架构中确保了广泛应用性。例如，与原始的Top-$p$~\cite{huang-etal-2024-harder} 路由相比，上下文感知方式在多域翻译任务上平均提高了0.75个BLEU分数，并且激活参数少于22%。 这段文字详细介绍了THOR-MoE模型如何改进现有的MoE架构以适应神经机器翻译的需求。通过引入层次化的任务引导和增强的上下文响应机制，该模型不仅能够更好地利用语言学和领域知识，还能更精确地为每个标记选择合适的专家，从而提高翻译质量。实验结果表明了THOR-MoE在多域与跨语言翻译中的优越性能，并且其灵活的设计使其适用于多种路由方案，增加了实际应用中的通用性和效率。
URL
https://arxiv.org/abs/2505.14173
