Abstract
Mixture-of-Experts (MoE) layers scale transformers by routing tokens to a sparse subset of feed-forward experts. Token-level routing, however, assigns an entire semantic spectrum to each expert, creating capacity bottlenecks, load-balancing pathologies, and limited specialization. We introduce SliceMoE, an architecture that routes contiguous slices of a token's hidden vector. A d-dimensional embedding is partitioned into S slices, and for each slice, a lightweight shared router predicts the top-k experts. Experts operate on their assigned slices independently, and outputs are reassembled, maintaining per-token FLOP efficiency. Because slices from different tokens interleave within an expert, utilization is naturally smoother. We propose a slice-level capacity loss, cross-slice dropout, and efficient fused batched GEMM kernels. Experiments on WikiText-103 language modeling, WMT En-De translation, and three text-classification datasets show SliceMoE attains up to 1.7x faster inference than dense baselines, 12 to 18 percent lower perplexity than parameter-matched token-MoE, and improved expert balance, with interpretable expertise over syntactic versus semantic subspaces.
Abstract (translated)
混合专家（Mixture-of-Experts，MoE）层通过将令牌路由到一组稀疏的前馈专家子集来扩展转换器模型。然而，基于令牌级别的路由为每个专家分配了一个完整的语义频谱，这会导致容量瓶颈、负载均衡问题以及专业化的限制。为了应对这些问题，我们引入了SliceMoE架构，该架构根据连续的片段将令牌隐藏向量路由到不同的专家。 在SliceMoE中，一个d维嵌入被划分为S个片段（slices）。对于每一个片段，一个轻量级共享路由器预测出前k个专家。每个专家独立地操作它们分配到的片段，并且输出会被重新组装起来以保持每令牌的计算效率不变。由于来自不同令牌的片段在一个专家内交错进行处理，因此利用率会更加平滑。 我们提出了一种基于切片级别的容量损失、跨切片dropout策略以及高效的批矩阵乘法（GEMM）核来改进模型性能和效率。实验结果表明，在WikiText-103语言建模任务、WMT英德翻译任务及三个文本分类数据集上，与密集基线相比，SliceMoE实现了高达1.7倍的推理加速；与参数匹配的令牌MoE相比，困惑度降低了12%至18%，同时改进了专家间的平衡性，并且使得每个专家专注于语法或语义子空间中更具可解释性的专业知识。
URL
https://arxiv.org/abs/2510.04286
