Abstract
Mixture-of-Experts (MoE) models achieve efficient scaling through sparse expert activation, but often suffer from suboptimal routing decisions due to distribution shifts in deployment. While existing test-time adaptation methods could potentially address these issues, they primarily focus on dense models and require access to external data, limiting their practical applicability to MoE architectures. However, we find that, instead of relying on reference data, we can optimize MoE expert selection on-the-fly based only on input context. As such, we propose \textit{a data-free, online test-time framework} that continuously adapts MoE routing decisions during text generation without external supervision or data. Our method cycles between two phases: During the prefill stage, and later in regular intervals, we optimize the routing decisions of the model using self-supervision based on the already generated sequence. Then, we generate text as normal, maintaining the modified router until the next adaption. We implement this through lightweight additive vectors that only update router logits in selected layers, maintaining computational efficiency while preventing over-adaptation. The experimental results show consistent performance gains on challenging reasoning tasks while maintaining robustness to context shifts. For example, our method achieves a 5.5\% improvement on HumanEval with OLMoE. Furthermore, owing to its plug-and-play property, our method naturally complements existing test-time scaling techniques, e.g., achieving 6\% average gains when incorporated with self-consistency on DeepSeek-V2-Lite.
Abstract (translated)
混合专家（Mixture-of-Experts，MoE）模型通过稀疏的专家激活实现了高效的扩展能力，但在部署过程中由于分布偏移往往会导致次优的路由决策。尽管现有的测试时间适应方法可以解决这些问题，但它们主要针对密集型模型，并且需要外部数据访问权限，这限制了其在MoE架构中的实际应用性。然而，我们发现，可以通过仅基于输入上下文来优化MoE专家选择，而无需依赖参考数据。因此，我们提出了一种“无数据、在线测试时间框架”，该框架可以在文本生成过程中持续调整MoE的路由决策，且不需外部监督或数据。 我们的方法在两个阶段之间循环进行：在预填充阶段以及之后的定期间隔内，模型使用自监督来优化其基于已经生成序列的路由决策。然后，我们像正常一样生成文本，并保持修改后的路由器直到下一次适应为止。通过轻量级加性向量实现这一过程，这些向量仅更新选定层中的路由器日志几率（logits），从而在防止过度调整的同时维持计算效率。 实验结果显示，在具有挑战性的推理任务上性能得到了持续的提升，并且能够保持对上下文变化的强大鲁棒性。例如，在使用OLMoE模型时，我们的方法实现了5.5%的人类评估改进。此外，由于其即插即用特性，该方法自然地补充了现有的测试时间扩展技术，如在与深度搜索V2 Lite的自我一致性结合使用时可以获得平均6%的性能提升。
URL
https://arxiv.org/abs/2510.14853
