Abstract
Mixture-of-Experts (MoE) models enable sparse expert activation, meaning that only a subset of the model's parameters is used during each inference. However, to translate this sparsity into practical performance, an expert caching mechanism is required. Previous works have proposed hardware-centric caching policies, but how these various caching policies interact with each other and different hardware specification remains poorly understood. To address this gap, we develop \textbf{SpecMD}, a standardized framework for benchmarking ad-hoc cache policies on various hardware configurations. Using SpecMD, we perform an exhaustive benchmarking of several MoE caching strategies, reproducing and extending prior approaches in controlled settings with realistic constraints. Our experiments reveal that MoE expert access is not consistent with temporal locality assumptions (e.g LRU, LFU). Motivated by this observation, we propose \textbf{Least-Stale}, a novel eviction policy that exploits MoE's predictable expert access patterns to reduce collision misses by up to $85\times$ over LRU. With such gains, we achieve over $88\%$ hit rates with up to $34.7\%$ Time-to-first-token (TTFT) reduction on OLMoE at only $5\%$ or $0.6GB$ of VRAM cache capacity.
Abstract (translated)
混合专家（Mixture-of-Experts, MoE）模型通过稀疏激活专家机制实现高效运行，这意味着在每次推理过程中仅使用了部分参数。然而，为了将这种稀疏性转化为实际性能提升，需要有效的专家缓存策略来支持。以往的研究工作提出了基于硬件的缓存政策，但这些不同缓存策略如何相互作用以及它们与各种硬件规格的关系仍不清楚。为了解决这一知识空白，我们开发了**SpecMD**，这是一个标准化框架，用于在不同的硬件配置上对自定义缓存策略进行基准测试。 利用SpecMD，我们在具有现实约束条件的控制环境中对几种MoE缓存策略进行了详尽的基准测试，并复制和扩展了先前的方法。我们的实验结果表明，MoE专家访问与时间局部性假设（如LRU、LFU）不符。基于这一观察，我们提出了**Least-Stale**，这是一种新的驱逐政策，利用MoE可预测的专家访问模式来减少冲突丢失，相比LRU最多可以提高85倍。 通过这种改进，我们在使用仅占5%或0.6GB VRAM缓存容量的情况下，在OLMoE模型上实现了超过88%的命中率，并且时间到第一个标记（Time-to-first-token, TTFT）减少了34.7%。
URL
https://arxiv.org/abs/2602.03921
