Abstract
Large Language Models (LLMs) have achieved significant success in various natural language processing tasks, but how wireless communications can support LLMs has not been extensively studied. In this paper, we propose a wireless distributed LLMs paradigm based on Mixture of Experts (MoE), named WDMoE, deploying LLMs collaboratively across edge servers of base station (BS) and mobile devices in the wireless communications system. Specifically, we decompose the MoE layer in LLMs by deploying the gating network and the preceding neural network layer at BS, while distributing the expert networks across the devices. This arrangement leverages the parallel capabilities of expert networks on distributed devices. Moreover, to overcome the instability of wireless communications, we design an expert selection policy by taking into account both the performance of the model and the end-to-end latency, which includes both transmission delay and inference delay. Evaluations conducted across various LLMs and multiple datasets demonstrate that WDMoE not only outperforms existing models, such as Llama 2 with 70 billion parameters, but also significantly reduces end-to-end latency.
Abstract (translated)
大型语言模型（LLMs）在各种自然语言处理任务中取得了显著的成功，但如何将无线通信支持LLMs进行广泛研究仍然是一个重要的挑战。在本文中，我们提出了一个基于Mixture of Experts（MoE）的无线分布式LLM范例，名为WDMoE，在无线通信系统的基站（BS）和移动设备上协同部署LLM。具体来说，我们在LLMs的MoE层中通过在BS上部署 gate network 和先前的神经网络层，同时将专家网络分布在设备上。这种安排利用了分布式设备上专家网络的并行能力。此外，为了克服无线通信的不稳定性，我们设计了一个专家选择策略，考虑了模型的性能和端到端延迟，包括传输延迟和推理延迟。在各种LLM和多个数据集上进行的评估证明，WDMoE不仅超越了具有700亿参数的现有模型，如Llama 2，而且显著减少了端到端延迟。
URL
https://arxiv.org/abs/2405.03131
