Abstract
On-device large language models (LLMs) are catalyzing novel mobile applications such as UI task automation and personalized email auto-reply, without giving away users' private data. However, on-device LLMs still suffer from unacceptably long inference latency, especially the time to first token (prefill stage) due to the need of long context for accurate, personalized content generation, as well as the lack of parallel computing capacity of mobile CPU/GPU. To enable practical on-device LLM, we present mllm-NPU, the first-of-its-kind LLM inference system that efficiently leverages on-device Neural Processing Unit (NPU) offloading. Essentially, mllm-NPU is an algorithm-system co-design that tackles a few semantic gaps between the LLM architecture and contemporary NPU design. Specifically, it re-constructs the prompt and model in three levels: (1) At prompt level, it divides variable-length prompts into multiple fixed-sized chunks while maintaining data dependencies; (2) At tensor level, it identifies and extracts significant outliers to run on the CPU/GPU in parallel with minimal overhead; (3) At block level, it schedules Transformer blocks in an out-of-order manner to the CPU/GPU and NPU based on their hardware affinity and sensitivity to accuracy. Compared to competitive baselines, mllm-NPU achieves 22.4x faster prefill speed and 30.7x energy savings on average, and up to 32.8x speedup in an end-to-end real-world application. For the first time, mllm-NPU achieves more than 1,000 tokens/sec prefilling for a billion-sized model (Qwen1.5-1.8B), paving the way towards practical on-device LLM.
Abstract (translated)
在本地大语言模型（LLMs）正在推动开发诸如UI任务自动化和个性化邮件自动回复等新型移动应用，而不会泄露用户的隐私数据。然而，在本地LLMs中，仍然存在可悲的推理延迟，特别是因为需要长上下文来生成准确、个性化的内容，以及移动CPU/GPU缺乏并行计算能力而导致的第一次token时间（预填充阶段）较长。为了实现实际的本地LLM，我们提出了mllm-NPU，这是第一个充分利用本地神经处理单元（NPU）卸载的LLM推理系统。本质上，mllm-NPU是一个基于算法和系统设计的LLM，它解决了LLM架构和当代NPU设计之间的几个语义差距。具体来说，它通过以下方式重构提示和模型：（1）在提示级别，它将变长提示切成多个固定大小的片段，同时保持数据依赖关系；（2）在张量级别，它识别并提取重要的异常值并利用CPU/GPU并行执行，同时最小化开销；（3）在块级别，它根据硬件亲和力和对准确性的敏感度，以异步方式调度Transformer块到CPU/GPU和NPU。与竞争基线相比，mllm-NPU实现了22.4倍的预填充速度和30.7倍的平均节能，并且在端到端现实应用中实现了32.8倍的速度提升。为了实现第一个亿级模型的本地LLM，mllm-NPU在预填充10亿个词以上时，取得了超过1,000个词/秒的预填充速度，为实际本地LLM的发展铺平道路。
URL
https://arxiv.org/abs/2407.05858
