Abstract
Enhancing on-device large language models (LLMs) with contextual information from local data enables personalized and task-aware generation, powering use cases such as intelligent assistants and UI agents. While recent developments in neural processors have substantially improved the efficiency of prefill on mobile devices, the token-by-token generation process still suffers from high latency and limited hardware utilization due to its inherently memory-bound characteristics. This work presents CoordGen, a mobile inference framework that integrates speculative decoding with dynamic hardware scheduling to accelerate context-aware text generation on mobile devices. The framework introduces three synergistic components: (1) adaptive execution scheduling, which dynamically balances compute graphs between prefill and decoding phases; (2) context-aligned drafting, which improves speculative efficiency through lightweight online calibration to current tasks; and (3) hardware-efficient draft extension, which reuses and expands intermediate sequences to improve processing parallelism and reduce verification cost. Experiments on multiple smartphones and representative workloads show consistent improvements of up to 3.8x in generation speed and 4.7x in energy efficiency compared with existing mobile inference solutions. Component-level analysis further validates the contribution of each optimization.
Abstract (translated)
通过在设备上的大型语言模型（LLM）中加入来自本地数据的上下文信息，可以实现个性化和任务感知的生成功能，支持诸如智能助手和UI代理等应用场景。尽管最近在神经处理器方面的进展大大提高了移动设备上预填充过程的效率，但由于生成过程中固有的内存瓶颈特性，逐令牌生成仍然面临着高延迟和有限硬件利用率的问题。 这项工作提出了一种名为CoordGen的移动推理框架，该框架通过将推测性解码与动态硬件调度相结合，加速了在移动设备上下文感知文本生成的过程。框架引入了三个相互协同的组件： 1. **自适应执行调度**：这种机制能够根据当前任务需求，在预填充和解码阶段之间动态平衡计算图。 2. **上下文对齐草稿**：通过轻量级在线校准，提升推测效率，使得生成过程更加贴近实际任务需求。 3. **硬件高效的草稿扩展**：该组件复用并扩展中间序列以提高处理的并行性，并减少验证成本。 在多个智能手机和典型工作负载上的实验表明，相比现有的移动推理解决方案，CoordGen框架能够在文本生成速度上实现最高达3.8倍的提升，在能源效率方面则可达到4.7倍的增长。组件级别的分析进一步证实了每一项优化对于整体性能改进的具体贡献。
URL
https://arxiv.org/abs/2510.15312
