Abstract
Vision-Language-Action Models (VLAs) have emerged as a key paradigm of Physical AI and are increasingly deployed in autonomous vehicles, robots, and smart spaces. In these resource-constrained on-device settings, selecting an appropriate large language model (LLM) backbone is a critical challenge: models must balance accuracy with strict inference latency and hardware efficiency constraints. This makes hardware-software co-design a game-changing requirement for on-device LLM deployment, where each hardware platform demands a tailored architectural solution. We propose a hardware co-design law that jointly captures model accuracy and inference performance. Specifically, we model training loss as an explicit function of architectural hyperparameters and characterise inference latency via roofline modelling. We empirically evaluate 1,942 candidate architectures on NVIDIA Jetson Orin, training 170 selected models for 10B tokens each to fit a scaling law relating architecture to training loss. By coupling this scaling law with latency modelling, we establish a direct accuracy-latency correspondence and identify the Pareto frontier for hardware co-designed LLMs. We further formulate architecture search as a joint optimisation over precision and performance, deriving feasible design regions under industrial hardware and application budgets. Our approach reduces architecture selection from months to days. At the same latency as Qwen2.5-0.5B on the target hardware, our co-designed architecture achieves 19.42% lower perplexity on WikiText-2. To our knowledge, this is the first principled and operational framework for hardware co-design scaling laws in on-device LLM deployment. We will make the code and related checkpoints publicly available.
Abstract (translated)
视觉语言行动模型（VLAs）已作为物理人工智能的关键范式而出现，并且越来越多地被部署在自主车辆、机器人和智能空间中。在这种资源受限的设备环境中，选择合适的大型语言模型（LLM）骨干是一个关键挑战：模型必须在严格的推理延迟和硬件效率限制下平衡准确性和性能。这使得软硬件协同设计成为设备上部署LLM的关键需求，每个硬件平台都要求定制的架构解决方案。我们提出了一项软硬件协同设计法则，该法则同时捕捉模型精度和推理性能。具体而言，我们将训练损失建模为架构超参数的显式函数，并通过顶点模型来表征推理延迟。我们在NVIDIA Jetson Orin上对1,942个候选架构进行了实证评估，每个选择了170种模型进行10B代币的训练，以拟合从架构到训练损失的关系。通过将这种扩展法则与延迟建模相结合，我们建立了一个直接的精度-延迟对应关系，并识别出软硬件协同设计LLM的帕累托前沿。进一步地，我们将架构搜索形式化为对精度和性能的同时优化，在工业硬件和应用预算下导出了可行的设计区域。我们的方法将架构选择的时间从数月缩短到了几天。在目标硬件上的相同延迟条件下，我们所提出的协同设计架构相比Qwen2.5-0.5B模型在WikiText-2上实现了19.42%更低的困惑度。据我们所知，这是首个系统且可操作的软硬件协同设计扩展法则框架，用于设备上LLM部署。我们将公开发布代码及相关检查点。
URL
https://arxiv.org/abs/2602.10377
