Abstract
There is a growing demand to deploy computation-intensive deep learning (DL) models on resource-constrained mobile devices for real-time intelligent applications. Equipped with a variety of processing units such as CPUs, GPUs, and NPUs, the mobile devices hold potential to accelerate DL inference via parallel execution across heterogeneous processors. Various efficient parallel methods have been explored to optimize computation distribution, achieve load balance, and minimize communication cost across processors. Yet their practical effectiveness in the dynamic and diverse real-world mobile environment is less explored. This paper presents a holistic empirical study to assess the capabilities and challenges associated with parallel DL inference on heterogeneous mobile processors. Through carefully designed experiments covering various DL models, mobile software/hardware environments, workload patterns, and resource availability, we identify limitations of existing techniques and highlight opportunities for cross-level optimization.
Abstract (translated)
越来越多的需求将计算密集型深度学习（DL）模型在资源受限的移动设备上部署，以实现实时智能应用。配备各种处理单元（如CPU、GPU和NPU），移动设备通过异构处理器并行执行交并计算来加速DL推理。已经探索了许多有效的并行方法来优化计算分布，实现负载均衡和最小化处理器之间的通信成本。然而，在动态和多样化的真实移动环境中，它们在平行DL推理方面的实际效果还有待进一步研究。本文进行了一项全面的实证研究，以评估异构移动处理器上并行DL推理的能力和挑战。通过精心设计的实验，覆盖各种DL模型、移动软件/硬件环境、工作负载模式和资源可用性，我们找出了现有技术的局限性，并强调了跨层优化的机会。
URL
https://arxiv.org/abs/2405.01851
