Abstract
This paper introduces a novel framework for designing efficient neural network architectures specifically tailored to tiny machine learning (TinyML) platforms. By leveraging large language models (LLMs) for neural architecture search (NAS), a vision transformer (ViT)-based knowledge distillation (KD) strategy, and an explainability module, the approach strikes an optimal balance between accuracy, computational efficiency, and memory usage. The LLM-guided search explores a hierarchical search space, refining candidate architectures through Pareto optimization based on accuracy, multiply-accumulate operations (MACs), and memory metrics. The best-performing architectures are further fine-tuned using logits-based KD with a pre-trained ViT-B/16 model, which enhances generalization without increasing model size. Evaluated on the CIFAR-100 dataset and deployed on an STM32H7 microcontroller (MCU), the three proposed models, LMaNet-Elite, LMaNet-Core, and QwNet-Core, achieve accuracy scores of 74.50%, 74.20% and 73.00%, respectively. All three models surpass current state-of-the-art (SOTA) models, such as MCUNet-in3/in4 (69.62% / 72.86%) and XiNet (72.27%), while maintaining a low computational cost of less than 100 million MACs and adhering to the stringent 320 KB static random-access memory (SRAM) constraint. These results demonstrate the efficiency and performance of the proposed framework for TinyML platforms, underscoring the potential of combining LLM-driven search, Pareto optimization, KD, and explainability to develop accurate, efficient, and interpretable models. This approach opens new possibilities in NAS, enabling the design of efficient architectures specifically suited for TinyML.
Abstract (translated)
这篇论文介绍了一种为小型机器学习（TinyML）平台设计高效神经网络架构的新型框架。该方法通过利用大型语言模型（LLM）进行神经架构搜索（NAS），结合基于视觉变换器（ViT）的知识蒸馏（KD）策略和可解释性模块，实现了准确度、计算效率以及内存使用之间的最佳平衡。由LLM引导的搜索在层次化的搜索空间中探索，通过以精度、乘积累加操作（MACs）和内存指标为基础的帕累托优化来精炼候选架构。最优表现的架构进一步利用预训练的ViT-B/16模型进行基于logits的KD微调，从而提升泛化能力而不增加模型大小。 在CIFAR-100数据集上评估并部署到STM32H7微控制器（MCU）上的三个提出的模型——LMaNet-Elite、LMaNet-Core和QwNet-Core，分别达到了74.50%、74.20% 和 73.00% 的精度。这三个模型均超越了当前最先进的（SOTA）模型，如MCUNet-in3/in4（69.62%/72.86%）和XiNet（72.27%），同时保持了低于1亿MACs的低计算成本，并且遵守严格的320KB静态随机存取内存（SRAM）限制。这些结果展示了该框架在TinyML平台上的效率与性能，强调了结合LLM驱动搜索、帕累托优化、KD和可解释性开发准确、高效及可解释模型的巨大潜力。 这种方法为NAS开辟了新的可能性，使得设计适合TinyML的高效架构成为可能。
URL
https://arxiv.org/abs/2504.09685
