Abstract
Large Language Models (LLMs) have been emerging as prominent AI models for solving many natural language tasks due to their high performance (e.g., accuracy) and capabilities in generating high-quality responses to the given inputs. However, their large computational cost, huge memory footprints, and high processing power/energy make it challenging for their embedded deployments. Amid several tinyLLMs, recent works have proposed spike-driven language models (SLMs) for significantly reducing the processing power/energy of LLMs. However, their memory footprints still remain too large for low-cost and resource-constrained embedded devices. Manual quantization approach may effectively compress SLM memory footprints, but it requires a huge design time and compute power to find the quantization setting for each network, hence making this approach not-scalable for handling different networks, performance requirements, and memory budgets. To bridge this gap, we propose QSLM, a novel framework that performs automated quantization for compressing pre-trained SLMs, while meeting the performance and memory constraints. To achieve this, QSLM first identifies the hierarchy of the given network architecture and the sensitivity of network layers under quantization, then employs a tiered quantization strategy (e.g., global-, block-, and module-level quantization) while leveraging a multi-objective performance-and-memory trade-off function to select the final quantization setting. Experimental results indicate that our QSLM reduces memory footprint by up to 86.5%, reduces power consumption by up to 20%, maintains high performance across different tasks (i.e., by up to 84.4% accuracy of sentiment classification on the SST-2 dataset and perplexity score of 23.2 for text generation on the WikiText-2 dataset) close to the original non-quantized model while meeting the performance and memory constraints.
Abstract (translated)
大型语言模型（LLMs）由于其高性能和生成高质量响应的能力，已成为解决许多自然语言任务的突出AI模型。然而，它们巨大的计算成本、庞大的内存占用以及对处理能力和能源的巨大需求，使其在嵌入式设备上的部署变得具有挑战性。鉴于此，最近的研究提出了一种称为脉冲驱动语言模型（SLMs）的方法，以显著降低LLM的处理功率和能耗。尽管如此，这些模型仍然因内存占用过大而难以应用于低成本、资源受限的嵌入式设备中。 手动量化方法可以通过压缩SLM的内存足迹来有效解决问题，但这种方法需要大量的设计时间和计算能力来为每个网络找到合适的量化设置，因此对于不同网络、性能需求和内存预算而言并不具备可扩展性。为了弥合这一差距，我们提出了QSLM——一种新的框架，它能够自动执行预训练SLMs的量化以压缩其大小，同时满足性能和内存约束。 为了实现这个目标，QSLM首先识别给定网络架构的层次结构以及在量化过程中各层的敏感性，然后采用分层量化策略（例如全局、块级和模块级量化），并利用一个兼顾性能与内存双重优化的目标函数来选择最终的量化设置。实验结果显示，我们的QSLM可以在满足性能和内存约束的同时，将模型的内存占用减少多达86.5%，能耗降低最多达20%。同时，在不同的任务上保持高精度（例如SST-2数据集情感分类准确率为84.4%，WikiText-2数据集文本生成的困惑度得分为23.2），接近未量化模型的表现水平。
URL
https://arxiv.org/abs/2601.00679
