Abstract
Although large language models (LLM) have achieved remarkable performance, their enormous parameter counts hinder deployment on resource-constrained hardware. Low-rank compression can reduce both memory usage and computational demand, but applying a uniform compression ratio across all layers often leads to significant performance degradation, and previous methods perform poorly during decoding. To address these issues, we propose the Fine-grained Low-Rank Compressor (FLRC), which efficiently determines an optimal rank allocation for each layer, and incorporates progressive low-rank decoding to maintain text generation quality. Comprehensive experiments on diverse benchmarks demonstrate the superiority of FLRC, achieving up to a 17% improvement in ROUGE-L on summarization tasks compared to state-of-the-art low-rank compression methods, establishing a more robust and efficient framework to improve LLM inference.
Abstract (translated)
尽管大型语言模型（LLM）已经取得了显著的性能成果，但其庞大的参数数量阻碍了它们在资源受限硬件上的部署。低秩压缩可以减少内存使用和计算需求，但是对所有层应用统一的压缩比率通常会导致明显的性能下降，并且之前的方法在解码过程中表现不佳。为了解决这些问题，我们提出了细粒度低秩压缩器（FLRC），它可以高效地为每一层确定最优的秩分配，并结合渐进式低秩解码来保持文本生成的质量。在各种基准测试中的全面实验表明了FLRC的优越性，在摘要任务中与最先进的低秩压缩方法相比，ROUGE-L评分提高了最多17%，从而建立了更为稳健和高效的框架以改进LLM的推理能力。
URL
https://arxiv.org/abs/2510.09332
