Abstract
Large language models (LLM) have achieved remarkable performance across a wide range of tasks. However, their substantial parameter sizes pose significant challenges for deployment on edge devices with limited computational and memory resources. Low-rank compression is a promising approach to address this issue, as it reduces both computational and memory costs, making LLM more suitable for resource-constrained environments. Nonetheless, naïve low-rank compression methods require a significant reduction in the retained rank to achieve meaningful memory and computation savings. For a low-rank model, the ranks need to be reduced by more than half to yield efficiency gains. Such aggressive truncation, however, typically results in substantial performance degradation. To address this trade-off, we propose SkipCat, a novel low-rank compression framework that enables the use of higher ranks while achieving the same compression rates. First, we introduce an intra-layer shared low-rank projection method, where multiple matrices that share the same input use a common projection. This reduces redundancy and improves compression efficiency. Second, we propose a block skipping technique that omits computations and memory transfers for selected sub-blocks within the low-rank decomposition. These two techniques jointly enable our compressed model to retain more effective ranks under the same compression budget. Experimental results show that, without any additional fine-tuning, our method outperforms previous low-rank compression approaches by 7% accuracy improvement on zero-shot tasks under the same compression rate. These results highlight the effectiveness of our rank-maximized compression strategy in preserving model performance under tight resource constraints.
Abstract (translated)
大型语言模型（LLM）在各种任务中表现出卓越的性能，然而，它们庞大的参数规模给计算和内存资源有限的边缘设备部署带来了巨大挑战。低秩压缩作为一种有前景的方法被提出以解决这一问题，通过减少计算和内存成本使大规模语言模型更适合于资源受限的环境。但是，简单的低秩压缩方法需要大幅度降低保留的秩才能实现有意义的记忆和计算节省。对于一个低秩模型来说，为了获得效率提升，其排名通常需要减半以上。然而，这种激进的截断往往会导致性能显著下降。 为了解决这一权衡问题，我们提出了SkipCat，这是一种新的低秩压缩框架，在相同的压缩率下能够使用更高的秩。首先，我们引入了一种层内共享的低秩投影方法，其中多个具有相同输入的矩阵共用一个投影器以减少冗余并提高压缩效率。其次，我们提出了一种块跳过技术，该技术省略了选定子块在低秩分解中的计算和内存传输操作。这两种技术共同使我们的压缩模型能够在相同的压缩预算下保留更多的有效排名。 实验结果显示，在没有额外微调的情况下，与先前的低秩压缩方法相比，我们的方法在零样本任务中实现了7%的准确率提升，并且在相同压缩率下表现更优。这些结果突显了我们提出的最大化的秩压缩策略在资源紧张约束下的模型性能保持方面具有高度有效性。
URL
https://arxiv.org/abs/2512.13494
