Abstract
Large language models have demonstrated capabilities in text generation, while their increasing parameter scales present challenges in computational and memory efficiency. Post-training sparsity (PTS), which reduces model cost by removing weights from dense networks, is an effective approach. However, native dense matrices lack high sparsity, making existing approaches that directly remove weights disrupt model states, resulting in unsatisfactory performance recovery even with post-tuning. We propose Sparsity Induction, which promotes models toward higher sparsity at both distribution and feature levels before pruning, to push the limits of PTS. At the distribution level, we enhance distributional sparsity through mathematically equivalent scaling transformations, which are fully absorbable and incur no extra parameters or inference-time overhead. At the feature level, we introduce Spectral Norm Loss to promote feature sparsity from a low-rank perspective. Experiments across diverse model architectures and tasks demonstrate that our method further enhances sparsity-friendliness, achieving superior pruning performance over existing approaches.
Abstract (translated)
大型语言模型在文本生成方面表现出强大的能力，但随着参数规模的增加，计算和内存效率问题也日益突出。后训练稀疏化（PTS）通过从密集网络中移除权重来降低模型成本，是一种有效的方法。然而，原有的密集矩阵缺乏高斯度特性，直接移除权重会破坏模型状态，即使经过后续调优也无法达到令人满意的性能恢复。我们提出了诱导稀疏性方法，该方法在剪枝前从分布和特征两个层面推动模型向更高的稀疏性发展，从而突破PTS的限制。 在分布层面上，我们通过数学等价的缩放变换增强分布特性上的稀疏性，这些变换可以完全吸收，并且不会增加额外参数或推理时间开销。在特征层面上，我们引入了谱范数损失（Spectral Norm Loss），从低秩角度促进特征稀疏性的形成。 跨多种模型架构和任务的实验表明，我们的方法进一步提高了模型对稀疏化的友好性，在剪枝性能方面超越了现有方法。
URL
https://arxiv.org/abs/2602.21652
