Abstract
With the growth of demand on neural network compression methods, the structured pruning methods including importance-based approach are actively studied. The magnitude importance and many correlated modern importance criteria often limit the capacity of pruning decision, since the filters with larger magnitudes are not likely to be pruned if the smaller one didn't, even if it is redundant. In this paper, we propose a novel pruning strategy to challenge this dominating effect of magnitude and provide fair chance to each filter to be pruned, by placing it on projective space. After that, we observe the gradient descent movement whether the filters move toward the origin or not, to measure how the filter is likely to be pruned. This measurement is used to construct PROscore, a novel importance score for IPPRO, a novel importance-based structured pruning with magnitude-indifference. Our evaluation results shows that the proposed importance criteria using the projective space achieves near-lossless pruning by reducing the performance drop in pruning, with promising performance after the finetuning. Our work debunks the ``size-matters'' myth in pruning and expands the frontier of importance-based pruning both theoretically and empirically.
Abstract (translated)
随着对神经网络压缩方法需求的增长，包括基于重要性方法的结构化剪枝方法得到了积极的研究。然而，基于幅度的重要性以及其他许多相关现代重要性标准往往限制了剪枝决策的能力，因为即使较大的滤波器是冗余的，如果较小的一个没有被剪枝的话，它也不大可能被剪枝。 在本文中，我们提出了一种新的剪枝策略来挑战这一主导效应，并为每个滤波器提供公平的机会去进行剪枝。我们将这些滤波器置于投影空间中，然后观察梯度下降运动是否朝向原点移动，以此衡量一个滤波器被剪枝的可能性。这种测量用于构建PROscore，这是一种新的重要性分数，专为IPPRO设计——一种基于重要性的结构化剪枝方法，在这种方法中，幅度的差异不再具有决定性作用。 我们的评估结果显示，使用投影空间中的新提出的重要的标准可以实现接近无损的剪枝，并通过减少剪枝后的性能下降来获得令人满意的结果。经过微调后，其表现仍然很有竞争力。我们的工作打破了“大小有关系”的剪枝神话，在理论和经验上都扩展了基于重要性的剪枝边界。
URL
https://arxiv.org/abs/2507.14171
