Abstract
Vision Transformer have set new benchmarks in several tasks, but these models come with the lack of high computational costs which makes them impractical for resource limited hardware. Network pruning reduces the computational complexity by removing less important operations while maintaining performance. However, pruning a model on an unseen data domain, leads to a misevaluation of weight significance, resulting in suboptimal resource assignment. In this work, we find that task-sensitive layers initially fail to improve the feature representation on downstream tasks, leading to performance loss for early pruning decisions. To address this problem, we introduce Pruning by Block Benefit (P3B), a pruning method that utilizes the relative contribution on block level to globally assign parameter resources. P3B identifies low-impact components to reduce parameter allocation while preserving critical ones. Classical pruning mask optimization struggles to reactivate zero-mask-elements. In contrast, P3B sets a layerwise keep ratio based on global performance metrics, ensuring the reactivation of late-converging blocks. We show in extensive experiments that P3B is a state of the art pruning method with most noticeable gains in transfer learning tasks. Notably, P3B is able to conserve high performance, even in high sparsity regimes of 70% parameter reduction while only losing 0.64% in accuracy.
Abstract (translated)
视觉变压器已经在多个任务中设立了新的基准，但这些模型伴随着高昂的计算成本，这使得它们在资源有限的硬件上变得不切实际。网络剪枝通过移除不太重要的操作来减少计算复杂性，同时保持性能不变。然而，在未见过的数据域上进行剪枝会导致权重重要性的评估失误，从而导致次优的资源分配。在这项工作中，我们发现任务敏感层最初未能改善下游任务中的特征表示能力，导致早期剪枝决策下的性能下降。 为了解决这个问题，我们引入了一种名为块收益剪枝（Pruning by Block Benefit, P3B）的方法，该方法利用块级别的相对贡献来全局分配参数资源。P3B识别低影响组件以减少参数分配同时保留关键部分。传统的剪枝掩码优化很难重新激活被设为零的元素。相比之下，P3B根据整体性能指标设定逐层保持比例，确保晚期收敛模块的重新激活。 在广泛的实验中我们展示了P3B是目前最先进的剪枝方法，在迁移学习任务中表现出最显著的优势。值得注意的是，即使是在高达70%参数减少（高稀疏度）的情况下，P3B仍能够维持高性能，仅损失了0.64%的准确性。
URL
https://arxiv.org/abs/2506.23675
