Abstract
Masked self-supervised vision transformers have become a dominant pretraining paradigm, yet their substantial model size poses significant challenges for resource-constrained deployment and efficient transfer learning. A fundamental question remains: are all transformer blocks equally important for downstream performance? In this paper, we show that block importance in masked self-supervised vision transformers can be accurately estimated without access to any data. Our key finding is that the information entropy of pretrained block weights strongly correlates with oracle sensitivity obtained via iterative block removal and finetuning. This observation enables Gardener, a data-free, one-shot, block-level pruning principle that identifies redundant blocks through simple information-theoretic measurements. We evaluate Gardener on VideoMAE-B across multiple pruning ratios and downstream video recognition benchmarks. Despite its negligible computational overhead, Gardener consistently matches or outperforms existing data-free pruning baselines and closely approaches sensitivity-based pruning. Remarkably, even after pruning up to 91.7\% of blocks, the pruned model retains competitive transfer performance. Our results reveal substantial block-level redundancy in masked self-supervised vision transformers and demonstrate that information-theoretic analysis offers a principled and efficient pathway for model compression and resource-efficient transfer learning.
Abstract (translated)
遮蔽自监督视觉变压器已成为预训练的主导范式，但其巨大的模型规模对资源受限部署和高效迁移学习带来了重大挑战。一个基本的问题是：所有的Transformer块对于下游任务的表现都是同样重要的吗？在本文中，我们展示了可以在没有任何数据访问的情况下准确估计遮蔽自监督视觉变压器中的块重要性。我们的关键发现是，预训练的块权重的信息熵与通过迭代删除块和微调获得的Oracle敏感度有着强烈的关联。这一观察结果使Gardener成为可能——这是一种无数据、一次性、基于块级别的剪枝原则，它通过简单的信息理论测量来识别冗余的块。我们在VideoMAE-B上评估了Gardener，在多个剪枝比例和下游视频识别基准测试中进行了测试。尽管其计算开销几乎可以忽略不计，但Gardener在与现有的无数据剪枝基线相匹配或超越它们的同时，也接近基于敏感度的剪枝方法的表现。值得注意的是，即使是在修剪掉高达91.7%块之后，被剪枝后的模型仍能保持有竞争力的迁移性能。 我们的结果揭示了遮蔽自监督视觉Transformer中大量的块级别冗余，并且展示了信息理论分析提供了一条原则性和高效的方法来压缩模型并实现资源高效的迁移学习。
URL
https://arxiv.org/abs/2602.03918
