Abstract
Spiking neural networks (SNNs) offer an energy-efficient alternative to traditional neural networks due to their event-driven computing paradigm. However, recent advancements in spiking transformers have focused on improving accuracy with large-scale architectures, which require significant computational resources and limit deployment on resource-constrained devices. In this paper, we propose a simple yet effective token pruning method for spiking transformers, termed TP-Spikformer, that reduces storage and computational overhead while maintaining competitive performance. Specifically, we first introduce a heuristic spatiotemporal information-retaining criterion that comprehensively evaluates tokens' importance, assigning higher scores to informative tokens for retention and lower scores to uninformative ones for pruning. Based on this criterion, we propose an information-retaining token pruning framework that employs a block-level early stopping strategy for uninformative tokens, instead of removing them outright. This also helps preserve more information during token pruning. We demonstrate the effectiveness, efficiency and scalability of TP-Spikformer through extensive experiments across diverse architectures, including Spikformer, QKFormer and Spike-driven Transformer V1 and V3, and a range of tasks such as image classification, object detection, semantic segmentation and event-based object tracking. Particularly, TP-Spikformer performs well in a training-free manner. These results reveal its potential as an efficient and practical solution for deploying SNNs in real-world applications with limited computational resources.
Abstract (translated)
脉冲神经网络（SNN）由于其事件驱动的计算范式，为传统的神经网络提供了一种节能的替代方案。然而，最近在脉冲变压器方面的进展主要集中在通过大规模架构来提高准确性上，这需要大量的计算资源，并限制了这些模型在资源受限设备上的部署。在这篇论文中，我们提出了一种简单而有效的脉冲变压器标记剪枝方法，名为TP-Spikformer，该方法减少了存储和计算开销，同时保持了竞争力的性能。 具体来说，首先我们引入了一个基于时空信息保留的标准来评估标记的重要性，给重要的标记分配较高的分数以保留，对不太重要的标记则分配较低的分数进行裁减。在此标准基础上，我们提出了一种信息保留标记剪枝框架，采用块级早期停止策略，而不是直接去除那些不太重要的标记，这也有助于在标记裁剪过程中保存更多信息。 通过广泛的实验验证了TP-Spikformer的有效性、效率和可扩展性，这些实验涵盖了多种架构（如Spikformer、QKFormer以及Spike驱动的Transformer V1 和V3），以及包括图像分类、目标检测、语义分割及基于事件的目标跟踪等多种任务。特别值得注意的是，TP-Spikformer在无需训练的情况下也能表现出色。 这些结果揭示了其作为部署资源受限的真实世界应用中的SNN的一个高效且实用解决方案的巨大潜力。
URL
https://arxiv.org/abs/2603.00527
