Abstract
The integration of event cameras and spiking neural networks holds great promise for energy-efficient visual processing. However, the limited availability of event data and the sparse nature of DVS outputs pose challenges for effective training. Although some prior work has attempted to transfer semantic knowledge from RGB datasets to DVS, they often overlook the significant distribution gap between the two modalities. In this paper, we propose Time-step Mixup knowledge transfer (TMKT), a novel fine-grained mixing strategy that exploits the asynchronous nature of SNNs by interpolating RGB and DVS inputs at various time-steps. To enable label mixing in cross-modal scenarios, we further introduce modality-aware auxiliary learning objectives. These objectives support the time-step mixup process and enhance the model's ability to discriminate effectively across different modalities. Our approach enables smoother knowledge transfer, alleviates modality shift during training, and achieves superior performance in spiking image classification tasks. Extensive experiments demonstrate the effectiveness of our method across multiple datasets. The code will be released after the double-blind review process.
Abstract (translated)
事件相机与脉冲神经网络的结合在节能视觉处理方面展现出巨大潜力。然而，有限的事件数据可用性和DVS输出的稀疏性给有效训练带来了挑战。尽管一些先前的工作尝试从RGB数据集中转移语义知识到DVS，但它们往往忽视了两种模式之间的显著分布差距。在这篇论文中，我们提出了一种新颖的时间步长Mixup知识传输（TMKT）策略，该策略通过在不同时间步对RGB和DVS输入进行插值来利用SNN的异步特性。为了实现在跨模态场景中的标签混合，我们进一步引入了感知模式的辅助学习目标。这些目标支持时间步长混洗过程，并增强了模型跨不同模式有效区分的能力。我们的方法能够实现更平滑的知识转移，在训练过程中减轻模式偏移问题，并在脉冲图像分类任务中取得卓越性能。广泛的实验验证了该方法在多个数据集上的有效性。代码将在双盲评审流程结束后发布。
URL
https://arxiv.org/abs/2509.12959
