Abstract
Spiking neural networks (SNNs) are widely applied in various fields due to their energy-efficient and fast-inference capabilities. Applying SNNs to reinforcement learning (RL) can significantly reduce the computational resource requirements for agents and improve the algorithm's performance under resource-constrained conditions. However, in current spiking reinforcement learning (SRL) algorithms, the simulation results of multiple time steps can only correspond to a single-step decision in RL. This is quite different from the real temporal dynamics in the brain and also fails to fully exploit the capacity of SNNs to process temporal data. In order to address this temporal mismatch issue and further take advantage of the inherent temporal dynamics of spiking neurons, we propose a novel temporal alignment paradigm (TAP) that leverages the single-step update of spiking neurons to accumulate historical state information in RL and introduces gated units to enhance the memory capacity of spiking neurons. Experimental results show that our method can solve partially observable Markov decision processes (POMDPs) and multi-agent cooperation problems with similar performance as recurrent neural networks (RNNs) but with about 50% power consumption.
Abstract (translated)
尖峰神经网络（SNNs）因其在节能和快速推理能力而广泛应用于各种领域。将SNN应用于强化学习（RL）可以显著降低代理程序的计算资源需求，并在资源受限条件下提高算法的性能。然而，在当前的尖峰强化学习（SRL）算法中，多个时间步的模拟结果只能对应于RL中的单步决策。这与大脑的实际时间动态以及SNNs处理时间数据的能力之间存在很大的差异。为了解决这一时间差问题，并更好地利用尖峰神经元的固有时间动态，我们提出了一个新的时间对齐范式（TAP）。它利用尖峰神经元的单步更新来累积历史状态信息，并引入门控单元来增强尖峰神经元的记忆容量。实验结果表明，我们的方法可以与具有类似性能的循环神经网络（RNNs）解决部分可观察的马尔可夫决策过程（POMDP）和多智能体合作问题，但功耗大约为RNN的50%。
URL
https://arxiv.org/abs/2404.15597
