Abstract
Stability in recurrent neural models poses a significant challenge, particularly in developing biologically plausible neurodynamical models that can be seamlessly trained. Traditional cortical circuit models are notoriously difficult to train due to expansive nonlinearities in the dynamical system, leading to an optimization problem with nonlinear stability constraints that are difficult to impose. Conversely, recurrent neural networks (RNNs) excel in tasks involving sequential data but lack biological plausibility and interpretability. In this work, we address these challenges by linking dynamic divisive normalization (DN) to the stability of ORGaNICs, a biologically plausible recurrent cortical circuit model that dynamically achieves DN and has been shown to simulate a wide range of neurophysiological phenomena. By using the indirect method of Lyapunov, we prove the remarkable property of unconditional local stability for an arbitrary-dimensional ORGaNICs circuit when the recurrent weight matrix is the identity. We thus connect ORGaNICs to a system of coupled damped harmonic oscillators, which enables us to derive the circuit's energy function, providing a normative principle of what the circuit, and individual neurons, aim to accomplish. Further, for a generic recurrent weight matrix, we prove the stability of the 2D model and demonstrate empirically that stability holds in higher dimensions. Finally, we show that ORGaNICs can be trained by backpropagation through time without gradient clipping/scaling, thanks to its intrinsic stability property and adaptive time constants, which address the problems of exploding, vanishing, and oscillating gradients. By evaluating the model's performance on RNN benchmarks, we find that ORGaNICs outperform alternative neurodynamical models on static image classification tasks and perform comparably to LSTMs on sequential tasks.
Abstract (translated)
循环神经网络（RNNs）的稳定性 poses 是一个重大的挑战，尤其是在开发生物 plausible 的神经动力学模型时，可以无缝训练。传统的皮质电路模型由于动态系统中的非线性拓扑结构，导致具有非线性稳定性约束的优化问题很难求解。相反，循环神经网络（RNNs）在涉及序列数据的任务上表现出色，但缺乏生物合理性和可解释性。在这项工作中，我们通过将动态分枝 normalization（DN）与 ORGaNICs 的稳定性联系起来，解决了这些挑战。ORGaNICs 是一种生物 plausible 的循环神经网络模型，具有动态地实现 DN 并已展示出模拟广泛的神经生理现象的能力。通过使用 Lyapunov 间接法，我们证明了任意维度的 ORGaNICs 电路在循环权重矩阵为 identity 时的条件局部稳定性惊人的特性。因此，我们将 ORGaNICs 连接到耦合阻尼谐波振荡器的系统中，这使得我们能够求出电路的能量函数，为电路和单个神经元提供了规范原理，即它们致力于实现的目标。此外，对于任意维度的通用循环权重矩阵，我们证明了 2D 模型的稳定性，并实验证明了在更高维度中稳定性依然存在。最后，我们证明了 ORGaNICs 可以通过反向传播通过时间进行训练，得益于其固有稳定性特性和自适应时间常数，这解决了爆炸、消失和振荡梯度的問題。通过评估模型在 RNN 基准测试上的性能，我们发现 ORGaNICs 在静态图像分类任务上优于其他神经动力学模型，同时在序列任务上与 LSTMs 相当。
URL
https://arxiv.org/abs/2409.18946
