Abstract
Gradient Descent (GD) and its variants are the primary tool for enabling efficient training of recurrent dynamical systems such as Recurrent Neural Networks (RNNs), Neural ODEs and Gated Recurrent units (GRUs). The dynamics that are formed in these models exhibit features such as neural collapse and emergence of latent representations that may support the remarkable generalization properties of networks. In neuroscience, qualitative features of these representations are used to compare learning in biological and artificial systems. Despite recent progress, there remains a need for theoretical tools to rigorously understand the mechanisms shaping learned representations, especially in finite, non-linear models. Here, we show that the gradient flow, which describes how the model's dynamics evolve over GD, can be decomposed into a product that involves two operators: a Parameter Operator, K, and a Linearized Flow Propagator, P. K mirrors the Neural Tangent Kernel in feed-forward neural networks, while P appears in Lyapunov stability and optimal control theory. We demonstrate two applications of our decomposition. First, we show how their interplay gives rise to low-dimensional latent dynamics under GD, and, specifically, how the collapse is a result of the network structure, over and above the nature of the underlying task. Second, for multi-task training, we show that the operators can be used to measure how objectives relevant to individual sub-tasks align. We experimentally and theoretically validate these findings, providing an efficient Pytorch package, \emph{KPFlow}, implementing robust analysis tools for general recurrent architectures. Taken together, our work moves towards building a next stage of understanding of GD learning in non-linear recurrent models.
Abstract (translated)
梯度下降（GD）及其变体是使递归动力系统（如循环神经网络（RNN）、神经ODE和门控循环单元（GRU））高效训练的主要工具。这些模型中形成的动态特性，例如神经坍缩和潜在表示的出现，可能支持网络显著的泛化能力。在神经科学领域，人们使用这些表示的定性特征来比较生物系统与人工系统的学习过程。尽管近年来取得了一些进展，但仍需要理论工具来严格理解塑造所学表示机制的过程，尤其是在有限且非线性的模型中。 本文展示了梯度流（描述GD过程中模型动态演变的方式）可以分解为一个涉及两个操作符的乘积：参数操作符K和线性化流动传播器P。其中，K在前馈神经网络中类似于神经切片核(Neural Tangent Kernel)，而P出现在李雅普诺夫稳定性(Lyapunov stability)和最优控制理论(optimal control theory)中。 我们展示了这种分解的两个应用：首先，展示了它们之间的交互如何在GD过程中产生低维潜在动态，并且具体地说明了坍缩是如何由网络结构决定的，而不仅仅是任务的本质。其次，在多任务训练的情况下，展示这些操作符可以用来衡量与各个子任务相关的目标的一致性。 我们通过实验和理论验证了这些发现，并提供了一个高效实用的Pytorch包*KPFlow*，用于实现对通用递归架构进行稳健分析的工具。总的来说，我们的工作朝着建立理解非线性递归模型中GD学习的新阶段迈进了一步。
URL
https://arxiv.org/abs/2507.06381
