Abstract
Neural Ordinary Differential Equations (ODEs) have been successful in various applications due to their continuous nature and parameter-sharing efficiency. However, these unique characteristics also introduce challenges in training, particularly with respect to gradient computation accuracy and convergence analysis. In this paper, we address these challenges by investigating the impact of activation functions. We demonstrate that the properties of activation functions, specifically smoothness and nonlinearity, are critical to the training dynamics. Smooth activation functions guarantee globally unique solutions for both forward and backward ODEs, while sufficient nonlinearity is essential for maintaining the spectral properties of the Neural Tangent Kernel (NTK) during training. Together, these properties enable us to establish the global convergence of Neural ODEs under gradient descent in overparameterized regimes. Our theoretical findings are validated by numerical experiments, which not only support our analysis but also provide practical guidelines for scaling Neural ODEs, potentially leading to faster training and improved performance in real-world applications.
Abstract (translated)
神经普通微分方程（ODE）由于其连续性特性和参数共享效率，在各种应用中取得了成功。然而，这些独特的特性也给训练带来了挑战，特别是在梯度计算精度和收敛分析方面。在本文中，我们通过研究激活函数的影响来解决这些问题。我们证明了激活函数的属性，特别是平滑性和非线性，对于训练动态至关重要。平滑激活函数保证了正向和反向ODE的全局唯一解，而足够的非线性对于保持训练过程中的神经切核（NTK）谱特性是必不可少的。这些性质共同使得我们在超参数化条件下通过梯度下降方法建立了Neural ODEs的全局收敛性。我们的理论发现得到了数值实验的支持，不仅验证了我们的分析，而且还为扩展Neural ODEs提供了实用指南，这可能导致更快的训练速度和在实际应用中的性能提升。
URL
https://arxiv.org/abs/2509.22436
