Abstract
The design of optimization algorithms for neural networks remains a critical challenge, with most existing methods relying on heuristic adaptations of gradient-based approaches. This paper introduces KO (Kinetics-inspired Optimizer), a novel neural optimizer inspired by kinetic theory and partial differential equation (PDE) simulations. We reimagine the training dynamics of network parameters as the evolution of a particle system governed by kinetic principles, where parameter updates are simulated via a numerical scheme for the Boltzmann transport equation (BTE) that models stochastic particle collisions. This physics-driven approach inherently promotes parameter diversity during optimization, mitigating the phenomenon of parameter condensation, i.e. collapse of network parameters into low-dimensional subspaces, through mechanisms analogous to thermal diffusion in physical systems. We analyze this property, establishing both a mathematical proof and a physical interpretation. Extensive experiments on image classification (CIFAR-10/100, ImageNet) and text classification (IMDB, Snips) tasks demonstrate that KO consistently outperforms baseline optimizers (e.g., Adam, SGD), achieving accuracy improvements while computation cost remains comparable.
Abstract (translated)
神经网络的优化算法设计仍然是一个关键挑战，大多数现有的方法依赖于基于梯度的方法的启发式改进。本文介绍了KO（Kinetics-inspired Optimizer），这是一种受动力学理论和偏微分方程（PDE）模拟启发的新颖神经网络优化器。我们重新构思了网络参数训练的动力学，将其视为由动力学原理支配的粒子系统的演化过程，并通过玻尔兹曼输运方程（BTE）的数值方案来模拟参数更新，该模型描述了随机粒子碰撞的过程。这种物理驱动的方法内在地促进了优化过程中参数的多样性，通过类似于物理系统中热扩散机制的方式缓解参数凝聚现象，即网络参数坍缩到低维子空间的现象。我们分析了这一特性，并建立了数学证明和物理解释。 在图像分类（CIFAR-10/100、ImageNet）和文本分类（IMDB、Snips）任务的广泛实验中表明，KO始终优于基线优化器（例如Adam、SGD），并在计算成本相当的情况下提高了准确性。
URL
https://arxiv.org/abs/2505.14777
