Abstract
Fractional-order differentiation has many characteristics different from integer-order differentiation. These characteristics can be applied to the optimization algorithms of artificial neural networks to obtain better results. However, due to insufficient theoretical research, at present, there is no fractional-order matrix differentiation method that is perfectly compatible with automatic differentiation (Autograd) technology. Therefore, we propose a fractional-order matrix differentiation calculation method. This method is introduced by the definition of the integer-order Jacobian matrix. We denote it as fractional-order Jacobian matrix differentiation (${\bf{J}^\alpha }$). Through ${\bf{J}^\alpha }$, we can carry out the matrix-based fractional-order chain rule. Based on the Linear module and the fractional-order differentiation, we design the fractional-order Autograd technology to enable the use of fractional-order differentiation in hidden layers, thereby enhancing the practicality of fractional-order differentiation in deep learning. In the experiment, according to the PyTorch framework, we design fractional-order Linear (FLinear) and replace this http URL in the multilayer perceptron with FLinear. Through the qualitative analysis of the training set and validation set $Loss$, the quantitative analysis of the test set indicators, and the analysis of time consumption and GPU memory usage during model training, we verify the superior performance of ${\bf{J}^\alpha }$ and prove that it is an excellent fractional-order gradient descent method in the field of deep learning.
Abstract (translated)
分数阶微分具有许多与整数阶微分不同的特性。这些特性可以应用于人工神经网络的优化算法，从而获得更好的结果。然而，由于理论研究不足，在目前阶段还没有一种完全兼容自动微分（Autograd）技术的分数阶矩阵微分方法。因此，我们提出了一种分数阶矩阵微分计算方法。这种方法通过整数阶雅可比矩阵的定义引入，并将其标记为分数阶雅可比矩阵微分（${\bf{J}^\alpha }$）。利用 ${\bf{J}^\alpha }$ ，我们可以进行基于矩阵的分数阶链式法则运算。结合线性模块和分数阶微分，我们设计了分数阶Autograd技术，以使隐藏层能够使用分数阶微分，从而增强分数阶微分在深度学习中的实用性。 通过根据PyTorch框架设计分数阶Linear（FLinear）并将其替换多层感知机网络中的相应部分，我们在实验中通过对训练集和验证集$Loss$的定性分析、测试集指标的定量分析以及模型训练过程中时间消耗和GPU内存使用的分析，验证了${\bf{J}^\alpha }$ 的优越性能，并证明它是深度学习领域中一种优秀的分数阶梯度下降方法。
URL
https://arxiv.org/abs/2506.07408
