Abstract
Neural networks commonly employ the McCulloch-Pitts neuron model, which is a linear model followed by a point-wise non-linear activation. Various researchers have already advanced inherently non-linear neuron models, such as quadratic neurons, generalized operational neurons, generative neurons, and super neurons, which offer stronger non-linearity compared to point-wise activation functions. In this paper, we introduce a novel and better non-linear neuron model called Padé neurons (Paons), inspired by Padé approximants. Paons offer several advantages, such as diversity of non-linearity, since each Paon learns a different non-linear function of its inputs, and layer efficiency, since Paons provide stronger non-linearity in much fewer layers compared to piecewise linear approximation. Furthermore, Paons include all previously proposed neuron models as special cases, thus any neuron model in any network can be replaced by Paons. We note that there has been a proposal to employ the Padé approximation as a generalized point-wise activation function, which is fundamentally different from our model. To validate the efficacy of Paons, in our experiments, we replace classic neurons in some well-known neural image super-resolution, compression, and classification models based on the ResNet architecture with Paons. Our comprehensive experimental results and analyses demonstrate that neural models built by Paons provide better or equal performance than their classic counterparts with a smaller number of layers. The PyTorch implementation code for Paon is open-sourced at this https URL.
Abstract (translated)
神经网络通常采用麦卡洛克-皮茨（McCulloch-Pitts）神经元模型，该模型由线性部分和逐点非线性激活函数组成。然而，一些研究人员已经提出了固有的非线性神经元模型，例如二次神经元、广义操作神经元、生成神经元以及超级神经元，这些模型提供了比逐点激活函数更强的非线性特性。在本文中，我们介绍了一种新型且更为优越的非线性神经元模型——帕德神经元（Paon），该模型灵感源自帕德逼近（Padé approximants）。Paon具有多种优势，例如其非线性的多样性：每个Paon都能学习输入的不同非线性函数；以及层效率：与分段线性近似相比，Paons能在更少的层数中提供更强的非线性特性。此外，Paon将所有先前提出的神经元模型作为特殊情况包含在内，因此任何网络中的任何神经元模型都可以被Paon替换。 值得注意的是，有人曾提出使用帕德逼近作为一种通用的逐点激活函数，这与我们的模型从根本上不同。为了验证Paons的有效性，在实验中，我们用Paon替换了基于ResNet架构的一些著名的神经图像超分辨率、压缩和分类模型中的经典神经元。通过全面的实验结果和分析表明，由Paon构建的神经网络在使用较少层数的情况下可以达到与传统模型相同或更好的性能。 帕德神经元（Paon）的PyTorch实现代码可在以下链接获取：[https URL]
URL
https://arxiv.org/abs/2601.04005
