Abstract
The Neural Tangent Kernel (NTK) has emerged as a fundamental concept in the study of wide Neural Networks. In particular, it is known that the positivity of the NTK is directly related to the memorization capacity of sufficiently wide networks, i.e., to the possibility of reaching zero loss in training, via gradient descent. Here we will improve on previous works and obtain a sharp result concerning the positivity of the NTK of feedforward networks of any depth. More precisely, we will show that, for any non-polynomial activation function, the NTK is strictly positive definite. Our results are based on a novel characterization of polynomial functions which is of independent interest.
Abstract (translated)
神经元归一核（NTK）已成为研究广义神经网络的基本概念。特别是，已知NTK的正值与足够宽的网络的记忆能力直接相关，即在训练过程中达到零损失的可能性。在这里，我们将超越前人工作，得到关于任何深度的前馈网络NTK正值的尖锐结果。具体来说，我们将证明，对于任何非多项式激活函数，NTK都是严格正定实的。我们的结果基于一个关于多项式函数的新颖刻画，该函数具有独立的有意义性。
URL
https://arxiv.org/abs/2404.12928
