Abstract
In the past, research on a single low dimensional activation function in networks has led to internal covariate shift and gradient deviation problems. A relatively small research area is how to use function combinations to provide property completion for a single activation function application. We propose a network adversarial method to address the aforementioned challenges. This is the first method to use different activation functions in a network. Based on the existing activation functions in the current network, an adversarial function with opposite derivative image properties is constructed, and the two are alternately used as activation functions for different network layers. For complex situations, we propose a method of high-dimensional function graph decomposition(HD-FGD), which divides it into different parts and then passes through a linear layer. After integrating the inverse of the partial derivatives of each decomposed term, we obtain its adversarial function by referring to the computational rules of the decomposition process. The use of network adversarial methods or the use of HD-FGD alone can effectively replace the traditional MLP+activation function mode. Through the above methods, we have achieved a substantial improvement over standard activation functions regarding both training efficiency and predictive accuracy. The article addresses the adversarial issues associated with several prevalent activation functions, presenting alternatives that can be seamlessly integrated into existing models without any adverse effects. We will release the code as open source after the conference review process is completed.
Abstract (translated)
在过去，研究单一低维激活函数在网络中的作用导致了内部协变量偏移和梯度偏差问题。一个相对较小的研究领域是如何使用函数组合来提供单个激活函数应用的属性完成。我们提出了一个网络对抗方法来解决上述挑战。这是第一个在网络中使用不同激活函数的方法。根据当前网络中的激活函数，构建了一个具有相反导数图像属性的对抗函数，并将其交替用于不同网络层的激活函数。对于复杂情况，我们提出了高维函数图分解（HD-FGD）方法，将其划分为不同的部分并传递给线性层。然后通过整合每个分解项的逆导数，我们得到了其对抗函数，通过分解过程的计算规则进行参考。网络对抗方法和HD-FGD单独使用可以有效替代传统的MLP+激活函数模式。通过上述方法，我们在关于训练效率和预测准确性的标准激活函数方面取得了显著的改进。本文讨论了几个普遍激活函数的对抗问题，提出了可以轻松集成到现有模型中而不会产生任何不利影响的可替代方案。在会议审稿过程中完成代码发布为开源。
URL
https://arxiv.org/abs/2405.03712
