Abstract
The combination of linear transformations and non-linear activation functions forms the foundation of most modern deep neural networks, enabling them to approximate highly complex functions. This paper explores the introduction of quadratic transformations to further increase nonlinearity in neural networks, with the aim of enhancing the performance of existing architectures. To reduce parameter complexity and computational complexity, we propose a lightweight quadratic enhancer that uses low-rankness, weight sharing, and sparsification techniques. For a fixed architecture, the proposed approach introduces quadratic interactions between features at every layer, while only adding negligible amounts of additional model parameters and forward computations. We conduct a set of proof-of-concept experiments for the proposed method across three tasks: image classification, text classification, and fine-tuning large-language models. In all tasks, the proposed approach demonstrates clear and substantial performance gains.
Abstract (translated)
线性变换与非线性激活函数的结合构成了大多数现代深度神经网络的基础，使它们能够近似复杂度极高的函数。本文探讨了引入二次变换以进一步增加神经网络中的非线性程度，旨在提升现有架构的表现能力。为了降低参数复杂性和计算复杂性，我们提出了一种轻量级的二次增强器，该增强器利用低秩、权重共享和稀疏化技术。对于固定的架构，在每一层中我们的方法引入了特征之间的二次交互作用，并且仅添加了几乎可以忽略不计的额外模型参数和前向计算。 我们在三个任务上进行了概念验证实验：图像分类、文本分类以及大型语言模型的微调。在所有这些任务中，所提出的方法均展示了明显的性能提升。
URL
https://arxiv.org/abs/2510.03276
