Abstract
To address the weight coupling problem, certain studies introduced few-shot Neural Architecture Search (NAS) methods, which partition the supernet into multiple sub-supernets. However, these methods often suffer from computational inefficiency and tend to provide suboptimal partitioning schemes. To address this problem more effectively, we analyze the weight coupling problem from a novel perspective, which primarily stems from distinct modules in succeeding layers imposing conflicting gradient directions on the preceding layer modules. Based on this perspective, we propose the Gradient Contribution (GC) method that efficiently computes the cosine similarity of gradient directions among modules by decomposing the Vector-Jacobian Product during supernet backpropagation. Subsequently, the modules with conflicting gradient directions are allocated to distinct sub-supernets while similar ones are grouped together. To assess the advantages of GC and address the limitations of existing Graph Neural Architecture Search methods, which are limited to searching a single type of Graph Neural Networks (Message Passing Neural Networks (MPNNs) or Graph Transformers (GTs)), we propose the Unified Graph Neural Architecture Search (UGAS) framework, which explores optimal combinations of MPNNs and GTs. The experimental results demonstrate that GC achieves state-of-the-art (SOTA) performance in supernet partitioning quality and time efficiency. In addition, the architectures searched by UGAS+GC outperform both the manually designed GNNs and those obtained by existing NAS methods. Finally, ablation studies further demonstrate the effectiveness of all proposed methods.
Abstract (translated)
为了应对权重耦合问题，一些研究引入了少量样本的神经架构搜索（NAS）方法，这些方法将超网络分割为多个子超网络。然而，这些方法往往在计算效率上存在不足，并且倾向于提供次优的分割方案。为了更有效地解决这个问题，我们从一个新的视角分析了权重耦合问题，其主要根源在于后续层中的不同模块对前一层模块施加了相互冲突的梯度方向。基于这一视角，我们提出了梯度贡献（GC）方法，该方法通过在超网络反向传播过程中分解矢量-雅可比积来高效计算模块之间梯度方向的余弦相似性。随后，具有冲突梯度方向的模块被分配到不同的子超网络中，而类似模块则被分组在一起。 为了评估GC的优势，并解决现有图神经架构搜索方法（仅限于搜索一种类型的图形神经网络，如消息传递神经网络(MPNNs)或图变换器(GTs)）的局限性，我们提出了统一图神经架构搜索(UGAS)框架，该框架探索了MPNN和GT的最佳组合。实验结果表明，GC在超网络分割质量和时间效率上达到了最先进的水平（SOTA）。此外，通过UGAS+GC搜索到的架构优于手动设计的GNNs以及现有NAS方法获得的结果。 最后，消融研究进一步证明了所有提出的方法的有效性。
URL
https://arxiv.org/abs/2506.01231
